Data Engineer

Hiring: Lead Data Engineer – Databricks | Manhattan, NY | Hybrid | Remote Also Available

We're looking for an experienced Lead Data Engineer with strong hands-on expertise in Databricks, SQL, Python, and PySpark to design and optimize scalable data pipelines and enterprise data platforms. The ideal candidate will provide technical leadership and implement data engineering best practices to support analytics, digital products, and AI-driven use cases.

Location: New York, NY (3 days/week or US Remote)

Experience: 6–12 Years

Interview Process: 3 Rounds, Including One Onsite Round in NY/NJ or at Any Client Office Across the US

Work Authorization: USC & GC Candidates Only

Key Skills:

• 6+ years of data engineering experience with substantial hands-on Databricks expertise
• Strong proficiency in SQL, Python, and PySpark
• Experience designing and developing scalable data pipelines using Medallion Architecture
• Strong understanding of Databricks Lakehouse architecture, Delta Lake, and Unity Catalog
• Experience with batch and incremental data processing, workflow orchestration, and data transformation
• Expertise in SQL query optimization, distributed processing, and performance tuning
• Experience with data quality, observability, lineage, and production troubleshooting
• Familiarity with Git, CI/CD, and production deployment practices
• Experience with AWS services, particularly S3, is a plus
• Knowledge of Auto Loader, Structured Streaming, CDC, Databricks Asset Bundles, or Apache Airflow is preferred
• Databricks Data Engineer Associate or Professional certification is a plus
• Experience with streaming, clickstream, or advertising datasets is an added advantage

What We're Looking For:

A technically strong Lead Data Engineer who can build production-grade Databricks solutions, optimize data pipelines for performance and cost efficiency, lead solution design and code reviews, and mentor engineers while delivering scalable and reliable data platforms.