About this opportunity
Hiring: Lead Data Engineer - Databricks
Important
Visa: USC and GC only
Location: Manhattan, New York (Onsite)
JD:
Role Overview
- Client is seeking a Lead Data Engineer with deep hands-on expertise in Databricks, SQL, Python, and
- PySpark to design, develop, and optimize scalable data engineering solutions.
- This role offers an opportunity to work on enterprise-scale data platforms supporting analytics, digital
- products, and emerging AI use cases. The candidate will drive Databricks engineering best practices,
- solve challenging technical problems, and provide technical leadership across multiple initiatives.
Key Responsibilities
- Design and develop scalable data pipelines on Databricks, following Medallion Architecture.
- Build and optimize data ingestion, transformation, and orchestration workflows for batch and
- incremental processing.
- Optimize data processing workloads, SQL queries, and compute configurations for performance
- and cost efficiency.
- Ensure data quality, observability, lineage, and reliability across production pipelines, including
- troubleshooting complex issues.
- Provide hands-on technical leadership through solution design, code reviews, reusable
- engineering patterns, and mentoring other engineers.
Required Qualifications
- 6+ years of Data Engineering experience, with substantial hands-on experience delivering
- production solutions on Databricks.
- Strong proficiency in SQL, Python, and PySpark, including distributed data processing and
- performance optimization.
- Strong understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and
- workflow orchestration.
- Familiarity with Git, CI/CD, and production deployment practices.
- Strong troubleshooting, problem-solving, and communication skills.
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience working with AWS cloud services, particularly S3 and cloud-based data integrations.
- Experience with advanced ingestion patterns such as Auto Loader, Structured Streaming, or
- Change Data Capture (CDC).
- Familiarity with Databricks Asset Bundles or Apache Airflow.
- Experience with large-scale media datasets: streaming, clickstream, or advertising data.
Interested in this role?
Review the details, then choose the resume you want to share.
Apply for this job Free for job seekers. Always.Share this job
About the employer