Lead Databricks Architect

Lead Databricks Architect

Location: Indianapolis, IN — Hybrid, 3 days onsite/week

Industry: Pharmaceutical

Contract: Long Term Contract

Interview/Selection: Deep technical background review — only highly qualified candidates

Role Overview

We are seeking a Lead Databricks Architect to lead the architecture, modernization, governance, and platform strategy for a large-scale Databricks environment running on AWS.

This is a true enterprise architecture and platform leadership role, not a general Databricks Developer or Data Engineer position.

The ideal candidate must have multiple enterprise-scale Databricks implementations/integrations and hands-on experience designing and migrating complex AWS data platforms into a Databricks Lakehouse architecture.

The candidate will lead architecture across multiple Databricks workspaces, establish platform standards, define governance and security models, and provide hands-on technical leadership throughout migration and implementation.

Key Responsibilities

• Lead architecture and modernization of an enterprise Databricks platform on AWS.
• Design and implement multi-workspace Databricks architectures.
• Establish standards for Unity Catalog, security, governance, RBAC/ABAC, data sharing, and access controls.
• Lead migrations from AWS Glue, Aurora PostgreSQL/PostgreSQL, and existing AWS data lakes to Databricks Lakehouse.
• Design scalable raw-to-refine/medallion data architectures using Databricks and Unity Catalog.
• Build metadata-driven and configuration-based ingestion pipelines.
• Develop and maintain Databricks Lakeflow Declarative Pipelines (formerly DLT).
• Implement CDC, materialized views, schema evolution, and incremental data processing.
• Integrate Databricks with Veeva Vault/Veeva Direct Data API (DDAPI) and related connectors.
• Develop advanced Databricks SQL, Delta Lake, Spark/PySpark solutions.
• Build reusable architectural patterns, accelerators, templates, and engineering standards.
• Implement Terraform-based Infrastructure as Code for Databricks workspaces, Unity Catalog, service principals, IAM, and permissions.
• Build/manage AWS infrastructure using CloudFormation, including S3, Glue, Lambda, Step Functions, EventBridge, and DynamoDB.
• Establish enterprise CI/CD and automated deployment processes, including GitHub Actions.
• Drive platform governance, compliance, security, privacy, auditability, and data-quality standards.
• Troubleshoot complex pipeline, schema evolution, integration, and platform issues.
• Partner with data engineering, security, infrastructure, architecture, regulatory, and business teams.
• Provide technical leadership and mentorship to engineering teams.
• Document architecture decisions, standards, deployment patterns, and operational procedures.

Must-Have Skills

Candidates must have strong experience in the following:

• Multiple enterprise-scale Databricks implementations/integrations
• Databricks Architecture on AWS
• Multi-workspace Databricks environments
• AWS Glue → Databricks migrations
• AWS Data Lake → Databricks Lakehouse modernization
• Aurora PostgreSQL/PostgreSQL → Databricks migration
• Unity Catalog
• Delta Lake
• Databricks Workflows
• Databricks SQL / SQL Warehouses
• Apache Spark / PySpark
• Lakeflow Declarative Pipelines / DLT
• AWS S3, IAM, networking, and security
• Terraform / Infrastructure as Code
• CI/CD for enterprise data platforms
• Platform governance, security, and architectural standards
• Enterprise migration strategy and reference architecture

Ideal Candidate Profile

The strongest candidate will be someone who has already architected and delivered multiple enterprise Databricks implementations on AWS, rather than someone who has primarily worked as a Databricks developer.

Top priority: Proven enterprise Databricks architecture + multiple implementations + AWS migration experience.

Important: Only submit candidates who can clearly demonstrate these experiences on their resume. The hiring manager will conduct a deep technical review of their background.

Submission Details

Please provide the following with each candidate submission:

• Full Name
• Current Location
• Phone Number
• Email Address
• Availability / Notice Period
• Expected Rate
• Work Authorization
• LinkedIn Profile
• Total Years of Experience
• Years of Databricks Experience
• Years of AWS Experience
• Number of Enterprise Databricks Implementations Led
• Databricks Architecture / Platform Leadership Experience
• AWS Glue → Databricks Migration Experience
• Aurora PostgreSQL/PostgreSQL → Databricks Migration Experience
• AWS Data Lake → Databricks Lakehouse Migration Experience
• Unity Catalog / Multi-Workspace Experience
• Terraform / IaC / CI-CD Experience
• Pharma / Healthcare / Regulated Industry Experience
• Veeva Vault / Veeva DDAPI Experience, if applicable
• Indianapolis, IN Hybrid Availability — 3 Days Onsite/Week

Note: Please submit only candidates who have multiple enterprise-level Databricks implementations/integrations and can demonstrate hands-on Databricks architecture and AWS migration experience.