Job Summary:
We are seeking a highly experienced and visionary Databricks Data Architect with over 14 years in data engineering and architecture, including deep hands-on experience in designing and scaling Lakehouse architectures using Databricks. The ideal candidate will possess deep expertise across data modeling, data governance, real-time and batch processing, and cloud-native analytics using the Databricks platform. You will lead the strategy, design, and implementation of modern data architecture to drive enterprise-wide data initiatives and maximize the value from the Databricks platform.
Key Responsibilities:
- Lead the architecture, design, and implementation of scalable and secure Lakehouse solutions using Databricks and Delta Lake.
- Define and implement data modeling best practices, including medallion architecture (bronze/silver/gold layers).
- Champion data quality and governance frameworks leveraging Databricks Unity Catalog for metadata, lineage, access control, and auditing.
- Architect real-time and batch data ingestion pipelines using Apache Spark Structured Streaming, Auto Loader, and Delta Live Tables (DLT).
- Develop reusable templates, workflows, and libraries for data ingestion, transformation, and consumption across various domains.
- Collaborate with enterprise data governance and security teams to ensure compliance with regulatory and organizational data standards.
- Promote self-service analytics and data democratization by enabling business users through Databricks SQL and Power BI/Tableau integrations.
- Partner with Data Scientists and ML Engineers to enable ML workflows using MLflow, Feature Store, and Databricks Model Serving.
- Provide architectural leadership for enterprise data platforms, including performance optimization, cost governance, and CI/CD automation in Databricks.
- Define and drive the adoption of DevOps/MLOps best practices on Databricks using Databricks Repos, Git, Jobs, and Terraform.
- Mentor and lead engineering teams on modern data platform practices, Spark performance tuning, and efficient Delta Lake optimizations (Z-ordering, OPTIMIZE, VACUUM, etc.).
Technical Skills:
- 10+ years in Data Warehousing, Data Architecture, and Enterprise ETL design.
- 5+ years hands-on experience with Databricks on Azure/AWS/GCP, including advanced Apache Spark and Delta Lake.
- Strong command of SQL, PySpark, and Spark SQL for large-scale data transformation.
- Proficiency with Databricks Unity Catalog, Delta Live Tables, Autoloader, DBFS, Jobs, and Workflows.
- Hands-on experience with Databricks SQL and integration with BI tools (Power BI, Tableau, etc.).
- Experience implementing CI/CD on Databricks, using tools like Git, Azure DevOps, Terraform, and Databricks Repos.
- Proficient with streaming architecture using Spark Structured Streaming, Kafka, or Event Hubs/Kinesis.
- Understanding of ML lifecycle management with MLflow, and experience in deploying MLOps solutions on Databricks.
- Familiarity with cloud object stores (e.g., AWS S3, Azure Data Lake Gen2) and data lake architectures.
- Exposure to data cataloging and metadata management using Unity Catalog or third-party tools.
- Knowledge of orchestration tools like Airflow, Databricks Workflows, or Azure Data Factory.
- Experience with Docker/Kubernetes for containerization (optional, for cross-platform knowledge).
Preferred Certifications (a plus):
- Databricks Certified Data Engineer Associate/Professional
- Databricks Certified Lakehouse Architect
- Microsoft Certified: Azure Data Engineer / Azure Solutions Architect
- AWS Certified Data Analytics – Specialty
- Google Professional Data Engineer
