Senior Databricks Engineer

Required Qualifications & Experience

• Bachelor's degree in computer science, Information Technology, Engineering, or a related discipline, with 6-10 years of experience in Data Engineering, Application Development, System Integration, or Enterprise Data Warehouse projects.
• Strong hands-on experience designing and implementing modern data platforms on Azure Databricks, leveraging Delta Lake, Delta Live Tables (DLT), and Medallion Architecture (Bronze, Silver, Gold) to build scalable and reliable data pipelines.
• Proven expertise in developing and optimizing batch and streaming data pipelines using PySpark, Spark SQL, and Python, with a strong focus on performance tuning, scalability, and cost optimization.
• Extensive experience in Databricks job performance optimization, including query tuning, partitioning strategies, data skew mitigation, caching, cluster sizing, workload optimization, and Spark execution plan analysis.
• Strong proficiency in SQL programming and data modeling concepts, including relational, dimensional, and lakehouse data architectures.
• Deep understanding of Delta Lake features such as ACID transactions, schema evolution, time travel, change data capture (CDC), and data quality enforcement.
• Experience implementing data quality, monitoring, and orchestration frameworks using Delta Live Tables expectations, Databricks Workflows, and Azure-native services.
• Strong exposure to the full Software Development Life Cycle (SDLC), including design, development, testing, CI/CD, deployment, and production support.
• Solid understanding of Lakehouse architecture, cloud-based analytics platforms, data governance, and data engineering best practices.
• Knowledge of real-time data processing, data integration patterns, and enterprise-scale data migration initiatives is a strong advantage.
• Demonstrated ability to stay current with evolving cloud and big data technologies and contribute effectively to architecture discussions, technology strategy, and platform modernization initiatives.
• Self-motivated, proactive, and capable of working independently in a fast-paced environment while collaborating with globally distributed teams.
• Strong analytical, problem-solving, communication, and stakeholder management skills.