Senior Consultant | Databricks + Pyspark | Mumbai | Engineering | Data Modernization & Migration

Job Title: Databricks - Sr. Consultant

Role Overview: As a Senior Consultant, you will lead the design and optimization of high-performance data processing frameworks on Azure Databricks. You will be responsible for advanced Spark tuning, implementing robust data governance through Unity Catalog, and driving the transition toward a modern Medallion architecture.

Experience:

• 5–7 years of experience in data engineering, with deep expertise in Azure Databricks and Apache Spark.
• Advanced experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold).
• Proven track record in Spark Performance Tuning (handling data skew, shuffle tuning, etc.).
• Strong experience with Unity Catalog for data governance and security.

Key Responsibilities: Technical Skills:

• Design and implement scalable Medallion Architectures using Delta Live Tables (DLT) and Databricks.
• Lead the optimization of complex Spark workloads to improve execution speed and reduce cloud compute costs.
• Implement enterprise-grade data governance and access control using Unity Catalog.
• Architect automated CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs) or Terraform.
• Design sophisticated data integration patterns between Databricks, Azure Synapse, and various RDBMS.
• Act as a technical mentor to junior engineers, conducting code reviews and enforcing best practices.

Soft Skills:

• Strong leadership and ability to drive technical delivery.
• Advanced problem-solving skills for complex architectural challenges.
• Excellent communication skills for interacting with stakeholders and technical leads.

Preferred Qualifications:

• Bachelor’s or Master’s degree in a technical field.
• Databricks Certified Data Engineer Professional is highly desirable.
• Experience with DataOps and Infrastructure as Code (Terraform).

Job Details:

• Location: Mumbai (Hybrid)