Senior Databricks Developer

We are partnering with an enterprise client to accelerate their modern lakehouse roadmap and advance large-scale distributed analytics. We are seeking an experienced Senior Databricks Developer to represent our organisation and take technical ownership of designing, building, and optimising high-performance data processing pipelines and lakehouse storage layers.

In this role, you will bridge massive data transformations and analytics enablement within our client’s cloud ecosystem. You will be instrumental in designing resilient Medallion architectures (Bronze/Silver/Gold), deploying production-grade data pipelines with PySpark and Delta Lake, implementing Unity Catalog governance, and optimising cluster compute efficiency.

🌏 Visa & Sponsorship Options

As the employer of record, we provide full visa and migration support for qualified engineering talent deployed to our clients:

• 482 On-Hire Sponsorship Transfers: Fully supported for qualified candidates currently in Australia on an existing 482 visa looking to transfer sponsorship to work with our clients.
• New 482 Visa Sponsorship: Available for qualified candidates meeting the commercial experience and technical requirements.
• Temporary & Working Visa Holders: Open to all working visa holders seeking a direct pathway to employer sponsorship.

Core Responsibilities

• Lakehouse Pipeline Engineering: Design, build, and maintain production-grade batch and real-time streaming data pipelines using Databricks, Apache Spark / PySpark, and Delta Lake.
• Architecture & Medallion Design: Implement scalable Bronze, Silver, and Gold architectural layers, ensuring optimal data partitioning, liquid clustering, and ACID transactional reliability.
• Governance & Access Controls: Enforce enterprise-grade security, data lineage, and fine-grained access policies using Databricks Unity Catalog.
• Performance Tuning & Cluster Optimisation: Optimise Spark execution plans, manage memory configurations, eliminate data skew, and fine-tune warehouse/cluster compute to drive cost efficiency.
• DataOps & CI/CD Automation: Develop automated CI/CD deployment pipelines for Databricks notebooks, modular libraries, and workflows using Databricks Asset Bundles (DABs), Azure DevOps, or GitHub Actions.
• Data Modelling & Analytics Support: Collaborate directly with client data architects, analytics engineers, and machine learning specialists to deliver clean, curated data models for downstream BI and AI consumption.

Selection Criteria

• Databricks & Spark Mastery: Extensive commercial experience engineering distributed data platforms using Databricks and Apache Spark / PySpark.
• Storage & Medallion Architecture: Deep hands-on knowledge of Delta Lake, Delta Live Tables (DLT), and Medallion architectural patterns.
• Programming & SQL: Advanced proficiency in Python / PySpark and expert-level SQL for complex data manipulation and performance tuning.
• Data Governance: Practical experience configuring and managing Unity Catalog for centralized access control and lineage tracking.
• Location Requirements: Currently residing in Australia with valid work rights or eligibility for 482 visa sponsorship/transfer.

Preferred Qualifications (Nice to Have)

• Databricks Certified Data Engineer Professional credential.
• Experience integrating Databricks with dbt (data build tool) or workflow orchestrators (Airflow).
• Cloud platform experience across Azure (Azure Databricks) or AWS.
• Exposure to streaming architectures using Apache Kafka, Event Hubs, or Spark Structured Streaming.