AI Data Engineer

We are hiring a Fabric data engineer to build and run production data pipelines on Microsoft Fabric. You will own ingestion through to curated, analytics-ready data in OneLake, working alongside analytics engineers and business stakeholders. This is a hands-on build role, not an advisory one. Expect to spend most of your time in Spark notebooks, Data Factory pipelines and SQL.

Key responsibilities

• Build and maintain ingestion pipelines in Data Factory and Dataflows Gen2 from source systems including SAP, Salesforce, flat files and APIs
• Design and implement medallion architecture (bronze, silver, gold) in Fabric Lakehouse using Delta Lake
• Write and optimise PySpark and Spark SQL transformations in Fabric notebooks
• Model curated data in Fabric Warehouse and expose it through semantic models for Power BI Direct Lake consumption
• Configure OneLake shortcuts to integrate data residing in ADLS, S3 or other external stores without duplication
• Implement CI/CD using Fabric deployment pipelines and Git integration across dev, test and production workspaces
• Monitor capacity consumption through the Capacity Metrics app and tune workloads to prevent throttling
• Apply data quality checks, error handling and pipeline observability across all production workloads
• Work directly with analysts and business owners to translate requirements into data products

Required qualifications

• 5 to 7 years in data engineering, of which at least 1 to 2 years is hands-on production work in Microsoft Fabric
• Strong PySpark and Spark SQL; able to read an execution plan and explain why a job is slow
• Advanced SQL, including window functions, performance tuning and incremental load patterns
• Production experience with Delta Lake and medallion architecture
• Data Factory or Azure Data Factory pipeline development, including parameterisation and error handling
• Working knowledge of Power BI semantic models, and specifically where Direct Lake falls back to DirectQuery
• Python for data engineering tasks beyond notebooks (utilities, testing, automation)
• Source control and CI/CD discipline; comfortable with Git-based workspace promotion
• Able to explain the lakehouse versus warehouse decision and defend it against a specific workload

Preferred qualifications

• DP-700 (Fabric Data Engineer) or DP-600 (Fabric Analytics Engineer) certification
• Prior migration experience from Azure Synapse, Databricks or a legacy warehouse into Fabric
• Fabric Real-Time Intelligence: eventstreams, KQL databases, eventhouse
• Microsoft Purview for lineage, classification and governance
• Capacity administration and FinOps exposure: F-SKU sizing, smoothing and bursting behaviour, chargeback
• Domain exposure in enterprise source systems, particularly SAP or Salesforce data structures
• Experience in a consulting or client-facing delivery model