Data Engineer

Responsibilities Build and maintain ETL/ELT pipelines in Microsoft Fabric Notebooks (Python/PySpark) ingesting data from multiple source systems via REST and SOAP APIs Implement Bronze layer landing (raw data, no transformation), Silver layer cleansing/typing/deduplication, and Gold layer aggression for business analytics Design and build custom API connectors with OAuth 2.0/Bearer token authentication, incremental sync, pagination handling, rate-limit/retry logic and error recovery Configure and manage Fabric Workspaces across Dev/Test/Production environments using Fabric Deployment Pipelines and Git integration for CI/CD Build and maintain Power BI semantic models (star/snowflake schemas) supporting operational reporting and analytics dashboards Implement row-level security (RLS) using Azure AD RBAC, warehouse-level DAX filters, and dynamic RLS platforms Set up monitoring, alerting, and telemetry using Azure Monitor, Log Analytics, and Application insights to track pipeline health and data freshness Manage API credentials and secrets via Azure Key Vault with automated rotation policies Embed within the client's development process, working directly with business stakeholders to gather requirements and co-develop solutions Contribute to data governance: data dictionary creation, lineage tracking, and documentation of transformation logic across all pipeline stages Requirements 5+ years experience in data engineering or a similar role in a commercial environment Hands-on experience with Microsoft Fabric (Lakehouses, Notebooks, Data Pipelines, OneLake, SQL Analytics Endpoints) or equivalent depth in Azure Synapse Analytics Advanced SQL for data transformation, performance tuning, and Delta Lake table management Proficient in Python/PySpark for data processing, API integration, and pipeline automation Proven experience building custom API connectors (REST, SOAP/XML) with OAuth 2.0, pagination, rate limiting, and incremental sync patterns Strong understanding of Medallion Architecture, data lakehouse concepts, and Delta Lake (merge, upsert, soft-delete handling, time travel) Experience with dimensional modelling (star/snowflake schemas) and Power BI semantic model development Working knowledge of Azure services: Key Vault, Azure Monitor, Log Analytics, Azure AD/Entra ID Experience translating business requirements into dimensional models iteratively (not just building pre-designed schemas) Comfort operating autonomously with minimal supervision in a client-embedded model Experience implementing row-level security in Power BI (DAX-based RLS, dynamic security models) Familiarity with CI/CD in a Fabric context: Deployment Pipelines, Git integration, environment promotion workflows Experience with semi-structured data formats (JSON, XML) and handling schema evolution across pipeline layers Preferred Prior work in retail, wholesale, distribution, or fresh produce / supply chain data environments Experience with ERP data extraction and replication tools in a Fabric or Azure context Knowledge of dbt, Power Query, or similar transformation frameworks Relevant Azure or Microsoft certifications (DP-600, DP-203, PL-300) Openness to AI coding assistants (GitHub Copilot, Cursor, Claude) as part of your development workflow What We Offer Competitive salary with mentorship and career growth opportunities Hands-on work with cloud-first technologies (Microsoft Fabric, Azure, Power BI) Long-term embedded role with a single client, providing deep domain expertise and continuity Fast-paced, innovative environment where AI-augmented development is the norm