Senior Data Engineer Consultant – Data Quality and Modern Cloud Platform

Function : Data & Digital Consulting Location : South Africa Hybrid / client-site as required Reports to : Data Engineering or Consulting Practice Lead About Keyrus Keyrus is an internationally recognised specialist in Data and Digital and a trusted partner to organisations across industries. We deliver practical business solutions using reputable, modern and scalable technologies. Our purpose is to help clients improve performance through transformation enabled by data. We are passionate about innovation, teamwork and collective success, driven by exceptional individuals who combine technical depth with sound consulting judgement. The role The Senior Data Engineer Consultant designs, builds and operationalises secure, scalable and governed modern data platforms for Keyrus clients. The role converts business and data requirements into reliable ingestion, transformation, data-quality, storage and serving solutions that support operational reporting, analytics, migration and responsible artificial intelligence use cases. The consultant works with data architects, business analysts, data-quality specialists, governance teams and client stakeholders. The role provides technical leadership, contributes to solution architecture and pre-sales, mentors other engineers, and ensures that solutions are tested, observable, documented, auditable and transferable to client teams. Key outcomes Production-grade cloud data platforms and pipelines that are secure, scalable, reliable and cost-conscious. Trusted, curated and reporting-ready data products with measurable quality and clear ownership. Repeatable data profiling, cleansing, remediation, reconciliation and monitoring capabilities. Traceable metadata, lineage, transformation logic and evidence supporting governance and formal acceptance. AI-ready data foundations and responsible, approved use of AI-assisted engineering tools. Operational runbooks, knowledge transfer and sustainable transition into business-as-usual support. Responsibilities 1. Data discovery and solution design Assess client systems, interfaces, datasets, integration patterns, business processes and reporting dependencies. Produce current-state data-flow, integration, dependency and lineage documentation. Translate business requirements into technical designs, source-to-target mappings, data contracts, acceptance criteria and delivery backlogs. Contribute to target-state data lake, lakehouse, warehouse and integration architecture. Identify technical risks, assumptions, dependencies and constraints and maintain appropriate delivery evidence. Explain complex architectures, trade-offs and recommendations clearly to technical and non-technical stakeholders. 2. Data platform engineering Design and implement reusable ingestion, transformation, orchestration and serving frameworks. Integrate relational databases, APIs, files, SaaS platforms and event or streaming sources where appropriate. Implement batch, incremental, change-data-capture and synchronisation patterns with reconciliation and recoverability. Build curated, business-ready data products using suitable dimensional, lakehouse or domain-oriented modelling patterns. Apply modular design, peer review, automated testing, version control, CI/CD and controlled environment promotion. Optimise SQL, Spark processing, pipeline execution and cloud storage for performance, reliability and cost. 3. Data quality and remediation Profile complex datasets and establish baselines for completeness, validity, consistency, uniqueness, timeliness and referential integrity. Identify and quantify duplicates, anomalies, gaps, conflicting records and cross-system misalignment. Implement validation, standardisation, enrichment, entity matching, deduplication and survivorship rules. Build repeatable cleansing and remediation pipelines with version-controlled rules and auditable before-and-after evidence. Develop exception handling, reconciliation, control totals, remediation workflows and escalation mechanisms. Implement data-quality scorecards, monitoring, alerts and preventative controls, including correction at source where practical. Work with data owners and stewards to validate rules, resolve exceptions and obtain business acceptance. 4. Metadata, governance, security and privacy Implement technical metadata, data catalogue, classification and end-to-end lineage capabilities. Maintain source-to-target mappings, transformation specifications, data standards and traceable change records. Apply role-based access, least privilege, secrets management, encryption, masking or tokenisation as appropriate. Implement logging, monitoring, alerting and evidence retention for data access, pipeline execution and data changes. Handle personal, banking, customer and other sensitive data in accordance with POPIA, contractual commitments and client-approved security, privacy and residency controls. Align technical controls with the client's established governance programme, decision rights and stewardship model. 5. Operational and migration enablement Implement pipeline observability, retry, recovery, backup and operational support procedures. Define and monitor data service levels, operational controls, ownership and escalation paths. Prepare deployment guides, support procedures, operational runbooks and maintainable technical documentation. Support migration readiness through profiling, reconciliation, exception management, acceptance gates and sign-off evidence. Conduct structured knowledge transfer, training and transition into client business-as-usual teams. 6. Responsible AI and continuous learning Design governed data foundations that can support advanced analytics and AI use cases, including curated training or retrieval datasets where approved and justified. Evaluate appropriate uses of AI and machine learning for profiling, anomaly detection, classification, entity resolution, metadata generation and engineering productivity. Use only Keyrus- and client-approved AI tools, models and processing environments. Ensure AI-assisted code, documentation, mappings and analysis are tested, reviewed and subject to human accountability before use. Never submit client data, credentials, proprietary code or confidential information to unapproved public AI services, and never permit client data to be used for model training without written authorisation. Understand and communicate AI risks including hallucination, bias, privacy, intellectual property, explainability, data residency and vendor lock-in. Maintain current knowledge of cloud data platforms, data-engineering practices, AI-assisted development and responsible-AI controls, sharing learning through demonstrations, standards, reusable assets and mentoring. 7. Consulting, leadership and commercial contribution Lead or supervise data-engineering workstreams from discovery through production handover and post-implementation support. Plan and estimate work, manage priorities, monitor delivery quality and escalate risks early. Define evidence-based acceptance criteria and support formal client review and sign-off. Mentor engineers, conduct design and code reviews, and promote consistent engineering standards. Contribute to proposals, solution demonstrations, client bids and technical pre-sales activities. Translate technical capabilities into clear business benefits, costs, risks and implementation choices. Build trusted stakeholder relationships and identify legitimate improvement opportunities through high-quality delivery. Role requirements Education and professional standing A relevant degree or diploma in computer science, information systems, engineering, data science or a related discipline; equivalent demonstrable professional experience will also be considered. Relevant Microsoft, Databricks, cloud architecture, security or data-engineering certifications are advantageous. Essential experience Typically seven or more years of relevant data engineering, data integration or data-platform experience, including at least three years delivering production-grade cloud data solutions. Advanced SQL skills and strong practical capability in Python or PySpark. Hands-on experience with Azure Data Factory or Fabric Data Factory, Azure Data Lake Storage and Azure Databricks, or closely comparable cloud technologies. Strong understanding of data lake, lakehouse and data-warehouse architecture and modelling patterns. Practical delivery experience across ingestion, transformation, orchestration, incremental loading and source-system synchronisation. Experience profiling, cleansing, reconciling and remediating large or complex datasets. Working knowledge of entity matching, duplicate management, survivorship, exception handling and data-quality controls. Experience implementing or integrating metadata, catalogue and lineage capabilities. Knowledge of cloud security, identity and access management, secrets, encryption, monitoring and audit logging. Experience with Git, automated data testing, CI/CD, deployment controls and environment management. Evidence of technical leadership, client engagement, documentation and operational handover. Desirable experience Microsoft Fabric, Delta Lake and medallion or lakehouse architecture. Microsoft Purview or comparable data catalogue, governance and lineage technology. Data-quality frameworks such as Great Expectations, Soda, dbt tests or equivalent. Infrastructure as code using Bicep, Terraform or equivalent. Power BI semantic models, KPI definition and reporting-ready data products. SQL Server, T-SQL, SSIS, SSAS, SSRS and PowerShell within legacy or hybrid environments. API integration, event-driven architecture, streaming data or message-based processing. Large-scale migration readiness, data reconciliation and cutover support. Customer, contract, product, pricing, billing, banking, collections or other regulated data domains. Applied understanding of generative AI, retrieval-augmented generation, embeddings, vector stores, ML lifecycle concepts and AI governance. Core capabilities and behaviours Analytical judgement - Finds root causes, tests assumptions and chooses proportionate, evidence-based solutions. Engineering discipline - Produces maintainable, tested, secure and observable solutions rather than one-off scripts. Consulting communication - Explains technical matters clearly and adapts communication to executives, business owners and engineers. Delivery ownership - Plans realistically, manages priorities, anticipates risk and follows work through to acceptance and handover. Collaboration - Works constructively across architecture, governance, business analysis, analytics and client teams. Commercial awareness - Understands value, cost, licensing, operational sustainability and the implications of technical choices. Learning mindset - Keeps skills current, evaluates new technology critically and turns useful learning into repeatable practice. Professional integrity - Protects client information, challenges unsafe practices and is transparent about limitations and risks. Role boundaries This is a senior data engineering role within a multidisciplinary delivery team. The consultant is expected to collaborate closely with data architecture, data quality, governance, business analysis, analytics and project leadership specialists. The role may lead a data-engineering workstream but is not intended to replace every specialist discipline on a complex enterprise data programme. Success measures Solutions meet agreed functional, data-quality, security, performance and acceptance requirements. Pipelines and data products operate reliably within agreed service levels and cost parameters. Data rules, changes, lineage and remediation outcomes are traceable and auditable. Client teams can support and extend delivered solutions using the documentation and knowledge transferred. Technical risks and delivery constraints are identified early and managed transparently. Reusable assets and lessons learned improve future Keyrus delivery quality and efficiency. Employment equity Keyrus is committed to employment equity and to creating an inclusive workplace. Applications from suitably qualified candidates are welcomed, with due regard to the achievement of equity in respect of race, gender and disability. Application information Candidates should provide a curriculum vitae highlighting relevant cloud data-platform implementations, data-quality or migration programmes, technical leadership responsibilities, and the scale and business outcomes of solutions delivered.