Senior Data Engineer

• Role : Senior Data Engineer

• Location : Cape Town [ Mon – Thu Work from Office, Friday remote work]

• Duration : 12 Months renewal

Job Specxc

Key Responsibilities

• Architectural Leadership Define and own end-to-end architecture for EDL/ETL pipelines, streaming platforms, and AI-ready data ecosystems.

• AWS Data Engineering Lead large-scale implementations using AWS services (S3, Glue, EMR, Lambda, Kinesis, Athena).

• Azure Data Integration Drive adoption of Azure Data Factory, Synapse, Databricks, and Event Hub for enterprise workloads.

• Streaming & Real-time Processing Architect resilient, high-throughput streaming pipelines with Kafka, Kinesis, and Event Hub.

• Kafka Topic Consumption Design and implement consumer applications to read, process, and transform data from Kafka topics using Kafka Streams, Spark Structured Streaming, and Flink. Ensure exactly-once semantics, consumer lag monitoring, and fault-tolerant replay mechanisms.

• AI-ready Data Platforms Build feature stores, ML-ready datasets, and automated retraining pipelines to accelerate AI/ML adoption.

• Data Quality & Governance Establish enterprise-wide frameworks for validation, reconciliation, metadata management, and lineage.

• Performance & Scalability Optimize pipelines for petabyte-scale datasets, ensuring cost efficiency and high availability.

• Mentorship & Collaboration Mentor junior engineers, collaborate with data scientists, and align delivery with business priorities.

Required Skills & Experience

• 10+ years of hands-on data engineering experience with AWS (S3, Glue, EMR, Kinesis, Lambda, Athena).

• Strong expertise in Azure Data Services (ADF, Synapse, Databricks, Event Hub).

• Advanced proficiency in Python, PySpark, and SQL for large-scale data workloads.

• Deep knowledge of streaming architectures (Kafka, Kinesis, Event Hub).

• Proven experience in Kafka topic consumption: consumer group management, offset handling, schema evolution, and integration with downstream systems.

• Proven experience in AI/ML workflows: feature engineering, model deployment pipelines, and MLOps.

• Expertise in data lakehouse paradigms (Data Mesh, Data Fabric).

• Strong background in metadata modeling, governance frameworks, and compliance.

• Experience with CI/CD pipelines, Infrastructure-as-Code (Terraform/CloudFormation).

• Track record of leading teams, delivering enterprise-scale solutions, and influencing platform strategy.

Nice-to-Have / Differentiators

• Multi-cloud expertise (AWS + Azure).

• Experience with feature store design (Feast, Databricks Feature Store).

• Familiarity with ML frameworks (TensorFlow, PyTorch, Scikit-learn) and their integration with pipelines.

• Certifications: AWS Certified Solutions Architect – Professional, Azure Data Engineer Expert.

Impact & Growth

This role is pivotal in enabling DNA’s mission to deliver enterprise-scale analytics and AI innovation. You will shape the future of data engineering across AWS and Azure, drive AI adoption, and ensure measurable business outcomes through scalable, governed, and intelligent data delivery.