Data Engineer (AWS, Python, Pyspark)

Role Description

This is a Contract/full-time hybrid role for a Data Engineer, based in the London/Edinburgh area, United Kingdom, with the flexibility to work remotely part of the time. The Data Engineer will be responsible for designing, building, and maintaining robust data pipelines and workflows. This includes developing, testing, and optimizing Extract, Transform, Load (ETL) processes, ensuring the integrity and availability of data warehouses, and collaborating with analytics teams to deliver actionable insights from data.

Data Pipeline & Platform Engineering

Cloud Infrastructure (AWS & Terraform)

Data Modelling & Architecture

Collaboration & Delivery

Essential

• 6+ years of hands-on data engineering experience in a professional environment

• Advanced SQL: complex joins, window functions, CTEs, query optimisation

• Proficient Python for data engineering: pandas, PySpark, or similar data processing libraries

• Microservices development, mucroservices using java and springboot framework.

• AWS: hands-on experience with core services (S3, Glue, Lambda, IAM, EC2/ECS)

• Terraform: writing and maintaining infrastructure-as-code modules for AWS

• Data modelling: dimensional modelling, normalisation/denormalisation, schema design

• Version control: Git branching strategies, pull request workflows

Highly Desirable

• Experience with dbt (data build tool) for transformation layer management

• Snowflake: table design, Snowpipe, streams & tasks, performance tuning, RBAC

• Familiarity with orchestration tools: Apache Airflow, AWS Step Functions, or Prefect

• Knowledge of streaming/real-time data: Kafka, Kinesis, or similar

• Exposure to data governance, GDPR compliance, or data privacy engineering