Data Engineer | Streaming & Event-Driven Systems

Location: London (1 day a week onsite)

Type: Permanent

Salary: £50,000 - £75,000 + Bonus, Shares, Pension

Security: BPSS required

Visa Sponsorship: Not available

Build Real-Time Data Platforms That Power Mission-Critical Decisions

We're looking for a Real-Time Data Engineer to design and build high-performance streaming data platforms that process and transform complex sensor, geospatial and operational data in near real-time. Working at the forefront of event-driven architecture, you'll help develop scalable data pipelines that deliver actionable insights from live data feeds across cloud, edge and on-premise environments.

This is an opportunity to work on a modern Azure-based platform where streaming technologies, distributed processing and real-time analytics sit at the heart of the engineering challenge. You'll collaborate with data scientists, software engineers and platform teams to create resilient data products that support advanced analytics, machine learning and operational decision-making.

The Role

As a Data Engineer, you'll design, develop and operate scalable streaming and data processing solutions that ingest, transform and serve large volumes of real-world data. You'll work across both real-time and batch architectures, building systems that are reliable, observable and production-ready.

The platform operates across Microsoft Azure, as well as edge and on-premise deployments, supporting a range of mission-critical applications.

What You'll Do

• Design, build and maintain real-time streaming and event-driven data pipelines
• Develop scalable ingestion, transformation and orchestration frameworks
• Monitor production data services and resolve performance, reliability and data quality issues
• Build reusable platform components, internal libraries and engineering tooling
• Implement automated testing, validation frameworks and deployment processes
• Design data models and storage architectures to support analytics, machine learning and AI-driven applications
• Document data flows, schemas and operational processes

What We're Looking For

Essential

• Strong Python development experience, writing clean, maintainable production code
• Strong SQL skills and experience designing analytical data models
• Experience building streaming, event-driven and real-time data pipelines
• Knowledge of distributed data processing technologies such as Apache Flink, Spark or similar
• Understanding of event streaming architectures, message brokers and stream processing concepts
• Experience working with high-volume, time-series or structured data stores
• Hands-on experience with Azure data and cloud platform services
• Experience with pipeline orchestration and workflow management tools
• Containerisation experience with Docker
• Experience deploying and operating workloads within Kubernetes environments
• Experience developing and integrating APIs
• Strong problem-solving skills and ability to work independently
• 3+ years' experience within a Data Engineering or Data Platform team

Nice to Have

• Experience with technologies such as Kafka, Confluent, Pulsar or similar streaming platforms
• Geospatial, sensor or time-series data experience
• Knowledge of TileDB or other multidimensional data technologies
• Exposure to AI, LLM or RAG-based analytics platforms
• Experience designing systems for edge or intermittently connected environments
• Degree in Computer Science, Data Engineering, Mathematics or equivalent experience

Benefits

• Annual bonus scheme
• Private healthcare and cash plan
• Life assurance (4x salary)
• Share option scheme

Join a growing engineering team building cutting-edge real-time data capabilities, where streaming technologies, event-driven architecture and modern cloud platforms are central to the mission.