Platform Engineer (AI / LLM Operations)
- Salary
- £85K–£95K
- Hiring from
- United Kingdom
- Work type
- Remote
- Posted
627,434 remote jobs, straight from company career pages
100% free · New jobs every minute
Show job descriptionHide job description
LLMOps Engineer – Production AI Monitoring & Evaluation
£85,000–£95,000
Fully Remote (UK) Permanent
**No visa sponsorship available**
My client is a substantial, private equity-backed UK business investing heavily in AI and automation as part of a major technology transformation.
They're looking for a hands-on LLMOps Engineer with proven experience building and operating monitoring and evaluation pipelines for LLM applications in live production environments.
This isn't a traditional Machine Learning Engineer, MLOps or AI application development position.
The focus is on ensuring production LLM and agentic AI systems deliver reliable, accurate and measurable results.
You'll be working on automated LLM output evaluation, hallucination and groundedness checks, agent tracing, regression testing, observability and production reliability.
Essential: Recent, hands-on experience personally building production LLM evaluation and monitoring pipelines. Experience limited to traditional ML model monitoring or building RAG applications without ongoing evaluation will not be sufficient.
Strong Python engineering skills are required, with AWS experience highly desirable.
The Role
You'll take ownership of monitoring, troubleshooting and improving live AI systems, working across LLMs, agentic workflows, retrieval pipelines and cloud infrastructure.
A key focus will be ensuring AI applications don't just remain available, but continue to produce accurate, consistent and useful results.
Key responsibilities:
- Build and maintain monitoring, observability and evaluation pipelines for production AI and LLM applications.
- Develop automated evaluation frameworks to measure response accuracy, retrieval quality, hallucinations, consistency and performance.
- Monitor and troubleshoot agentic AI workflows, including tool calls, orchestration failures, context handling and multi-step execution.
- Identify and investigate changes in model behaviour, output quality and performance over time.
- Build dashboards, alerts and quality metrics to help the team understand how AI applications are performing.
- Support model upgrades, prompt changes, version management and provider transitions, ensuring changes don't negatively impact quality.
- Investigate production incidents across AI applications, data pipelines, APIs and supporting infrastructure.
- Work closely with AI Engineers and the wider technology team to improve reliability, performance and operational processes.
What They're Looking For
We're particularly interested in candidates who can demonstrate real commercial experience operating AI systems in production, rather than solely building prototypes or proofs of concept.
Essential experience:
- Hands-on experience building or maintaining monitoring and evaluation pipelines for production AI, ML or LLM applications.
- Strong Python development and debugging skills.
- Experience with observability, logging, tracing, alerting and troubleshooting production systems.
- Understanding of LLM behaviour, including prompting, context management, output consistency and evaluation.
- Experience deploying, monitoring or supporting applications in AWS.
- Familiarity with CI/CD, automated testing and production software operations.
- Strong problem-solving skills and the ability to investigate issues across multiple interconnected systems.
Particularly interesting experience:
- LLM evaluation frameworks, automated regression testing and hallucination detection.
- Agentic AI frameworks such as LangGraph, LangChain or similar.
- LLM observability tools such as Langfuse, LangSmith, Helicone or equivalent.
- RAG pipelines, retrieval evaluation and vector databases.
- MLflow, Databricks, SageMaker or Bedrock.
- Model routing, fallback strategies, provider management and inference monitoring.
- Kubernetes, Docker and Terraform.
You don't need experience with every tool listed. We're more interested in what you've personally built, monitored and improved in production.
Why Consider It?
This is an opportunity to join a business making a significant investment in applied AI, with the chance to influence how its AI systems are monitored, evaluated and operated.
You'll be working on real production applications, collaborating with an established engineering team and helping shape the standards for AI reliability and quality across the organisation.
For a confidential discussion, please apply or get in touch with Better Placed Technology.
