Lead Quality Engineer – AI Delivery

The Role

We are looking for an experienced Lead Quality Engineer – AI Delivery on behalf of our client, a global organization specializing in insurance, risk management, and professional services, to lead Quality Engineering across multiple AI delivery pods building enterprise-scale AI solutions.

Our client's engineering teams already operate within an established shift-left Secure SDLC, with core engineering, security, code quality, and pipeline controls embedded into delivery. This role builds on that foundation, ensuring each AI solution has an appropriate risk-based quality strategy, with particular focus on GenAI evaluation, end-to-end quality, UAT readiness, and release confidence.

You will work closely with Product Managers, Product Owners, Engineering Leads, Architects, and developers, combining strategic quality leadership with hands-on technical capability.

We will count on you to:

• Lead Quality Engineering across multiple AI delivery pods and establish consistent, risk-based quality standards.

• Partner with Product and Engineering to define test strategy, acceptance criteria, UAT approach, quality risks, and release-readiness criteria.

• Ensure appropriate coverage across functional, integration, end-to-end, non-functional, AI-specific, and business acceptance testing.

• Define testing and evaluation approaches for LLM, RAG, agentic, and AI-enabled solutions, including grounding, hallucination risk, regression, tool calling, workflow behaviour, and non-deterministic outputs.

• Establish appropriate AI evaluation datasets, test harnesses, baselines, thresholds, and regression approaches.

• Guide and, where needed, implement reusable automation across web, APIs, backend services, integrations, asynchronous workflows, and AI components.

• Ensure products are ready for UAT, with appropriate test evidence, environments, test data, defect status, entry/exit criteria, and known risks.

• Provide Pod Leads and programme leadership with clear quality metrics, risks, defect trends, AI evaluation results, UAT readiness, and release-readiness reporting.

• Conduct quality reviews across pods and identify gaps in testing, testability, observability, or quality evidence.

• Use production incidents, escaped defects, telemetry, and user feedback to continuously improve quality approaches.

• Coach engineers and delivery teams to strengthen shared ownership of quality across the lifecycle.

What you need to have

• Proven experience leading Quality Engineering for enterprise-scale AI / GenAI solutions in production.

• Hands-on experience testing LLM, RAG, agentic, workflow-based, or AI-enabled applications.

• Strong experience defining end-to-end test strategies, UAT approaches, quality gates, and release-readiness criteria.

• Experience working across multiple delivery teams or pods in complex enterprise environments.

• Strong software engineering and automation skills in TypeScript / JavaScript and/or Python.

• Experience with modern automation frameworks such as Playwright, Cypress, Selenium, Jest, Mocha, or PyTest.

• Strong experience testing web applications, APIs, backend services, integrations, and distributed systems.

• Experience with AI evaluation frameworks, regression suites, test harnesses, and representative evaluation datasets.

• Good understanding of CI/CD, cloud-native systems, observability, and Secure SDLC practices.

• Strong stakeholder management skills and the ability to translate quality evidence into clear risks, metrics, and recommendations for leadership.

What makes you stand out

• Experience with LangChain, LangSmith, Mastra, or similar AI engineering and evaluation tooling.

• Experience testing complex RAG, multi-agent, or tool-calling solutions.

• Experience with performance, resilience, accessibility, or security testing.

• Experience using production telemetry and incidents to improve test and evaluation strategies.

• Ability to balance strong quality controls with pragmatic, fast-moving AI delivery.

Technology context

Typical technologies include:

React, TypeScript, Node.js / NestJS, Python / FastAPI, REST APIs, async workflows, AWS / Azure, GitHub Actions / GitLab CI / Azure DevOps, LLM APIs, RAG, agentic workflows, LangChain, LangSmith, and Mastra.