Senior AI QA Engineer

Title: Senior QA Automation Engineer, AI & Agentic Systems | Sovereign AI

Location: Abu Dhabi

About the role:

We're hiring a Senior QA Automation Engineer for one of the leading players in Sovereign AI, a UAE-based company with frontier-model access and agentic AI products live across critical sectors including large-scale construction, government, healthcare, and defense.

This is not traditional QA. You'll own quality for AI and agentic systems, defining what "good" looks like when a system reasons, calls tools, and can be wrong in subtle ways. If you've figured out how to test something whose output is different every time, this is for you.

What you'll do:

• Test agentic systems end to end: tool-use and trajectory evaluation, grounding and citation verification, honest-failure and refusal calibration, guardrail and adversarial testing (prompt injection, jailbreaks)
• Build evaluation for non-deterministic AI: scoring for hallucination, consistency, and accuracy against gold datasets, including LLM-as-judge calibrated against human judgement
• Own prompt and model regression suites that catch quality drift before it ships
• Run continuous evaluation on live production traffic, with quality monitoring, alerting, and incident response
• Cover the fundamentals too: integration and API testing, secure gateway validation (RBAC, PII), and performance/load testing

What we're looking for:

• 5+ years in QA or quality engineering, with 2+ years testing ML, LLM, or AI agent systems
• Hands-on with LLM/agent evaluation frameworks (DeepEval, TruLens, RAGAS, or custom Python evaluators) and LLM-as-judge
• Agent observability and tracing (LangSmith, Langfuse, OpenTelemetry)
• Expert Python (Pytest, custom harnesses), plus Selenium/Playwright and API testing

The setup:

• Abu Dhabi hybrid, contract-based
• Onboarding ASAP
• World-class environment: elite talent from FAANG and frontier AI labs