Inference QA Engineer

Inference QA Engineer Applied Computing was founded in 2024 to build Orbital, a physics-informed foundation model for energy operations. We’re live across oil and gas, refineries, and petrochemicals, working towards our mission: sustainable abundance for a growing planet. The hydrocarbon industry keeps the world running. But its complexity has left operators tied to legacy systems, making critical decisions on less than 10% of available data. We built Orbital to change that. It’s a foundation model built specifically for energy that lets companies use AI at scale, harnessing all of their operational data and optimising in real time for any metric. Decisions get faster, operations get safer, and carbon intensity falls. We’ve raised over $32 million, including one of the largest seed rounds for an AI company in the UK. We’re just getting started About the role We build Orbital, an agentic LLM platform that answers hard operational and analytical questions for enterprise clients in energy and heavy industry. It routes questions across a graph of tools (data retrieval, SQL-generating coders, statistical analysis, anomaly detection, forecasting, RAG) and synthesises answers that engineers act on. When it is right, it compresses hours of analyst work into seconds. When it is subtly wrong, quiet, or slow, that costs us trust with technical customers who check our numbers. We are hiring a senior engineer to own inference quality end to end: the person who watches every deployment, decides what “working” means, builds the systems that prove it, and catches regressions before a client does. This is a founding-level QA function. You will define the discipline, not just execute a checklist. What you’ll own Evaluation frameworks. We test on a tiered model: T1 data retrieval, T2 statistical analysis, T3 open-ended inference and root-cause, each with its own pass threshold. You will own and extend this framework, design test sets with real ground-truth rubrics (expected answer, pass criteria, known failure modes, source tables), and set the bar for what ships. The test harness and its runs. We run prompts against live model endpoints at scale (tens of thousands of runs), on schedules and on demand, measuring consistency across repeated iter-ations, ground-truth match, structural correctness of generated queries, and paraphrase robust-ness. You will run these executions, keep the harness healthy, and turn raw runs into a verdict. Model and agent monitoring systems. You will build the observability layer for a multi-step agentic system: not just “did the endpoint return 200” but did the planner route to the right tool,did the tool actually execute, did the agent loop terminate, and is the final answer grounded. Concrete failure modes we already fight and want caught automatically: • Silent empty responses — the reasoning trace renders but the answer stays empty, while every layer reports success. • False refusals — a tool crashes or times out, returns nothing, and the model fabricates “I don’t have that data” instead of erroring loudly. • Tool-routing misses — the planner should have fired a tool and didn’t, or double-counts raw identifiers instead of canonical ones. • Latency and non-termination — multi-tool agent loops that blow past timeout budgets. • Paraphrase and run-to-run instability — the same question three ways, or the same prompt three times, giving materially different answers. Ground truth and the feedback loop. You will work with subject-matter experts to extract vetted ground truths from deployment feedback, feed them back into the eval sets, and close the loop so every confirmed defect becomes a permanent regression guard. Regression discipline. After every fix ships, you devise tier-appropriate tests biased at the failure mode plus regression guards, run them against the deployment, and report pass-rate against threshold and whether the original failure recurred. You are the gate. What we’re looking for • 5+ years in software, ML, data, or QA engineering, with real ownership of a quality-critical system. • Strong Python. Comfortable in a FastAPI + Postgres + Docker world, reading logs across services and tracing a request through a distributed pipeline. • Fluency with LLM behaviour: prompting, tool/function calling, agentic loops, RAG, and the ways they fail (hallucination, refusal, silent truncation, non-determinism). • Experience designing evaluation: LLM-as-judge, deterministic checks, ground-truth scoring, statistical consistency measures (e.g. coefficient of variation across repeated runs). • SQL literacy — you can read a generated query and judge whether it answers the question and hits the right tables. • A monitoring and observability instinct: you reach for dashboards, alerts, and trace inspection by default, and you build them when they don’t exist. • Rigour about uncertainty. You report calibrated ranges, not overclaimed point estimates and you say plainly when something is unverified. Bonus • Experience evaluating or red-teaming agentic / multi-tool LLM systems specifically. • MLflow or similar trace and experiment tooling. • A background talking to technical end users (engineers, analysts) and translating their “it feels off” into a reproducible test. • Time-series, forecasting, or industrial and operational data domains. Why it matters Our customers are engineers who verify our output. Inference quality is the product. This role decides whether we can look a client in the eye and say the system works, and back it with numbers. You will have the mandate to build that assurance layer from the ground up. Department AI Role Engineering Locations Remote UK Remote status Fully Remote About Applied Computing Applied computing is one of its kind revolution with a mission to deliver sustainable abundance for a growing planet, through AI that works for the Energy Industry Founded in 2023 Co-workers 10-50