Relomote
Avenga logo

Senior AI Quality & Evaluation Engineer

AvengaEngineering & Development
Hiring from
Bulgaria
Work type
Hybrid
Posted
Apply for this job100% free

614,800 remote jobs, straight from company career pages

100% free · New jobs every minute

Show job description

Avenga is an international engineering firm helping businesses operate with AI at the core. With 6,000+ experts worldwide, we combine engineering expertise with AI-native thinking to turn ambitious ideas into real-world impact across industries, technologies, and markets.

At the core of the role

In Bulgaria, are looking for a highly skilled Senior AQA Engineer to design and build automated testing frameworks for AI-powered applications, with a strong focus on LLM evaluation, AI safety, RAG and agent testing, and integrating quality checks into CI/CD pipelines.


Core skills you’ll bring

  • 5+ years in software quality or test engineering, with strong Python (C#/.NET is a plus). You write test frameworks, not only test cases.

  • Hands-on experience evaluating LLM applications: evaluation frameworks (for example Promptfoo, DeepEval, Ragas, Langfuse or LangSmith), building datasets, designing judges and metrics.

  • Working knowledge of RAG and agent architectures and how they fail: retrieval quality, tool-call errors, loops, hallucination, context loss.

  • Experience with adversarial or security testing, of AI systems or in general.

  • CI/CD and infrastructure-as-code literacy; comfortable running tests on at least one of Azure, AWS or GCP.

  • Clear written reporting: you turn findings into decisions.

  • Full working proficiency in English.

Nice-to-have

  • Familiarity with the EU AI Act, ISO/IEC 42001 or the NIST AI RMF.

  • Performance and load testing of LLM-backed services.

  • Benchmarking open-weight models.

  • Experience in regulated industries or in client-facing delivery.

The impact you'll create

  • Design and own the automated test strategy for the platform: unit, integration, contract, end-to-end and load tests, wired into CI/CD.

  • Build and maintain evaluation suites: datasets, rule-based and LLM-as-judge scorers, and regression tests that run on every prompt, model or tool change.

  • Evaluate new models and tools against quality, safety, latency and cost criteria before they go into use.

  • Test AI controls adversarially: prompt injection, jailbreaks, data leakage and tool misuse. Keep a red-team suite current and report findings with severity and reproduction steps.

  • Define quality gates and release criteria, and make quality visible in dashboards.

  • Monitor AI quality in production and raise alerts when behavior degrades.

  • Produce clear quality evidence for security reviews, AI governance requirements and clients.

  • Coach engineers on testable design and review their test and evaluation coverage.

 

What supports your journey

Guided by our values, we aim to provide you with the tools, autonomy, trust, and support you need to excel. Our benefits include private health insurance, well-being programs, flexible and hybrid work arrangements, laptops and equipment, training opportunities, social events, great offices, and more.

People are at the core of Avenga. We provide equal opportunities regardless of race, ethnicity, gender identity, sexual orientation, disability, age, religion, or any other characteristic. We are committed to a respectful environment where everyone can be themselves, share their ideas, and feel that they belong.

Similar jobs