WeQuel logo

AI Evaluation & Quality Engineer

Hiring from
Sweden
Work type
Hybrid
Posted

624,714 remote jobs, straight from company career pages

100% free · New jobs every minute

Show job description

AI Evaluation & Quality Engineer

WeQuel is looking for an AI Evaluation & Quality Engineer for an assignment with one of our clients. You will make sure agentic workflows are measured honestly before and after they reach production – so the organisation knows what works, what does not, and why.

About the assignment

  • Workload: 100%

  • Location: Hybrid

  • Language: English

Your responsibilities

  • Design evaluation strategies and quality metrics for LLM agents (accuracy, task success, safety, cost, latency)

  • Build and maintain evaluation datasets, test cases and automated regression tests

  • Set up LLM-as-judge and human review processes where needed

  • Run red-teaming and robustness tests, including prompt injection scenarios

  • Monitor agent quality in production and feed findings back to development teams

  • Report results clearly to product owners and stakeholders

Requirements

  • At least 3 years of experience in test automation, QA engineering or data science

  • Hands-on experience evaluating LLM or ML solutions

  • Strong Python skills

  • Understanding of statistics and experiment design

  • Fluent English, written and spoken

Meritorious

  • Experience with evaluation and observability tools (e.g. Langfuse, LangSmith, Ragas, DeepEval, promptfoo)

  • Experience with CI/CD pipelines for ML or LLM applications

  • Knowledge of AI safety and the EU AI Act

  • Background in industrial or manufacturing environments

  • Swedish

Who you are

You are analytical, sceptical in a constructive way and thorough. You enjoy finding where things break, explain results in plain language, and help teams make better decisions based on evidence.

Applications are reviewed continuously – apply today!

Similar jobs

Apply for this job100% free