AI Automation QA

Unleash innovation in AI testing — drive the future of safe, reliable AI-powered solutions!

Porto-based opportunity with remote work

As a Senior AI & Automation QA Engineer, you will be working for our client, a leader in AI-powered features across conversational AI, productivity tools, and rating systems. You will play a pivotal role in building and maintaining the testing infrastructure that ensures the accuracy, safety, and regression-free performance of cutting-edge AI applications. This position offers a dynamic environment dedicated to advancing AI reliability and fostering professional growth.

Your main responsibilities:

• Develop and maintain automated test frameworks using tools like Playwright, Cypress, or Selenium to ensure AI features perform flawlessly.
• Script custom testing solutions leveraging Python or JavaScript/TypeScript to enhance automation and evaluation pipelines.
• Design and implement evaluation datasets and automated pipelines, including innovative LLM-based judgment approaches.
• Analyze AI outputs to identify hallucinations, errors, and biases, supporting continuous improvement.
• Integrate AI evaluation and functional tests into CI/CD pipelines for seamless deployment.
• Utilize AI observability and tracing tools for debugging model behavior in production environments.
• Craft and execute test cases for conversational AI, including dialogue flows, edge cases, and adversarial prompts.
• Collaborate on ensuring AI outputs meet fairness, bias, and compliance standards through rigorous testing.

You're ideal for this role if you have:

• 3+ years of experience in QA/SDET roles, especially in automation testing.
• Hands-on experience building automated test frameworks with Playwright, Cypress, Selenium, or similar.
• Proficiency in scripting languages such as Python or JavaScript/TypeScript.
• Understanding of AI-specific testing challenges, including prompt regression, hallucination detection, and output scoring.
• Experience designing evaluation datasets and using AI observability/tracing tools.
• Knowledge of fairness, bias mitigation, and compliance testing for AI models.
• Familiarity with CI/CD pipelines to automate testing processes.
• Strong root-cause analysis skills covering both code and probabilistic AI outputs.

It is a strong plus if you have:

• Experience with AI-assisted test generation or self-healing scripts.
• Knowledge of LLM evaluation approaches and AI debugging tools.
• Certifications related to QA or AI safety.

Language Required for the role: Fluent English

Eligibility to work in Europe: Only candidates with an existing legal right to work in the European Union will be considered for this role.

#MAKEYourCareerBETTER

Interested? Apply now and include your CV (preferably in English) along with a statement confirming your consent to the processing and storage of your personal data.