Senior Automation Tester in Python (for Gen AI powered applications)
- Hiring from
- Poland
- Work type
- Remote
- Posted
- Sep 26, 2026
Is this job info correct?
Back to Search
Senior Automation Tester in Python (for Gen AI powered applications)
Remote in Poland
Automated Testing in Python& 9 others
apply
Looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer.
Find me a jobWe are looking for a Senior Automation Tester in Python to join Production Pods within the program. This is not a manual/functional QA role. You will build and maintain automated test frameworks for AI/GenAI-powered clinical applications, validate non-deterministic AI outputs, and ensure GxP compliance through automated IQ/OQ/PQ qualification. You will work across multiple use cases in Safety, Regulatory, Clinical Operations, and SAGE platform tooling.
Responsibilities
- Build automated test frameworks from scratch for AI-powered clinical applications using Playwright, Selenium, and pytest
- Design and execute test strategies for non-deterministic AI/GenAI outputs, including accuracy, RAG relevance, hallucination checks, and prompt regression
- Integrate automated tests into CI/CD pipelines with quality gates such as block-on-failure and coverage thresholds
- Execute GxP-compliant IQ/OQ/PQ scripts and generate qualification evidence packages
- Manage AI-specific test data, including edge cases, adversarial inputs, and golden datasets
- Monitor model drift, regression, and performance degradation over time
- Produce test strategy documents, test plans, and compliance-ready test evidence
- Collaborate with Data Scientists and Backend Developers to define testable acceptance criteria for AI features
Requirements
- 5+ years of hands-on QA/test automation experience
- Expertise in building and maintaining automated test suites from scratch using Playwright, Selenium, Cypress, and pytest
- Skills in API and backend test automation, including REST APIs, microservices, and serverless architectures with Postman/Newman and pytest + requests
- Proficiency in Python as the primary automation language, with the ability to write harnesses, fixtures, and utilities independently
- Knowledge of CI/CD test integration, embedding automated tests into pipelines such as Jenkins and GitHub Actions with quality gates
- Background in testing AI/GenAI applications, validating non-deterministic AI/LLM outputs for accuracy, RAG relevance, hallucination, and prompt regression
- Familiarity with AI/ML model testing, including accuracy, bias, and performance validation using RAGAS, LLM-as-judge, and golden datasets
- English proficiency at B2 level or higher
Nice to have
- Background in pharmaceutical / life sciences domain, with GxP awareness and IQ/OQ/PQ qualification
- Experience with Jira/Xray, TestRail, or Zephyr test management
- Skills in performance/load testing using JMeter and Locust
- Understanding of security testing basics such as OWASP
- Familiarity with AWS services including Lambda, API Gateway, S3, and CloudWatch