Remote Lead QA Engineer

Role Overview

This position focuses on developing and implementing QA processes for AI and ML systems in a banking context. You will design and execute evaluation frameworks, create automated test suites, and establish quality gates in CI/CD pipelines to ensure operational and regulatory compliance.

Key Responsibilities

• Design and execute AI evaluation frameworks, defining behavioral contracts and regression baselines.
• Develop and maintain automated tests for APIs, pipelines, and AI inference workflows, including performance and load testing.
• Build and enforce quality gates in CI/CD pipelines, triage production bugs, and write regression tests.
• Produce audit-ready documentation, test artifacts, and logs meeting SOC 2 standards.
• Collaborate with engineering on client validation and production issue reproduction.

Qualifications & Skills

• 7+ years in software QA, with at least 2 years testing AI/ML systems.
• Proficiency in Python, pytest, Playwright, or Selenium.
• Hands-on experience with evaluation tools like RAGAS, DeepEval, LangSmith.
• Ability to trace failures across systems and integrate QA in CI/CD pipelines.
• Experience in fintech, banking, or regulated environments is advantageous.

Success Indicators

Initial success includes diagnosing current test coverage, building evaluation frameworks, and establishing quality gates. Long-term success involves scaling coverage, maintaining audit-ready documentation, and embedding quality in every deployment, with the foundation laid by your direct contributions.