AI Senior QC

About The Role

SoftServe is delivering the AI Student Tutor education company — a Canvas-integrated, context-aware AI tutor and learning coach. The product routes student conversations through a single-assistant orchestration layer to specialized AI capabilities (tutoring, content generation, safety/distress handling), and must meet strict pedagogical and safety data requirements ahead of a production pilot launch.

The Senior QC Engineer owns quality across this engagement, working within the SoftServe delivery team alongside AI/ML engineering, application engineering, product, and data engineering, and coordinating directly with client's Product, SME, IT, and Security/Compliance stakeholders through UAT and pilot-readiness activities.

Responsibilities

• Own creation of the Acceptance Testing Plan (ATP), defining acceptance criteria, test types, roles and responsibilities, test data, evidence expectations, defect severity definitions, retest process, and formal sign-off requirements, in coordination with Covista.
• Design and execute the overall test strategy spanning functional, integration, regression, load/performance, accessibility, safety, and pedagogical-quality testing.
• Test the AI agent application itself, including single-assistant routing/orchestration logic, prompt governance and version-tracking, persona-adaptive behavior, content-validation and grounding/citation pipelines, and resistance to identity-override and prompt-injection attempts.
• Run safety scenario testing across low, medium, high, and critical distress-handling cases, measuring detection accuracy and false-negative/false-positive rates.
• Drive UAT: build test scripts, execute with client's Product, SMEs, IT, Security/Compliance, and pilot stakeholders; manage defect triage, disposition, and re-test.
• Serve as the primary client-facing point of contact for quality matters: communicate testing status, defect trends, and pilot-readiness risk directly to client's Product, SMEs, and other UAT stakeholders.
• Jointly verify and reproduce reported bugs with Covista during UAT and Hypercare, confirming severity, scope, and expected vs. actual behavior before logging.
• Translate verified client-reported issues into clear, actionable defect write-ups (repro steps, environment, evidence, severity) and hand them off to the Europe-based delivery team, working within the daily US/Europe overlap window to ensure timely triage and unblock next-day resolution.
  
  
  

Requirements

• Proven experience leading QA/QC for complex, multi-workstream software programs, including authoring test strategies and acceptance test plans from scratch.
• Experience testing AI/LLM-based or conversational agent systems — prompt-driven behavior, RAG/grounding validation, model-output quality rubrics, and safety/guardrail testing.
• Familiarity with LMS integration testing (Canvas/LTI, SSO, roster sync) is a plus.
• Strong written documentation skills for test plans, traceability matrices, defect reports, and readiness packages.
• Strong client-facing communication skills, comfortable running bug-verification calls/sessions directly with client stakeholders and writing defect descriptions clear enough for a remote team to act on without follow-up questions.
• Availability during the US–Europe daily overlap window to support live client bug verification and same-day handoff to the Europe-based team.
  
  
  

SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.