Artificial Intelligence Quality Engineer (Full-time Hybrid, Morrisville, NC Based)

The AI Quality Engineer is responsible for ensuring the reliability, accuracy, and safety of AI-driven solutions from the earliest stages of development. This role is deeply integrated into the development lifecycle, focusing on building quality products by designing sophisticated evaluation frameworks for LLMs, agentic systems, and intelligent automation. The AI Quality Engineer ensures that AI models and AI Agents behave predictably, follow complex workflows, and meet the high standards required for clinical and operational healthcare environments.

This position is full-time hybrid. Selected candidate must come into the Home office in Morrisville on Wednesdays, for business meeting as needed, reside within 50 miles of Morrisville North Carolina.

Responsibilities & Duties

AI Evaluation & Framework Design

• Design and implement scalable AI Quality Engineering strategies supporting LLM solutions, agentic systems, and enterprise AI platforms
• Develop and maintain AI evaluation frameworks, including the creation of validation datasets, quality scoring methodologies, and automated testing workflows
• Execute advanced evaluation techniques, including running LLM-as-judge, code-based, and multi-turn evaluators on real production traces
• Calibrate LLM judges to ensure their assessments align with human expert preferences and clinical accuracy
• Drive model and prompt optimization by helping the team determine the most effective system prompts and optimal model selections for specific use cases

Agentic System Validation & Regression Testing

• Perform side-by-side performance analysis to quantify how changes to agent components impact overall performance, ensuring updates do not introduce regressions
• Trace full agentic runs and conversations to visualize every step an agent takes, utilizing distributed tracing and specialized AI assistants (e.g., Polly) to pinpoint failure modes
• Identify usage patterns and common failure modes using insights agents to provide executive-level summaries of system reliability

Collaborative Development & Human-in-the-Loop (HITL)

• Partner with AI Engineering, AIOps, and Security from the beginning of the development cycle to build quality into the architecture
• Enable Subject Matter Experts (SMEs) (Clinical, Operational, and Data teams) to review agent outputs and annotate traces, fostering a robust human-in-the-loop validation process
• Collaborate across agile delivery teams to coordinate testing activities, including sprint planning, defect management, and release readiness

Observability & Runtime Reliability

• Support runtime reliability by integrating quality controls with observability tools, monitoring for model drift, and assisting in incident response
• Leverage telemetry and distributed tracing to ensure AI-enabled systems maintain high availability and performance in production environments

Minimum Requirements

Education & Experience

Bachelor’s degree in Computer Science, Data Science, Artificial Intelligence, or a related field from an accredited college or university, and three (3) years of experience developing, implementing, and deploying software, machine learning, and/or artificial intelligence solutions. Experience must include integrating, deploying, monitoring, and maintaining AI/ML models and/or third-party AI/ML tools within production environments for internal business operations and/or customer-facing applications.

At least two (2) years of experience utilizing Python and related machine learning and artificial intelligence libraries and frameworks to develop, test, deploy, and support AI/ML solutions is required.

Knowledge, Skills, & Abilities

• AI Quality Engineering & Evaluation: designing and executing AI-native testing approaches, including LLM-as-judge, code-based evaluators, and multi-turn evaluators for agentic systems.
• Agentic & Generative AI Validation: evaluating the reasoning, tool-use, and workflow reliability of Agentic workflows and Large Language Models (LLMs) using both proprietary and open-source models.
• AI Observability & Tracing: using platforms like LangSmith for distributed tracing, monitoring agent runs, and analyzing production traces to identify failure modes.
• Advanced Programming & Frameworks: Python and libraries used for automated testing and evaluation.
• RAG & Vector Database Evaluation: validating retrieval-augmented generation (RAG) architectures.
• LLMOps & Lifecycle Management: managing prompt configurations, evaluation pipelines, and version control for models, prompts, and outputs to ensure consistency and prevent regressions.
• AI-Augmented Productivity: Ability to utilize AI tools (GitHub Copilot) to accelerate testing automation, documentation, and quality engineering tasks.
• Technical Communication & Leadership: Ability to translate complex AI performance metrics and evaluation findings into actionable insights for clinical, operational, and engineering stakeholders.