QA Engineer

The Role

We're looking for a QA Engineer (Mid or Senior level) who can own quality across AI-driven systems and the integrations that hang off them. This is not traditional app QA. You'll test LLM-powered features, prompt pipelines, agent workflows, MCP integrations, and the GitHub-based delivery pipelines that power them. You'll work directly inside our repos (including degen-engine and skeleton), partner with engineers using Claude Code and Gemini Code Assist, and shape how we verify non-deterministic systems.

If "how do you QA an LLM?" is a question you've already started answering — keep reading.

What You'll Do

• Own end-to-end QA for Skeleton: AI agents, prompt pipelines, MCP server integrations, scheduled jobs (Vercel Cron), data ingestion (Apify), and database flows (Drizzle ORM).
• Design test strategies for non-deterministic systems: evaluation harnesses, golden datasets, regression suites for prompts, output quality scoring, hallucination and drift detection.
• Write and maintain integration tests across our stack (Next.js, TypeScript, pnpm, Vercel, Sentry, Jira) including API contract tests for third-party integrations.
• Test inside GitHub directly: review PRs, run test suites in CI/CD, validate auto-deploys to main, and verify fixes before they ship.
• Partner with engineers using Claude Code, Gemini Code Assist, and our broader AI dev workflow — including writing test prompts, validating tool-use outputs, and stress-testing prompt caching strategies.
• Build and maintain monitoring and observability for AI features in production (Sentry, custom eval dashboards, cost and latency tracking).
• Define quality gates and release criteria for AI-powered features, and partner with engineering on incident response when production outputs drift.
• Triage and reproduce issues across integrated systems — when something breaks, you trace it from Slack notification through Vercel logs, Sentry traces, the database, and back to the prompt.

What We're Looking For

Mid-Level (3–5 years)

• 3+ years of QA / SDET / Test Engineering experience on production software.
• Hands-on experience testing AI / LLM-powered features in production (OpenAI, Anthropic, Gemini, or similar) — prompt evals, output validation, regression testing.
• Strong TypeScript / JavaScript fundamentals; comfortable reading and writing code, not just black-box testing.
• Experience with modern web stacks: Next.js, REST/GraphQL APIs, serverless (Vercel / AWS Lambda), and at least one ORM (Drizzle, Prisma, etc.).
• Fluency in Git and GitHub workflows: PR review, branch protection, CI/CD pipelines, status checks.
• Experience writing automated tests with modern frameworks (Vitest, Jest, Playwright, Cypress).
• Comfort working in repos alongside engineers and contributing test code directly — not just filing tickets.

Senior (5+ years)

• Everything in Mid-Level, plus: deep experience defining QA strategy for AI / ML systems in production.
• Track record of building eval frameworks for LLM outputs (LLM-as-judge, golden datasets, A/B prompt testing, regression suites for non-deterministic systems).
• Experience with MCP (Model Context Protocol), tool use / function calling, agent frameworks, or multi-step LLM workflows.
• Comfort with observability stacks (Sentry, Datadog, custom dashboards) and ability to build them where they don't exist.
• Experience mentoring engineers on quality practices and shaping team-wide testing culture.
• Familiarity with prompt caching, model selection, context management, and other techniques for keeping AI systems fast and cheap in production.

Nice to Have

• Direct experience with Claude (API, Claude Code, Anthropic SDK), Gemini Code Assist, or similar AI dev tools.
• Experience with Apify, Playwright, or other scraping / browser automation frameworks.
• Background testing data pipelines, ETL flows, or analytics systems.
• Experience with Jira automation, Slack apps, or Notion API.
• Open-source contributions to AI tooling or testing frameworks.
• Curiosity about prompt engineering, agent design, or the science of evaluating language models.

Why You'll Love It Here

• Work on genuinely novel problems: QA for AI systems is being invented right now, and you'll help invent it here.
• Direct access to a small senior team building production AI pipelines from scratch — not a maintenance role, a frontier one.
• Modern stack, modern tools, no legacy debt to drag through.