Member of Technical Staff – AI Evaluation

Member of Technical Staff – AI Evaluation | Physical AI

Location: US or Canada (Remote) | On-site in San Mateo, CA, one week every six weeks

Experience: 5–15 years

Employment Type: Full-time

Equity: Competitive equity

Work Authorization: Existing work authorization required; visa sponsorship is not available.

About the Role

What does it take to evaluate an AI system that can engineer real-world physical systems?

We’re looking for a Member of Technical Staff – AI Evaluation to build evaluation frameworks for an AI engineer working across physical engineering domains. This is an opportunity to define how AI engineering capabilities are measured, benchmarked against human engineers, and improved over time.

You will work at the intersection of agentic AI, AI evaluation, and physical engineering, building systems that determine not just whether an AI model produces an answer, but whether its engineering decisions are technically sound, reliable, and practically useful.

This is a high-ownership, individual contributor role for someone who enjoys solving ambiguous problems, building from scratch, and turning rigorous evaluations into measurable improvements.

What You'll Do

• Build and own AI evaluation frameworks from the ground up, defining what high-quality engineering design and reasoning look like when there is no single correct answer.
• Create realistic evaluation tasks based on real-world engineering workflows, constraints, edge cases, and failure modes.
• Design automated evaluation judges, including LLM-as-a-Judge systems, and validate their results against expert human judgment.
• Develop human-in-the-loop evaluation processes and reproducible testing environments for agentic AI systems.
• Collaborate with engineering subject matter experts, customers, partners, and internal technical teams to understand how expert engineers work.
• Analyze evaluation results, identify model weaknesses, and translate findings into actionable priorities for improving AI reasoning and performance.

What We're Looking For

• 5–15 years of relevant experience as a Member of Technical Staff, AI Engineer, AI Research Scientist, or in a closely related technical role.
• Hands-on AI evaluation experience is mandatory: You have created evaluation tasks, designed evaluation judges, and conducted human evaluations for AI systems.
• A strong understanding of agentic AI systems and the challenges involved in evaluating multi-step reasoning, task execution, and complex outputs.
• A technical background in physical engineering, such as mechanical, electrical, aerospace, robotics, or power systems engineering, with the ability to assess engineering design quality.
• Experience validating automated evaluation results against human or expert judgment.
• Strong ownership and communication skills, with experience working directly with technical experts, customers, partners, or cross-functional teams.
• Ability to work independently in an ambiguous, fast-moving environment.

Nice to Have

• Experience at a physical AI, robotics, or engineering automation company.
• Experience building technical systems, products, or evaluation frameworks from zero to one in an early-stage startup.
• Familiarity with human-in-the-loop evaluation, evaluation harnesses, benchmark design, and LLM-as-a-Judge.
• Exposure to supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), simulation environments, or engineering design tools.

Why Consider This Opportunity?

• Own a critical evaluation framework for an ambitious AI engineering product.
• Work at the intersection of frontier AI, agentic systems, robotics, and physical engineering.
• Collaborate directly with engineering experts and technical teams to shape how AI engineering capabilities are measured and improved.
• Take end-to-end ownership of a challenging, zero-to-one technical initiative.
• Receive competitive equity and comprehensive benefits.

Location and Eligibility

This is a full-time opportunity for candidates based in the United States or Canada, with remote work and a required on-site week in San Mateo, California, every six weeks. Relocation is considered on a case-by-case basis.

Candidates must already have appropriate work authorization. Visa sponsorship is not available.