Research Engineer, Benchmarks

Experience: Technical aptitude and learning potential matter more than years Pay: $100k-$200k, full time Location: San Francisco, CA, US or Singapore (on-site) Visa: Relocation and visa support available for strong candidates The role Build high-quality benchmarks for evaluating frontier agents on domain-specific tasks.

You will create benchmarks that are technically rigorous, practically useful, and credible to frontier labs. What you'll do

• Own the design, implementation, and quality of HUD's internal agent benchmarks
• Work with subject-matter experts to define realistic domain-specific tasks
• Build infrastructure to run models and agents reliably against benchmark tasks
• Develop metrics and analyses for benchmark difficulty, reliability, and failure modes
• Validate whether benchmark performance correlates with real-world evaluations, customer needs, and lab expectations
• Write clear documentation and benchmark reports for technical audiences

What we're looking for

• Proficiency in Python, Docker, and Linux environments
• Published papers or technical blogs about benchmarks, model failure modes, or related topics
• A strong understanding of what makes a benchmark realistic, reliable, and useful
• Experience with environments and evaluations
• Curiosity and the ability to understand how workflows in unfamiliar domains actually work
• First-principles reasoning about task design, scoring, and failure modes

Strong signals

• Detail orientation and an eye for subtle inconsistencies and edge cases
• Independent work in unstructured problem spaces
• Early-stage startup experience
• Strong communication across time zones