Research Scientist

About the Role

As a Research Scientist on an AI research and engineering team, you will develop benchmarks and evaluation methods for large language models. Your work will help determine how advanced AI systems are assessed and support the translation of research into practical evaluation tools.

What You'll Do

• Evaluate newly released AI models across a range of capabilities and use cases.

• Design benchmarks from the ground up, including dataset construction, labeler coordination, and research publication.

• Develop scalable, accurate methods for automatically evaluating generated text.

• Partner with engineering teams to implement and scale evaluation methods in production systems.

• Work with AI labs and enterprise customers to understand evaluation needs and shape research initiatives.

What We're Looking For

• At least 3 years of full-time AI or machine learning research experience, with a track record in benchmarking, evaluation methods, or language model research.

• Published research, ideally in NLP or benchmarking, and experience building benchmarks or evaluation methodologies.

• Strong Python skills and experience building clean, maintainable code, including HTTP servers with Django, Flask, or similar frameworks.

• Hands-on experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant AI/ML techniques.

• Strong computer science fundamentals and the ability to turn research insights into practical, scalable systems.

• Experience working collaboratively with Git, code review, and cross-functional feedback.

• A degree in Computer Science, Machine Learning, or a related field; a Master's or PhD is preferred. Candidates with a Bachelor's degree and 1 to 3 years of applied experience may also be considered.

Compensation & Benefits

Compensation is $150,000 to $200,000 USD annually. Visa sponsorship is available.

Location

On-site in San Francisco, California, United States.