Research Scientist

About the Role

Join a research and engineering team at an early-stage AI benchmarking startup to develop methods for evaluating advanced language models. Your work will help establish practical standards for understanding model capabilities before they are deployed in production.

What You'll Do

• Evaluate newly released AI models across a range of use cases and analyze their capabilities.

• Design and build benchmarks, including dataset development and coordination with labelers.

• Advance automated evaluation methods for generated text to improve their accuracy and scalability.

• Work with engineering teams to implement evaluation methods in production systems.

• Collaborate with AI labs and enterprise customers to understand evaluation needs and shape research initiatives.

• Share research findings through publications and communicate insights to cross-functional partners.

What We're Looking For

• At least 3 years of full-time AI or machine learning research experience, with a track record in benchmarking, evaluation methods, or language model research; published work is valued.

• Strong Python skills and experience writing clean, maintainable code, along with Git workflows, code review, and collaborative development practices.

• Experience with deep learning frameworks such as PyTorch or TensorFlow, language modeling, and related AI or machine learning techniques.

• Familiarity with LLM infrastructure and Python web frameworks such as Django or Flask.

• Background building benchmarks or evaluation methodologies, ideally with NLP research experience.

• A degree in computer science, machine learning, or a related field is preferred; candidates with a bachelor's degree and relevant applied experience may also be considered.

• Ability to translate research findings into practical systems and give, receive, and incorporate feedback.

Compensation & Benefits

Salary range: $150,000 to $200,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California.