Research Scientist

About the Role

Develop benchmarks and evaluation methods for assessing advanced language models. This research and engineering role combines hands-on AI research with building practical evaluation systems, helping shape how models are tested before deployment.

What You'll Do

• Evaluate newly released AI models across a range of capabilities and use cases.

• Build benchmarks from the ground up, including dataset creation and coordination with labelers.

• Develop research findings and contribute to publications on evaluation methods.

• Improve automated methods for evaluating generated text.

• Work with engineering teams to implement and scale evaluation approaches in production systems.

• Collaborate with AI labs and enterprise customers to understand evaluation needs and define research initiatives.

What We're Looking For

• At least 3 years of full-time AI or machine learning research experience, especially in benchmarking, model evaluation, NLP, or language models.

• A track record of published research, ideally in NLP or benchmarking, alongside an interest in translating research into applied systems.

• Strong Python skills and experience building clean, maintainable software, including HTTP services with Django, Flask, or similar frameworks.

• Hands-on experience with PyTorch or TensorFlow, language modeling, and relevant AI/ML infrastructure and techniques.

• Experience designing benchmarks and evaluation methodologies.

• Strong computer science fundamentals and experience collaborating through Git workflows, code reviews, and cross-functional feedback.

• A degree in computer science, machine learning, or a related field; a master's or PhD is preferred. Candidates with a bachelor's degree and 1 to 3 years of applied experience may also be considered.

• Experience in an AI startup, research lab, or early-stage company is helpful.

Compensation & Benefits

Base salary: $150,000 to $200,000 annually. Visa sponsorship is available.

Location

On-site in San Francisco, California, United States.