Senior Research Scientist

About the Role

Join an applied AI research team developing benchmarks and evaluation methods for large language models. Your work will help organizations assess how AI systems perform on practical tasks before deployment.

What You'll Do

• Evaluate newly released AI models and communicate their performance.

• Design benchmarks from the ground up, including building datasets, coordinating labeling, and writing reports or white papers.

• Improve automated methods for evaluating generated text and increase their accuracy and reliability.

• Partner with engineers to implement and scale evaluation approaches.

• Work with AI research teams and enterprise customers to understand evaluation needs.

What We're Looking For

• At least 3 years of AI or machine learning research and evaluation experience, particularly with benchmarking, language models, or NLP.

• A master's or PhD in computer science, machine learning, or a related field, or a bachelor's degree with relevant experience.

• Strong Python skills and experience writing clean, maintainable code, collaborating in development sprints, using Git, and reviewing pull requests.

• Experience with PyTorch or TensorFlow, language modeling, LLM infrastructure, and relevant AI techniques, including diffusion models.

• Experience designing benchmarks and evaluation methods; applied research experience is important, while relevant publications are valued but not required.

• Familiarity with Django or Flask, AWS deployment and scaling, and React with TypeScript for dashboards or interface components.

• Ability to translate research into practical systems and collaborate effectively with engineering and customers.

Compensation & Benefits

Annual salary range: $150,000 to $200,000. Visa sponsorship is available.

Location

On-site in San Francisco, California, United States.