Research Engineer, Evals

About the Role

Build evaluation systems for a web data extraction product that turns pages into structured, LLM-ready information. You will shape how quality is measured, create the infrastructure to measure it at scale, and connect evaluation results to model and product decisions.

What You'll Do

• Design metrics for extraction quality across diverse websites, formats, and edge cases.

• Build scalable pipelines and evaluation harnesses to measure output quality rigorously.

• Generate and curate ground-truth datasets for trustworthy evaluation.

• Close the feedback loop by translating evaluation results into model and product improvements.

• Make quality results clear and actionable for the broader team.

What We're Looking For

• Four or more years of relevant engineering experience, including at least two years focused on evaluation work.

• Experience building evaluations for LLM extraction of unstructured data, measuring whether fields are correctly extracted from messy, real-world pages or documents.

• Experience building or evaluating large-scale web or document extraction systems and developing evaluation metrics and systems from the ground up.

• Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, and evaluation frameworks.

• Statistical rigor in metric design and interpretation, including sampling, significance, estimator bias and variance, error bounds, and judge-to-human agreement.

• Experience creating ground-truth datasets and using evaluation findings to inform real model or product decisions.

• A combination of research methodology and hands-on engineering, plus comfort working through ambiguous problems.

Compensation & Benefits

Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.

Location

Hybrid, based in San Francisco, California, or Toronto, Canada. The role requires working on-site five days a week.