Benchmarks Storyteller

About Openbenchmarks

Agents are becoming first-class users of the internet. They research, compare tools and increasingly make build-versus-buy decisions on behalf of people. Every company will need its product picked by agents, and agents prefer open, independent, grounded benchmarks to make that call.

Openbenchmarks is the evaluation infrastructure for agents: domain-specific, reproducible evaluations that help agents and people pick tools with confidence. Our mission is to be the trusted evaluation layer for agents.

Founders previously led AI research and infra teams at Oracle and Appfolio. We're a small team of researchers and engineers working with fast-growing AI-first companies like Parallel, Firecrawl, Telnyx, TinyFish and more.

About the role

Every benchmark we run produces a lot of data: accuracy, latency, cost, failure modes, and how results change between versions. You make sure what we publish is correct, clear and convincing, and you communicate it. That means rigorous statistics, graphs that show the real trade-off at a glance, and technical writing that holds up, from reports to posts.

The bar is a benchmark report a skeptical researcher can't poke holes in. Every number has an error bar, every comparison is controlled, and every claim is backed by a test.

What you'll work on

Technical writing - Write up each benchmark: methodology, results, limitations, and enough detail for anyone to reproduce our work. Technical reports by default, research papers when the work warrants it.

Statistics - Standard errors and confidence intervals on every result, stated sample sizes, and proper tests (e.g. mixed-effects models with per-task effects) before we call one tool better than another. Explain the outliers, don't hide them.

Methodology - Design controlled comparisons: isolate one variable, hold the rest fixed, and write up the grading rules and judge setup so anyone can reproduce the result.

Graphs - Score vs. cost with Pareto frontiers, cost broken into components, latency and cost distributions, and per-category results with error bars. Built in code, clear before pretty.

Communicating results - Turn each benchmark into reports, launch posts and threads for X, LinkedIn, Reddit and HN. Lead with one sharp finding and one graph.

Finding the story - Work with our researchers on the raw results. Dig into individual failures, spot what changed between runs, and decide what's worth saying.

You might be a fit if

• You have a strong background in statistics: regression, mixed-effects models, significance testing, and you know when a result isn't significant
• You've designed or run evals (ML or LLM evaluation) and know where they go wrong
• You write clear technical prose: methodology, results and limitations that a skeptical reader trusts
• You've made graphs that changed how people saw a result, and can show them

Nice to have

• You've written research papers (published or not)
• Familiarity with existing benchmarks (SWE-bench, Terminal-bench, BrowseComp) and how vendors cite them
• An audience on X, LinkedIn or a blog

Visa: US citizenship/visa not required.