This article is part of our AI Agent Guides. See how models compare this week on the AI Agent Leaderboard, a ranking of rankings refreshed every week.
An AI agent benchmark gives many agents the same tasks, grades the results the same way, and reports a number you can compare. To read one well, you need only five ideas: pass rate, pass@1, mean score, Elo and the confidence interval.
Each answers a different question, which is why two boards can both be honest and still disagree. Once you know the terms, a leaderboard stops looking like a sports table and starts looking like evidence.
Key Takeaways
- A benchmark has three parts: tasks, an environment and a grader. Weakness in any part weakens the number.
- Pass rate is the share of tasks solved. Pass@1 is the share solved on a single attempt. Mean score gives partial credit.
- Elo comes from head-to-head comparison, and it is relative. It is not a percentage of anything.
- A confidence interval shows the uncertainty. If two ranges overlap heavily, treat the models as close to tied.
- See how this works in practice on the AI Agent Leaderboard and its methodology.
What a benchmark is made of
Every agent benchmark makes the same three choices, and they decide what its number means.
Tasks. These may be command-line jobs, code fixes or professional deliverables. Terminal-Bench says its best tasks come from real engineering work, described in natural language.
Environment. The agent gets a working space, tools and limits. Anthropic has shown that resources alone can change agent results, so this is not a minor detail.
Grader. The grader may be automated tests, expert-written rubrics scored by an AI judge, or human reviewers. Vals AI describes using both strict accuracy checks and rubric-based AI judges. Mercor's methodology page lists the grading method for each benchmark it runs.
The scoring terms, in plain English
| Term | Question it answers | Everyday comparison | Watch out for |
|---|---|---|---|
| Pass rate (or resolve rate) | How many tasks were solved? | Exam questions marked right | All-or-nothing, so near misses count as zero |
| Pass@1 | How often is it right first time? | First-attempt pass on a driving test | Hides improvement from retries |
| Mean score | How much of each task was right? | Partial credit on an essay | A high average can hide unfinished work |
| Elo | Who wins when compared directly? | Chess ratings | Relative to the pool, and not a percentage |
| Net improvement | How much does this choice change outcomes? | A trial of a treatment against a baseline | Only defined relative to its baseline |
| Confidence interval | How sure is this number? | The margin of error in a poll | Often left out when results get quoted |
Pass rate
Pass rate is the simplest measure: tasks solved divided by tasks attempted. Scale's SWE-Bench Pro calls it resolve rate. A task counts only if the new tests pass and none of the old ones break.
It is a strict test, and that is the point. A patch that fixes the bug but breaks something else scores nothing.
Pass@1
Pass@1 asks how often an agent succeeds on a single attempt. Mercor defines it as the share of tasks completed in one attempt, which in its case means scoring 100% on the rubric. It says it combines repeat runs into one pass@1 figure and does not report the best of several tries.
That matters because agents vary between runs. If a board reports only an agent's best try, it flatters the agent. For a business, the first-attempt number is closer to what you will see.
Mean score
Mean score gives partial credit. Mercor describes it as the average percentage of criteria passed across all tasks, and it uses both measures. The two can tell very different stories.
Here is a hypothetical example. Model A gets 90% of the rubric right on every task, but never all of it. Its mean score is 90%, and its pass@1 is zero.
Elo
Elo is a rating system originally designed for chess. It estimates relative skill from who beats whom. In AI, a related approach is used by Chatbot Arena, where people compare two answers and vote.
Read Elo-style numbers as gaps, not grades. A rating means nothing alone, and it only says something when compared with others in the same pool. It also reflects human preference, which is not the same as correctness.
Net improvement
Not every agent board uses Elo. Arena's Agent board replaced pairwise votes with what it calls causal tracing. It randomizes which model runs a session, then estimates how much that choice changes outcomes compared with a baseline.
The result is reported as "net improvement". Arena publishes 95% confidence intervals next to each estimate. We explain the details in what Arena's Agent leaderboard measures.
Confidence intervals: the number most people skip
A score from a limited set of tasks is an estimate. The confidence interval shows how far that estimate could reasonably be off, like the margin of error in a poll.
Publishers present uncertainty in different ways. Terminal-Bench draws whiskers spanning a 95% confidence interval, and Vals AI reports standard errors. Vals also warns that its error bars do not reflect variation across prompts, seeds or deployment settings.
Here is a hypothetical comparison on a 100-task benchmark:
| Model | Pass rate | 95% range | How to read it |
|---|---|---|---|
| Model A | 62% | 53% to 71% | Likely close to B |
| Model B | 60% | 50% to 70% | Likely close to A |
| Model C | 40% | 30% to 50% | Clearly behind A, barely touching B |
A table sorted by the middle number makes A look like the winner. The ranges say A and B cannot be separated. Scale's board adds a rank based on confidence bounds, so a model drops only when others clearly beat it.
Anthropic adds a practical rule: treat leaderboard gaps below 3 points with skepticism until the setup is documented and matched. That comes from its Terminal-Bench 2.0 experiments, and it is a useful default elsewhere.
Who grades the grader?
Grading is harder than it sounds. OpenAI audited SWE-bench Verified and found material issues in 59.4% of the 138 hard problems it reviewed, including tests that rejected correct solutions.
AI judges bring their own bias. In a 2026 Arena study, across 12 models, a judge picked its own answer 23 percentage points more often than people did. If a board uses an AI judge, check which model it is and who wrote the rubric.
Terminal-Bench's team goes as far as saying that huge portions of tasks in popular agent benchmarks are broken in subtle ways. It now versions tasks like software, and it fixes them openly. That is a healthy sign, and it means you should always note the version.
A checklist for reading any leaderboard
- What was tested? A coding benchmark says little about tax work.
- Who ran it? Publisher-run results are generally stronger than self-reported ones.
- Which version? Scores from different versions are not comparable.
- Which setup? Note the harness, effort level and time limit. See model vs agent vs harness.
- What is the metric? Pass rate, pass@1, mean score and Elo are not interchangeable.
- How big is the gap? Compare it with the error bars before you believe it.
- What does it cost? Vals and Arena both report cost. A small win may not justify a large bill.
- Could it be gamed? See can AI benchmarks be gamed.
Why we average positions and not scores
These numbers cannot be added together, since a pass rate and a net improvement estimate are on different scales. So our leaderboard uses each model's position on each board and averages those positions. The formula is public, and so are its trade-offs, including that rank hides the size of gaps.
Frequently asked questions
What is a good pass rate?
There is no universal answer, because difficulty varies by benchmark and by version. Harder versions lower every model's number. Compare models on the same version, and check the current spread on each source board.
What is the difference between pass rate and pass@1?
Pass rate is the general share of tasks solved. Pass@1 specifically means solved on a single attempt, counted across repeated runs. Some boards use the terms loosely, so read the definition on the board you are using.
Do benchmarks tell me which agent to buy?
They tell you which models to shortlist. They cannot test your data, tools or exceptions. Use them to narrow the field, then run a small trial on your own work.
Why do scores drop when a new benchmark version ships?
Newer versions often remove easy tasks and add harder ones. Terminal-Bench's 4.0 release removed tasks every current model solved consistently. A lower number can mean a harder test, not a worse model.
