This article is part of our AI Agent Guides. See how models compare this week on the AI Agent Leaderboard, a ranking of rankings refreshed every week.
AI leaderboards disagree mainly because they are not measuring the same thing: they use different tasks, setups, scoring methods and dates. So a model can place high on one and mid-table on another with no one being wrong.
That disagreement is information, not noise. It often tells you what a model is good at, and where to be careful.
Key Takeaways
- Boards differ on tasks, harness, settings, scoring, version, who gets listed, and noise.
- Small gaps are often not real. Compare them with error bars before you read anything into them.
- A rank alone hides the size of the gap between positions.
- When two boards conflict, pick the one closest to your work, then run a small trial yourself.
- See where boards agree and disagree in the Agreement column of our AI Agent Leaderboard.
A hypothetical conflict
Here is an invented example with made-up names, to show how this plays out.
| Board | Tests | Model A | Model B |
|---|---|---|---|
| Board 1 | Hard terminal tasks with automated tests | 2nd | 7th |
| Board 2 | Real-user sessions on one platform | 6th | 3rd |
| Board 3 | Finance, legal and tax work | 4th | 5th |
Which model is better? It depends on the work. Model A looks stronger on technical work, Model B looks easier day to day, and they are nearly level on professional work.
The seven reasons boards disagree
1. They test different work
Terminal-Bench sets hard computer tasks. The Vals Index blends finance, coding, legal and tax. Arena's Agent board watches whatever real users bring, and code writing is only part of that.
Even inside one company, results shift by category. Arena added Code, Chat and Work filters in August 2026 because it decided one performance-only list was no longer sufficient.
2. They grade in different ways
Some boards use automated tests. Others use rubrics applied by an AI judge, human experts or signals from user behavior. Each one has its own failure mode.
Grader choice can matter a lot. In a 2026 Arena study, AI judges agreed with each other far more often than with human voters. A board judged by AI and a board judged by people may simply prefer different things.
3. They use different metrics
A pass@1 score and a mean score can tell opposite stories about one model. Mercor reports both because one captures full completion and the other partial credit. Our benchmark terms guide explains each.
4. They use different setups
Harness, effort level and limits all change. Vals runs public benchmarks itself, under one harness and settings. So its Terminal-Bench figure may not match the source board, which lists model and agent pairs.
Mercor says its runs use maximum reasoning effort, and it notes where it strays from a benchmark's original setup. For example, its RedlineBench run uses one harness for all models where the source used several. See model vs agent vs harness for why this matters.
5. They are measured at different times, on different versions
Boards update on different schedules, and tests change. Terminal-Bench 4.0 replaced 3.0 within about a month. The Vals Index has changed versions several times in 2026, and one change lowered its scores.
So check the date on every number. A model missing from one board may just not have been tested yet. And a rival may look stronger simply because its test is older and easier.
6. They do not all test the same models
Coverage differs. A model absent from a hard board avoids a result that might have lowered its average. Our method calls this out, and we show a dash where a board has no rank.
Rankings are also relative. When a strong new model enters a board, everyone below it drops a place, though nothing about them changed. A fall in rank is not the same as a fall in ability.
7. Some of it is noise
Agents vary from run to run, and test sets are limited. Most boards publish error ranges for this reason.
Anthropic showed that setup alone moved results by 6 percentage points on Terminal-Bench 2.0. It suggests skepticism about gaps under 3 points when setups are not matched.
Rank makes noise look worse. Two models a fraction of a point apart still display as 1st and 2nd, and several positions can sit inside a single error range.
What the disagreement tells you
Once you know the causes, the pattern itself becomes useful.
| Pattern | What it might mean | What to do |
|---|---|---|
| High everywhere | Broadly capable | Shortlist it, and then check cost |
| High on hard tests, lower on real sessions | Strong but may need more steering | Trial it with an expert user |
| High on real sessions, lower on hard tests | Pleasant for everyday tasks, may hit limits | Trial it on your hardest task |
| High on one board only | Specialist, or a quirk of that board | Find out which, then decide |
| Missing from a hard board | Unknown, not proven strong | Do not assume, and test it yourself |
These are starting hypotheses, not conclusions. A board rarely tells you why a model landed where it did. That is for your trial to find out.
How to read two boards that conflict
- Read what each measures. Open the methodology. Our guide to Arena's board is one example.
- Choose the closer match. Pick the board whose tasks look most like your work, and weigh it above the other.
- Check dates and versions. Set aside anything out of date, or measured on an older test.
- Check the gap. If the models overlap within error ranges, treat them as tied, and decide on cost or fit.
- Run your own test. Give the finalists five real tasks, with the same limits, and record the review time.
Step 5 settles more arguments than steps 1 to 4 combined. A leaderboard narrows the field, and only you can test your own data and exceptions.
Why we average, and what that hides
Averaging helps because a single board can flatter a model. Our leaderboard takes each model's rank on several independent boards and averages them, with one board per publisher so no one gets two votes.
We also publish an Agreement column, which is the gap between a model's best and worst rank. A gap of 2 or less reads "Boards agree", 3 to 6 "Mostly agree", and anything wider "Boards disagree". The methodology page has the full rules.
Averaging has real costs. It hides the size of gaps, and three boards can move a lot when one changes. So use it the way you would use a consumer review average: a good way to start and a poor way to finish.
Frequently asked questions
Which AI leaderboard is the most accurate?
None is accurate for every purpose. Each is accurate about what it measures, under its own conditions. Pick by fit: a coding team should weigh coding boards, while a professional services firm should weigh the knowledge work board.
Does disagreement mean benchmarks are useless?
No. It means each one covers part of the picture. Wide disagreement is a warning to test first, while strong agreement is a good reason to shortlist.
How big a gap should I take seriously?
There is no universal figure. Start with the published error ranges for that board. Anthropic's rule of thumb for agentic coding tests is to doubt gaps under 3 points unless the setups are documented and matched.
Could a board be wrong on purpose?
It can be distorted, by cheating or by favorable testing. Publishers respond to this, and some public examples exist. See can AI benchmarks be gamed for specific cases.
