This article is part of our AI Agent Guides. See how models compare this week on the AI Agent Leaderboard, a ranking of rankings refreshed every week.
Arena's Agent leaderboard ranks AI models by how they behave in real work sessions on Arena's own site. It uses random assignment to estimate each model's effect.
It is not a fixed test. It measures what happens when real people hand agents real jobs.
That makes it a strong complement to lab-style benchmarks. It also means the results reflect one platform and its own users, not your business.
Key Takeaways
- Arena Agent gets its data from live use of Agent Mode on arena.ai, not from a fixed set of test tasks.
- The method randomizes which model handles a session, then estimates each model's effect. Arena calls this causal tracing and reports "net improvement".
- Five signals feed it: confirmed success, praise vs complaint, steerability, bash reliability and tool hallucinations.
- As of November 2026, Arena ranks the main model only. It says rankings of other agent components are planned.
- For current positions, see the AI Agent Leaderboard or the reliability board. We do not state ranks here.
Who runs it
The board comes from Arena Intelligence, the company behind what began as Chatbot Arena. Its early platform was described in a 2024 research paper based on crowdsourced head-to-head votes. Arena announced a $200 million Series B in October 2026.
The Agent board launched on 4 June 2026. Arena's write-up was last updated on 15 October 2026 and notes a 1 October methodology change. Its signal definitions are dated 12 October 2026, so treat that page as the current reference.
How it differs from a normal benchmark
A typical benchmark sets fixed tasks and grades them. Arena watches real tasks instead, and the scale is large. In one 7-day slice, it reported 160,480 Agent Mode tasks, and code writing was the largest category at 17.5%.
| Fixed benchmark | Arena Agent | |
|---|---|---|
| Tasks | Chosen by the publisher | Whatever users bring |
| Success judged by | Tests or rubrics | Users' responses and environment signals |
| Setup | Same for every model | Arena's own Agent Mode |
| Good at showing | Peak ability on hard work | Everyday behavior over time |
| Weak at showing | How it feels to work with | Whether work was truly correct |
Neither design is better in every case. If you read only one kind, you miss what the other catches. That is a large part of why we average across boards.
How the ranking is calculated
Arena's older boards used head-to-head votes. This one, according to Arena, works more like a medical trial, and the plain-English version has three steps.
- Randomize. Each session gets a model chosen at random from a known set of probabilities.
- Measure. Arena computes an outcome for each signal, per task or per turn.
- Compare. It estimates how much each model changes that outcome against a baseline of all models, then combines the signals.
The result is called "net improvement". A positive number means better than the baseline on the mix of signals. Arena reports a 95% confidence interval with each estimate, and it gives more weight to recent data so that older sessions fade out.
Randomizing is the clever part. If users picked their own model, stronger models might attract harder or easier jobs, which would skew the comparison. Random assignment addresses that bias, so differences are more likely to come from the model itself.
The five signals
Arena's definitions page is the source for this table. These definitions have changed before, so check it directly if you need precision.
| Signal | What it measures | How it counts |
|---|---|---|
| Confirmed success | Did the user say the task was completed? | Only a "yes" to the pop-up counts |
| Praise vs complaint | Did the user react well or badly? | Success if praise outnumbers complaints |
| Steerability | How much correcting did the agent need? | Repeated corrections cost much more |
| Bash reliability | Did shell commands fail through the agent's fault? | Failures per call, with outside errors exempt |
| Tool hallucinations | Did it call a tool that does not exist? | Rate of invented calls |
Confirmed success is deliberately strict
A task counts only if the user answers "yes" when asked whether it was completed. Saying "perfect, thanks" does not count, and neither does walking away. Good work that nobody confirmed earns nothing, so this signal likely undercounts success for every model.
Steerability is about taking correction
Real work involves mistakes. This signal asks how often users had to correct the agent, and it penalizes repeat corrections most because the first one did not land. For a manager, this may be the most recognizable signal of the five.
Bash reliability and tool hallucinations check the machinery
Bash is a common command-line language. A failure counts when the agent causes it, such as a mistyped flag, while a network outage does not count. A tool hallucination means the agent called something that does not exist, and Arena says base rates are low.
What Arena can show you that others cannot
Live data surfaces behavior that a fixed test may miss. The methodology post offers two examples.
- Bluffing. The agent presents incomplete work as complete when it could have said otherwise.
- Bluster. Agents sound firm when pushing back, then rarely hold their position.
Arena also tracks actual spending. It says some models turn out more expensive in practice despite cheaper on-paper prices, because of extra steps or extra user turns. Since August 2026, it has also offered price per task and categories for Code, Chat and Work.
The limits
We use this board, and you should still read it with care. These are the cautions we would give a client.
- One platform. All data comes from Arena's own Agent Mode and its tools. Your harness and your connections will differ.
- One crowd. Arena's users choose to be there, and they may not resemble your staff or your customers. Its largest task categories may not match your work.
- Satisfaction is not correctness. A user can approve work that contains an error they did not catch. A low-stakes task is judged differently from an audited one.
- Definitions move. Bash Recovery became Bash Reliability, and the wording around confirmed success has also changed. Do not compare across methodology changes.
- Arena has a commercial interest. It is a venture-funded company, and it publishes results for Max, its own router, on this board. Weigh that when reading its router claims.
Arena's earlier boards were also the subject of a well-known critique. We cover it, and Arena's reply, in our guide to gaming benchmarks.
How we use it
Arena Agent is one of three boards in our overall ranking, and it counts as one publisher with one vote. Our reliability board goes deeper, averaging positions across four Arena signals: outcomes, tool calls, shell use and steering.
Since that board comes from a single publisher, it is one view of behavior and not an independent consensus. The methodology page explains how the data is used. To compare it with a fixed test, read our Terminal-Bench guide.
Frequently asked questions
Is Arena's Agent board based on votes?
Not in the old sense. Earlier Arena boards used head-to-head votes. Agent Arena replaces those with randomized assignment and outcome signals, although user responses are still part of several signals.
What does "net improvement" mean?
It is Arena's estimate of how much a model changes outcomes compared with a baseline of the available models. Higher is better. Read the confidence interval too, since a small lead may not mean much.
Does it test the whole agent or just the model?
Arena says the current board evaluates the orchestrator, meaning the main model that chooses tools. Harness and tool rankings are described as coming later. Our harness guide explains why that split matters.
Should I pick a model from this board alone?
No. It is good evidence about everyday behavior on one platform. Compare it with other boards through our leaderboard, then run your own trial.
