This article is part of our AI Agent Guides. See how models compare this week on the AI Agent Leaderboard, a ranking of rankings refreshed every week.
Terminal-Bench is an open benchmark that tests whether AI agents can complete hard, realistic jobs at a computer's command line. Automated tests decide pass or fail. It is hosted by Stanford, Harbor and the Laude Institute, and it is one of three boards in our overall ranking.
It is one of the most useful single signals for agent work. It also has limits, and it changes often enough that old numbers can mislead you.
Key Takeaways
- Terminal-Bench gives agents terminal tasks and grades the end result with automated checks. As of November 2026, the current version is 4.0.
- The main number is the resolution rate: the share of tasks solved, shown with a 95% confidence interval.
- Results pair a model with an agent, so the same model can appear more than once.
- It is a continuous benchmark. Do not compare a score from one version with a score from another.
- For current positions, see the coding board or the overall leaderboard. This article explains the method, not the standings.
What a "terminal" is, and why test there
A terminal, or command line, is the text window where you control a computer by typing commands. It is how developers install software, run programs, read logs and connect to servers.
Terminal-Bench's creators built on the idea that code and the command line are powerful tools for agents. If an agent can work there, it can do a lot of what a computer can do. So this is not only a programming test.
Version 3.0 widened the scope to coding and non-coding work. Its outputs include database snapshots, machine learning checkpoints, formal proofs, music scores, CAD files and spreadsheets. If you are not technical, read this as a test of general computer work, with a heavy tilt toward technical tasks.
How it works
The original paper, covering version 2.0, describes the basic design. Each task has its own environment, a solution written by a person and tests that verify the result. The current benchmark keeps that structure and has added to it.
A run looks like this:
- Set up. The agent starts in a container, an isolated computer environment built for that task.
- Work. It gets instructions in plain language and works on its own within a time limit.
- Hand over. Its output is collected and passed to a separate verifier container.
- Grade. Tests check the outcome and record a pass or fail.
That separation is deliberate. Keeping agent and verifier apart blocks many reward hacks, meaning ways to win without doing the work. It also allows old runs to be regraded when a verifier is fixed.
The tests are meant to check outcomes rather than implementation details, so alternate correct solutions can pass. In other words, the agent is graded on whether the job got done, not on whether it copied one approach.
What is in version 4.0
Terminal-Bench moves quickly, so here is the version history from its benchmarks page. Dates are as published there.
| Version | Released | What changed |
|---|---|---|
| 1.0 | 19 May 2025 | First release |
| 2.0 | 7 Nov 2025 | Harder and better verified, 89 tasks per the paper |
| 2.1 | 6 May 2026 | Fixed 28 tasks |
| 3.0 | 30 Jul 2026 | 74 tasks across 7 domains in its first release |
| 4.0 | 28 Aug 2026 | Removed 8 tasks, fixed 19, recalibrated resources |
The 4.0 update set a flat 8-hour agent timeout on all tasks, and it reports fewer timeouts and errors than 3.0. The team removed tasks for four reasons: saturation, refusals, public solutions and unresolved quality problems.
"Saturated" has a precise meaning. A task qualified when every class within every family of the latest generation of models solved it five times out of five. A task that everyone passes no longer separates anyone.
For the exact 4.0 task list, check the Harbor Hub listing.
How to read the leaderboard
The Terminal-Bench board lists rank, model, agent, resolution rate, cost and tokens for each row. Here is what the main ones tell you.
| Column | What it means | How to use it |
|---|---|---|
| Model | The AI doing the reasoning | Your shortlist |
| Agent | The harness running that model | Check whether it is one you could use |
| Resolution rate | Share of tasks solved | Compare within this version only |
| Whiskers | 95% confidence interval | Overlap means they may be tied |
| Cost and tokens | What the run consumed | Weigh against small score gaps |
Read the pair, not just the model
A row describes a combination. The same model can appear several times with different agents or effort settings. To learn why that matters, see model vs agent vs harness.
Do not ignore cost
Models at similar pass rates can have very different bills. The 3.0 write-up showed this with cost and token comparisons, and it notes that agents reach similar results by different routes. If two rows sit within each other's error bars, cost often decides it.
Do not compare across versions
This is the most common mistake. Terminal-Bench calls resource and task-set changes breaking changes that require re-running trials. A score quoted in a vendor's old blog post may come from a different test.
Who pays for it, and can you trust it?
Funding is worth knowing. The team says its leaderboard experiments are supported by grants from OpenAI, Anthropic, Z.ai, SpaceX AI and the Laude Institute. So labs being measured help pay for the measuring.
In its favor, a lot of the work is public. The team says issues, fixes and milestones are all handled in the open, and anyone can run the benchmark with its Harbor tool. You can check the work, which you cannot do with a private test.
It has also dealt with real problems in public. Its integrity update from April 2026 named submissions caught cheating or reward hacking, then set new rules. Passing runs now need full records, and an agent judge reviews them.
Limits to keep in mind
- It is an open-internet test. The team tells agents not to look up solutions, but agents did so on an earlier version.
- Verifiers are still improving. Its next planned milestone, 4.1, is about tamper-resistant verifiers.
- Infrastructure affects scores. Anthropic found a 6 point gap on version 2.0 from setup alone, which is why the 4.0 calibration matters.
- It is a technical test. A strong result says little about customer service or legal drafting.
How it fits into our leaderboard
We use Terminal-Bench as one of three boards in the overall ranking, along with Arena Agent and the Vals Index. It also feeds the coding board. The methodology page explains how we pick each model's best configuration.
Vals AI also runs Terminal-Bench itself, under one harness and three runs per model. So you may see two sets of numbers for one test, and they can both be accurate. Our guide to why AI leaderboards disagree covers this.
Frequently asked questions
Who created Terminal-Bench?
The site lists it as hosted by Stanford, Harbor and the Laude Institute. The paper credits 85 authors, led by Mike Merrill, and later releases credit a large community of contributors and reviewers.
Can I run Terminal-Bench myself?
Yes, if you are comfortable with developer tools. The run guide uses the Harbor tool and notes that some tasks need GPUs, so a cloud sandbox with GPU access is required. Budget for it, since full runs consume a lot of tokens.
Is Terminal-Bench only for coding?
No, though it leans technical. Since version 3.0, it has included non-coding work done through a computer. It does not test talking with customers or reading a room.
Which model is best on Terminal-Bench?
That changes as new models and versions ship, so we do not state it here. Check the source board or our coding ranking, which is refreshed weekly.
