Terminal Bench: Agent Evaluation Platform

Terminal-using AI agents are graded on toy tasks — a single command, a clean success signal, a benchmark that stops mattering the week a new model ships. That's not what production agents face. Real DevOps, security, and database work is multi-step: a task only counts as solved if a chain of shell commands, config edits, and service restarts lands in the right end state, often with more than one valid path to get there.

We built Terminal Bench to close that gap: a benchmark of 500+ tasks spanning 8+ domains — Linux administration, DevOps, security, and databases among them — where every task demands genuine multi-step reasoning inside a real terminal environment, not a single memorized command.

A benchmark decays the moment a task can be solved by pattern-matching instead of reasoning. Frontier models are trained on enormous corpora of shell history, Stack Overflow threads, and infrastructure-as-code repos, so single-command tasks saturate fast — every lab reports near-100% pass rates within a release cycle or two, and the benchmark stops discriminating between a genuinely capable agent and one that memorized the answer.

Our design brief was the opposite: tasks hard enough that a frontier model still fails on the majority of them. Each task requires the agent to hold state across multiple commands, interpret ambiguous error output, and make judgment calls about which of several valid remediations to apply — the kind of reasoning chain that doesn't show up in a single scraped snippet.

A benchmark is only as useful as its headroom. If the model everyone is trying to beat can already pass 90% of your tasks, you've built a leaderboard, not a benchmark.

Coverage spans 8+ domains, each with its own failure modes and its own definition of "correct":

• Linux administration — filesystem permissions, process management, systemd units, package dependency resolution.
• DevOps — CI/CD pipeline debugging, container orchestration, infrastructure-as-code drift correction.
• Security — privilege escalation triage, log forensics, hardening a misconfigured service without breaking it.
• Databases — schema migration under constraints, query performance regression, backup/restore under data-loss risk.

Every task ships with a scripted starting environment, a natural-language objective, and a hidden end-state specification the agent never sees — so success has to come from correctly interpreting the goal, not from spotting a grading script.

Task quality is the whole product, so every task passes through 4 layers of validation before it enters the benchmark:

That last layer is where the benchmark earns its teeth: with the full 4-layer gate in place, GPT-5 passes ≤20% of tasks — leaving substantial headroom for the next generation of agents to actually be measured against.

Terminal Bench gives teams building terminal-native agents a benchmark with real discriminative power: 500+ tasks, 8+ domains, 4 layers of validation standing behind every one, and a ceiling far above what any current frontier model clears. That headroom is the point — it's a benchmark built to still be measuring something meaningful a year from now, not a leaderboard that gets saturated in a quarter.