Terminal-Bench
An AI evaluation benchmark focused on real-world software engineering tasks in command-line terminal environments. It requires agents to complete end-to-end engineering tasks via the command line—such as installing dependencies, running scripts, and debugging errors—assessing an AI's ability to execute tasks independently without continuous human intervention.
Core Facts
Terminal-Bench is a benchmark specifically designed to evaluate AI models' ability to execute complex tasks in command-line environments, covering scenarios such as file operations, script writing, and system debugging.
Terminal-Bench 4.0 is an evaluation suite for command-line agents, requiring models to complete multi-step operations in real terminal environments, testing coherence and self-correction ability in long-horizon execution.
Terminal Bench requires models to complete a full software engineering workflow in a simulated terminal environment, including reading error logs, debugging, configuring environment variables, and operating the file system.
Timeline (last 90 days)
Terminal-Bench scores are heavily influenced by the agent scaffold and retry strategy, and cross-model comparisons require confirming that the evaluation conditions are consistent.
This benchmark is not a SWE-bench- or Terminal-Bench-style task but long-cycle agentic building of real applications, and the scoring checks were written by the team itself and may have blind spots.
Public sources directly comparing V4.1-Flash and K3 on Terminal-Bench are currently lacking.
GPQA Diamond, MMLU-Pro, Humanity's Last Exam, SWE-bench Verified, Terminal-Bench, MMMU-Pro, LEXam, and Open ASR Leaderboard are all official benchmarks marked by Hugging Face as official benchmark.
Tev-27B, trained only on ALE-related data, can nonetheless generalize to Terminal-Bench, suggesting that it has learned some domain-agnostic long-horizon task execution capability rather than merely memorizing specific task solutions.
Terminal-Bench 4.0 is an evaluation suite for command-line agents, requiring models to complete multi-step operations in real terminal environments, testing coherence and self-correction ability in long-horizon execution.
Without having seen Terminal-Bench training data, Tev-27B surpassed GLM 5.2 and Gemini 3.7 Flash on the TB 3.0 and 4.0 subsets, and reached the performance of Claude Opus 4.8 on TB 2.1.
Anthropic found that under strict per-task CPU and memory limits, as many as 6% of Terminal-Bench tasks failed due to pod errors unrelated to the model; after loosening the limits, scores moved by up to 6 percentage points.
On Terminal-Bench 4.0, Haiku 5.5 scored 39.2 and GPT-6 Luna scored 16.4.
In both the DeepSWE and Terminal-Bench evaluations, Replit Agent achieved Pareto optimality relative to Astra running alone, with no published Astra baseline achieving both lower cost and higher score.
12 more timeline events
All Facts (20)
Terminal-Bench is a benchmark specifically designed to evaluate AI models' ability to execute complex tasks in command-line environments, covering scenarios such as file operations, script writing, and system debugging.
90%VerifiedTerminal-Bench 4.0 is an evaluation suite for command-line agents, requiring models to complete multi-step operations in real terminal environments, testing coherence and self-correction ability in long-horizon execution.
65%VerifiedTerminal Bench requires models to complete a full software engineering workflow in a simulated terminal environment, including reading error logs, debugging, configuring environment variables, and operating the file system.
65%UnverifiedTerminal-Bench requires models to perform multi-turn interactions in a real shell environment, including issuing commands, parsing stdout/stderr output, handling permission issues, and managing filesystem state.
60%UnverifiedTerminal-Bench scores are heavily influenced by the agent scaffold and retry strategy, and cross-model comparisons require confirming that the evaluation conditions are consistent.
50%UnverifiedThis benchmark is not a SWE-bench- or Terminal-Bench-style task but long-cycle agentic building of real applications, and the scoring checks were written by the team itself and may have blind spots.
50%UnverifiedPublic sources directly comparing V4.1-Flash and K3 on Terminal-Bench are currently lacking.
50%UnverifiedGPQA Diamond, MMLU-Pro, Humanity's Last Exam, SWE-bench Verified, Terminal-Bench, MMMU-Pro, LEXam, and Open ASR Leaderboard are all official benchmarks marked by Hugging Face as official benchmark.
50%UnverifiedTev-27B, trained only on ALE-related data, can nonetheless generalize to Terminal-Bench, suggesting that it has learned some domain-agnostic long-horizon task execution capability rather than merely memorizing specific task solutions.
50%UnverifiedAnthropic found that under strict per-task CPU and memory limits, as many as 6% of Terminal-Bench tasks failed due to pod errors unrelated to the model; after loosening the limits, scores moved by up to 6 percentage points.
50%UnverifiedOn Terminal-Bench 4.0, Haiku 5.5 scored 39.2 and GPT-6 Luna scored 16.4.
50%UnverifiedIn the Terminal-Bench Core 0.1.1 pilot across 12 selected tasks, both the baseline group and the reliability-layer group each passed 17/36 under strict trials, with the reliability layer consuming 37.49% more input tokens and 50.73% more output tokens.
50%UnverifiedThe paper explicitly acknowledges that the Terminal-Bench pilot does not support a superiority conclusion, as the reliability layer delivered no better success rate at higher cost.
50%UnverifiedAfter fine-tuning on agent trajectories, CLM achieved 87.6% on Terminal-Bench 2.1 and 81.6% on DeepSWE.
50%UnverifiedTerminal-Bench focuses on the ability to complete multi-step command-line tasks in a terminal environment, while SWE-Atlas (and its predecessor SWE-bench) focuses on the end-to-end ability to locate and fix issues in real GitHub repositories.
50%UnverifiedThe original Terminal-Bench focuses on command-line operation evaluation.
50%UnverifiedTerminal-Bench-Science is the science branch of the Terminal-Bench family.
50%UnverifiedQwen3 27B's performance on coding and agent benchmarks is: SWE-bench Pro 61.7, Terminal-Bench 2.1 73.0, and LiveCodeBench V6 90.3.
50%UnverifiedOn Terminal-Bench 4.0, the standard version of Claude 5.1 scored 55.8% and the trusted version reached 60.9%; the OS World 2.0 partial completion rate was 77.9%.
50%UnverifiedSWE-bench要求模型在真实GitHub仓库中定位问题并生成可通过CI测试的补丁，以Resolve Rate作为核心指标
50%