ViBench 2.0 Released: A Long-Horizon Software Construction Benchmark, Top Models Fall Short of 65% Pass Rate

Long-horizon software construction capability is becoming a key yardstick for measuring the practical utility of AI agents. ViBench 2.0 has now been officially released—a benchmark specifically targeting long-horizon software construction, designed to evaluate an agent's ability to build a complete product from scratch without a global view.

What Is ViBench 2.0? An Extreme Test from Empty Project to Complete Product

Unlike common code-generation benchmarks, ViBench 2.0 simulates a real development scenario: the agent starts from an empty project, implements only one feature at a time, and cannot anticipate subsequent requirements. This "blind men and the elephant"-style construction process closely mirrors the engineering challenges human developers face when requirements gradually become clear.

The core of the benchmark lies in two constraints: "long-horizon" and "no lookahead." The agent must independently handle requirement understanding, architecture design, code implementation, and integration testing, all without being able to see what comes next. This demands strong context retention, modular design awareness, and self-verification mechanisms from the model.

According to the release information, ViBench 2.0 has been included in FireworksAI_HQ's Specialized Intelligence Index, along with official Fireworks run results, providing a foundation for reproducibility and cross-comparison of the benchmark.

"Context retention" refers to a model's ability to maintain memory of and reference to earlier information across long conversations or long code sequences. In long-horizon construction, the agent must repeatedly revisit implemented module interfaces, data structures, and design conventions. If the context window is limited or attention decays, early decisions get "forgotten," causing subsequent code to conflict with existing implementations. ViBench 2.0's "no lookahead" constraint further amplifies this challenge: the model cannot rely on future requirements to correct memory drift, and can only reduce dependence on long-range memory through proactive context management and modular encapsulation.

How Do Top Models Perform? The Truth Revealed by Pass Rate Data

According to official disclosures, even the strongest current models—GPT-6.1 Sol, GPT-6 Astra, and Opus 5.5—pass all checks across all construction tasks only 49% to 65% of the time. This means that, under the best circumstances, more than one-third of construction processes still encounter issues that prevent them from passing all checks.

This data sends a clear signal: long-horizon software construction is far from reliable. High scores on single-point code generation do not directly translate into end-to-end product construction success. As the task chain lengthens, error accumulation and context decay are significantly amplified.

From another perspective, the 49–65% pass rate also shows that ViBench 2.0's difficulty is well-calibrated for discrimination—it neither lets top models score a perfect run easily nor is it so low that capability differences fail to surface.

Why Is Long-Horizon Construction So Difficult?

The difficulty of long-horizon software construction lies in the fact that the agent must make local decisions without a global blueprint, and those decisions must remain compatible with future unknown requirements. This is essentially a sequential decision-making problem, where each step can affect the feasibility of all subsequent steps.

Current models' main shortcomings cluster in three areas: first, maintaining cross-feature consistency—new features easily break existing logic; second, insufficient ability to infer implicit requirements—models cannot reasonably deduce unstated constraints from existing code; third, limited depth of self-verification—models often struggle to detect deep flaws in their own implementations.

The value of ViBench 2.0 is that it shifts the evaluation focus from "can it write correct code" to "can it build a correct system." This is closer to the evaluation standards of real software engineering and points the way for future model optimization.

A sequential decision-making problem is one in which an agent makes choices step by step over time, and each decision changes the set of subsequent available actions and payoffs. In software construction, early choices of architectural style, interface granularity, and data models constrain how all subsequent features can be implemented. Unlike one-shot generation, errors in sequential decision-making are cumulative: a locally reasonable decision may become infeasible at some future step, and without a global blueprint the agent struggles to backtrack and correct. This is precisely the fundamental difference in problem nature between long-horizon construction and single-point code generation.

Implications for AI Coding Tools and Agent Development

The release of ViBench 2.0 provides the AI coding field with an evaluation dimension that is closer to real-world practice. For agent developers, the benchmark reveals the capability gap between code generation and system construction, suggesting the need to strengthen research in long-context management, modular planning, and iterative verification.

For enterprise users, a 49–65% pass rate means current agents are not yet suitable for fully autonomous long-horizon construction tasks, but they remain valuable in scenarios such as assisted development and feature prototyping. Benchmark results can serve as a reference for selection, helping determine at what task complexity it is safe and efficient to introduce agents.

With its inclusion in Fireworks' Specialized Intelligence Index, ViBench 2.0 is poised to become an important yardstick for measuring agent engineering capability, pushing the industry from chasing single-point performance toward end-to-end reliability.

One-line Summary

ViBench 2.0 evaluates agents' ability to build complete products from scratch without a global view; top models pass only 49–65%.

Paragraph Summary

ViBench 2.0 is a benchmark specifically targeting long-horizon software construction. It simulates a realistic development scenario in which an agent starts from an empty project, implements features one by one, and cannot anticipate subsequent requirements. Its core constraints are "long-horizon" and "no lookahead," requiring context retention, modular design, and self-verification capabilities. Official data shows that top models such as GPT-6.1 Sol, GPT-6 Astra, and Opus 5.5 pass all checks only 49% to 65% of the time, indicating that long-horizon construction is far from reliable. The article analyzes three major shortcomings—cross-feature consistency, implicit requirement inference, and depth of self-verification—and notes that the benchmark has been included in Fireworks' Specialized Intelligence Index, providing a more practice-oriented evaluation dimension for enterprise selection and model optimization.