The Harness Matters More Than the Model: YC's Deep Dive into Agent Architecture Evolution

While most people are still chasing bigger, smarter models, YC made a striking argument at a technical event called "Harness Night": what truly separates capable AI agents isn't the model weights themselves — it's the "Harness" wrapped around them.

This layer, which many researchers dismiss as "just prompt engineering" or "just scaffolding," turns out to matter enormously. YC's data makes the point bluntly: on ARC-AGI, a benchmark measuring fluid intelligence, the same model with different Harness designs can jump from 30% to 95% — or even 100%.

Why the Harness Deserves to Be Taken Seriously

The Harness has long been treated as second-tier work in academia — not worthy of a top conference. Some on social media have gone so far as to say "context engineering isn't a research problem." YC pushed back with numbers: the difference between Harness 1 and Harness 2 alone can produce an 18% performance gain.

Take ARC-AGI as an example. The benchmark's core value lies in testing how quickly a model adapts to new problems and new distributions — so-called fluid intelligence. Claude Opus's best native score on the private holdout set is 30%. With the right Harness, Prime Agent reached 95%, and NVIDIA's ABO hit 100%.

This reflects a deeper issue: current progress is overwhelmingly focused on pushing model IQ higher, while almost no one is fully exploiting test-time experience. We generate enormous amounts of test-time interaction data, but models can't quickly adapt when they encounter new domains. We're forced to toggle between a fragmented training pipeline — from ICL (in-context learning) to small-rank and large-rank LoRA to SFT. The Harness is precisely where it can plug that "rapid adaptation" gap.

ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) was proposed by François Chollet in 2019 and is widely considered one of the hardest AI benchmarks to "game." Each puzzle is a visual pattern transformation problem that asks the model to infer an abstract rule from very few examples (typically 2–5) and apply it to a new case. Unlike QA benchmarks that rely on vast pretraining knowledge, ARC-AGI puzzles are deliberately designed to fall outside any training distribution — memorization won't help. Chollet characterizes it as a measure of fluid intelligence (the ability to quickly build mental models when facing novel problems) rather than crystallized intelligence (stored knowledge). This makes it an ideal testbed for evaluating the real value of a Harness: the advantage of knowledge stored in model weights is largely neutralized, so the Harness's ability to organize test-time reasoning becomes especially visible.

From V0 to V1: Six Years of Static Harness Evolution

YC traced a line of development from GPT-2 in 2019 to the present — a history that's especially valuable for practitioners new to the field.

V0: The Simplest Possible Loop

The earliest Harness was extremely simple: a while not end-of-sequence loop, top-p sampling, and an environment. No tool calls, no skills, nothing else. Take the classic GSM8K math benchmark: the system prompt sets a persona ("You are a math teacher"), presents the problem, the model outputs an answer, and it scores +1 or -1.

Incremental Features in the Static Harness Era

The past six years have essentially been about stuffing more capabilities into this static framework:

• Few-shot (2020): Putting examples into context to aid learning
• Chain-of-Thought: Spreading computation across more tokens rather than predicting the answer directly
• WebGPT / Toolformer: Giving the model tool-calling ability, essentially a structured JSON object
• MemGPT: Enabling CRUD operations on the model's own context, rather than append-only
• Voyager: Solidifying and distilling tool chains back into the system prompt — the origin of the "skill" concept
• ReAct / Self-Refine / Reflection: Introducing multi-role agents with self-improvement through environmental feedback

Building on this, we get RLMs (Recursive Language Models) that can recursively spawn sub-agents, and primary orchestrator agents. YC calls this entire architecture — one that doesn't improve its system prompt or update the Harness itself — Harness V1: it includes agent specs, system prompts, turn and tool-call limits, tool lists, skill lists, sub-agent lists, plus a loop and a context compilation mechanism.

Self-Improving Harnesses: The Most Exciting Frontier

The real breakthrough of the past six months has been making the Harness itself learn — either learning the system prompt, or learning the Harness code itself, the latter being, as YC put it, "very trippy."

• DSPy (Demonstrate-Search-Predict): Uses a small set of examples and a genetic-programming-style candidate merge-and-evaluate process to learn the optimal system prompt
• Darwin Gödel Machine: Goes further by allowing the agent to modify the Harness code itself. It maintains an archive of agents (each a Harness + system prompt pair), samples, evaluates, and backfills in a loop — getting stronger over time
• Continual Harness: Introduces finer-grained memory categorization and DAgger-style online learning, updating weight files directly at test time — which YC sees as an extremely important research direction

DAgger (Dataset Aggregation) is an imitation learning algorithm proposed by Ross et al. in 2011. The core idea: collect the actual state distribution the model encounters at test time, and continuously append expert-annotated feedback to the training set — thereby closing the covariate shift gap between training distribution and real execution distribution. Traditional supervised learning trains once on offline data; once the model makes a mistake at test time, it enters states it has never seen, and errors cascade. DAgger's online iteration ensures the model is always corrected on the paths it actually takes. Continual Harness brings this idea into the agent setting: each test-time interaction becomes a new training signal, and weight files are updated directly during execution — enabling genuine "learn while running." This is fundamentally different from offline fine-tuning (SFT) and fixed in-context learning (ICL), which is why YC considers it an important research direction: it has the potential to fill the core gap in current LLM systems — rapid adaptation to new domains.

Darwin Gödel Machine takes its name from two references: Darwin's mechanism of evolutionary selection (continuous optimization through mutation and selection) and the Gödel Machine (Jürgen Schmidhuber's 2003 theoretical framework for self-modifying programs, which allows an agent to rewrite its own code given a formal proof of improvement). The real-world Darwin Gödel Machine achieves this idea pragmatically: it replaces strict formal proofs with an evolutionary loop of sampling, evaluation, and backfilling, enabling agents to continuously rewrite and improve their own Harness code without human intervention.

Prime Agent: LLM as Turing Machine, Harness as Von Neumann Computer

Prime Intellect's Seth offered an elegant analogy. A bare LLM is more like a Turing machine: a tape, a set of instructions, execute operations, output. A Harness-augmented system is more like a Von Neumann computer — capable of reading and writing to external storage, making it far more expressive on a class of problems.

Seth mapped context management to a computer's cache hierarchy: model weights are L1 (fastest but expensive to update), active input context is the middle layer, and filesystem/REPL/sub-agents are the L2/L3 scheduling state. Critically, once you exceed the context window, you can't just do compaction — you have to think about how to update and delete state. He calls this "agentic garbage collection."

Prime Agent scored around 78% on ARC-AGI with GPT series models and 95.5% with Opus. By comparison, some popular Harnesses (like Hermes Agent) "burned through $5,000 quickly with minimal performance gains" — underscoring the importance of cost-efficiency. Seth emphasized that long-horizon evaluation should look at the "practical plateau" — at what point does investing more test-time tokens yield only marginal returns. The team also ran a continuous seven-day experiment deploying 633 agents consuming 23 million output tokens in a Factorio-style setup. The most important finding: the model doesn't get stuck — it keeps advancing the tech tree.

Local Deployment and Engineering in Practice: OpenJarvis and QM

Beyond pure research, the event showcased two engineering directions.

Stanford's John presented OpenJarvis — a personal AI stack that runs almost entirely on local hardware. The core argument: local LLMs are only 6 to 12 months behind the frontier, and today's Qwen 3 32B-class models can approach what Claude Opus previously delivered. By using cloud LLMs to "optimize" the local stack's configuration (rather than bearing cloud costs at inference time), OpenJarvis achieves roughly an 800x cost reduction alongside significant latency improvements.

YC's own QM is an open-source agent Harness built for real office environments. It went through a complete evolution: from a "general agent" in early 2025, to integrating Slack and Cron, to becoming a dedicated assistant for each employee. QM's design philosophy is worth unpacking:

• Pull the "brain" out of the sandbox: All sessions are offloaded to Postgres for centralized management; the sandbox shifts from "the agent's home" to "an on-demand resource"
• Keep the Harness extremely thin: The core has only three tools — remote sandbox execution, object storage read/write, and publishing internal applications
• The grind tool: Sets a time or token budget for a task and prevents the agent from giving up too early — significantly improving research output quality

The QM team also candidly shared hard lessons learned: LLM-as-judge auto-improvement loops can produce a "protagonist syndrome" — the agent sees only "part of the elephant" and starts fixing things, so humans in the loop remain indispensable. Additionally, agents lack social context awareness, meaning privileged information can easily leak — which implies that "what information you can put in the brain is fundamentally bounded by how good your permission system is."

Qwen 3 32B is an open-source large language model released by Alibaba's Tongyi Qianwen team in 2025, with 32 billion parameters and support for a hybrid thinking mode (switchable between "thinking mode" and "non-thinking mode"). On multiple public benchmarks, its performance approaches the previous Claude Opus 3 level, while inference costs are a fraction of closed-source frontier models. For local deployment, the 32B parameter count runs in quantized form on a single or dual consumer/professional GPU (e.g., RTX 4090, A100), allowing individuals or small teams to access near-frontier model capability without relying on cloud APIs. OpenJarvis's core argument — "local LLMs are only 6 to 12 months behind the frontier" — found strong confirmation with this generation of models: as the capability gap between open-source and closed-source frontier models narrows rapidly, the cost and privacy advantages of local deployment increasingly dominate real engineering decisions.

Final Thoughts

The core signal from this event is clear: against a backdrop of exponentially improving model capabilities, the Harness is evolving from "scaffolding" into something more like an agent operating system. Whether it's Prime Agent chasing peak ARC-AGI scores, OpenJarvis optimizing for local cost savings, or QM targeting enterprise deployment, they're all answering the same question — how do you use a better foundation to unlock the "untapped intelligence" already inside the model?

For builders, one actionable takeaway: let your Harness run slightly ahead of the current model's capabilities in certain areas. That way, you can collect high-quality reasoning traces and use them to bootstrap yourself to a higher tier of performance.