Larger Context Windows Make AI Agents Less Reliable? 4 Failure Modes and How to Fix Them

Over the past year, AI agents' context windows have ballooned from a few thousand tokens to hundreds of thousands or even millions of tokens. Intuitively, this means an agent can remember more instructions, more tool results, and more project documentation, so long-running tasks should be completed more reliably. But a study covering 18 frontier models reached the opposite conclusion: as context windows grow larger, all models become less reliable. The researchers call this phenomenon "context rot"—the more tokens in the window, the harder it is for the model to reliably use any of them.

This leads to a counterintuitive proposition: how could an agent become better by "remembering less"? The answer lies in context engineering—at every step of an agent's run, actively deciding what should enter the context and what should be discarded.

What Actually Fills the Context Window

To understand the failures, we first need to see the fillers clearly. Take an agent tasked with "migrating an old codebase to a new database driver" as an example:

• System prompt: placed at the very front of the window, defining long-term instructions for how it should work overall;
• Task description: possibly just a few lines, telling it the migration goal;
• Project instructions: files like agents.md that the agent framework automatically reads in, explaining how the project is built;
• Tool definitions: the names and descriptions of every callable tool;
• Run logs: what the agent did at each step, dominated by tool results—file contents read, test output, error logs, database schema.

An agent that has been running for an hour may already have a fairly large message history. The task description itself occupies only a tiny fraction; almost everything else accumulates bit by bit during execution. This is the structural root of the problem: the window keeps expanding, but the proportion of content actually relevant to "what to do next" keeps getting diluted.

More critical is the model's attention mechanism. A model does not process context like a database query that can precisely locate one row and read only that record—every token in the window competes for limited attention. As the original video points out: the window may hold a million tokens, but the total amount of attention is finite and must be spread across all of them.

Context window refers to the upper limit of tokens a model can read in a single inference pass, determining the total capacity of all inputs such as the system prompt, conversation history, and tool results. Note that window size is not equivalent to how much information the model can effectively use: the window is the physical capacity of "how much it can hold," while attention is the computational resource of "how much it can focus on." As the number of tokens in the window grows, attention weights must be distributed among them, and the average attention each token receives declines. This is why expanding the window does not linearly improve an agent's task capability—capacity expands, but effective attention is spread thinner. Understanding this distinction is a prerequisite for understanding the "context rot" phenomenon.

Four Failure Modes: Pollution, Distraction, Confusion, Conflict

Context Pollution: Wrong Conclusions Treated as Facts

Return to the migration case. The agent skims the driver documentation, misreads it, and writes a note: "The new driver automatically retries failed connections, so the old retry code can be deleted." This conclusion is completely wrong, but it enters the context and is treated by the model as a reliable basis—to the model, its own notes look no different from facts. The agent then starts deleting perfectly correct retry code file by file. Once an error enters the context, it spreads like a contaminant into all subsequent decisions.

Attention mechanism is the core component of the Transformer architecture, allowing the model to assign weights to all other tokens in the context based on relevance when processing each token. This mechanism is global and soft: the model cannot precisely "read only one row" like a database, but performs a weighted sum over everything in the window. Therefore, when incorrect or outdated information is mixed into the context, it is not automatically masked; instead, it participates with some weight in the computation of all subsequent tokens. The more tokens there are, the more severely attention is diluted, and the lower the relative weight that key instructions and facts receive. This explains at the mechanistic level why "stuffing more in" does not equal "using it better," and provides a theoretical basis for the "actively discard" remediation strategies discussed later.

Context Distraction: History Overrides the Current Task

Language models are essentially machines for continuing existing patterns. If the window has accumulated an hour of past operations, this history becomes a very strong pattern that may even override the reasoning ability the model learned in training. It may stop asking "what does this task truly need next" and instead be led along by its own message history, repeatedly doing what it has already done, going in circles on the old path.

Context Confusion: Too Many Tools, Wrong Choice

Agents are often connected to multiple systems—version control, database consoles, ticketing systems. Whether or not these tools will actually be used, their definitions must be stuffed into the context. The more types of tools, the harder it is to select the one best suited to the current task. The original video cites a benchmark: the model failed the task when provided with 46 tools, but passed the same task when the tools were reduced to 19.

Context Conflict: Two Versions Coexist

Sooner or later the agent discovers that retries are not happening, realizes the note was wrong, and may write the correct situation into the message history. The problem is that two conflicting versions now coexist in the context, and the model must judge between contradictory information on its own, with reliability declining accordingly.

Four Fixes: Actively Throw Some Information Away

For the four failure modes, the original video offers corresponding remediation paths, all sharing the same theme: active deletion.

Verify: Check Before Conclusions Enter the Context

To counter pollution, the fix is to run a simple verification against real information before a conclusion is written into the context. For example, running a connection test once would have prevented the agent from writing that wrong note, "the new driver requires no retry code." The cost of verification is far lower than the cost of rolling back file by file after contamination.

Compress: Summaries Instead of Full Logs

To counter distraction, the key is compression. When history becomes especially long, first summarize it, then discard the full logs. But compression is lossy: some details in the summary disappear, and summarizing a summary causes information to drift gradually, like a photocopy of a photocopy. A more robust approach is to save content that can be removed from the context into long-term files—detailed material stays in files and is re-read only when the task truly needs it.

Long-term files here refer to persistent storage maintained by the agent framework outside the context window, such as note files in the project, progress documents, or structured state records. Unlike the message history inside the window, long-term files are not constrained by the token limit and do not continuously consume attention resources. The agent can read a fragment from them via a tool call when needed and then not keep it in the context. This "external memory" pattern separates "storage" from the "current working set": detailed logs remain in files as a searchable archive, while only the small amount of information truly needed for current decisions stays in the window. It complements the compression strategy—summaries compress history within the window, while long-term files move out details that should not be compressed.

Filter: Provide Only the Tools Actually Needed Now

To counter confusion, this is an information organization problem. A database migration task does not need 46 tools; providing only the few truly needed at the moment is enough; when needed, the agent can dynamically fetch other tools. Trimming the toolset by task stage is more reliable than mounting everything at once.

Clear: Old Versions Must Not Coexist with New Conclusions

To counter conflict, when information is superseded by a new conclusion, delete the old version—do not let both coexist. You can go further and open a completely fresh window containing only the current correct state, leaving previous incorrect or outdated information outside.

Why "Remembering Less" Is Actually More Reliable

Looking at these four fixes together, the logic is consistent:

Failure Mode	Symptom	Fix
Pollution	Wrong conclusions treated as facts	Verify
Distraction	History overrides the current task	Compress
Confusion	More tools than the task needs	Filter
Conflict	Context contradicts itself	Clear

All four fixes share the same action: discarding. This challenges the default assumption that "more information is better": given limited total attention, irrelevant or outdated tokens are liabilities rather than assets—they dilute the model's attention allocation to key information. The core of context engineering is not stuffing more into the window, but continuously judging what should not be in the window at this moment.

For developers, this means the design focus of agent frameworks will shift from "how to read in more context" to "how to remove context at the right moment." Dynamic loading of tool calls, layered strategies for history summarization, and verification steps before writing conclusions will all become key engineering points for agent reliability.

One-line Summary

Larger context windows make agents less reliable; four failure modes require active discarding to fix.

Paragraph Summary

The article presents a study covering 18 frontier models: the larger the context window, the less reliably the model uses the information within it—a phenomenon researchers call "context rot." The article analyzes the window's filler structure—tool results and run logs dominate, while the task description occupies a tiny fraction, diluting key information—and explains the mechanistic root from the perspective of limited total attention. On this basis it identifies four failure modes: pollution (wrong conclusions treated as facts), distraction (historical patterns overriding the current task), confusion (too many tools to choose from), and conflict (contradictory old and new information coexisting). It gives four corresponding fixes—verify, compress, filter, clear—all sharing the theme of actively discarding part of the context. The article concludes that the core of context engineering is not stuffing in more, but removing irrelevant or outdated information at the right time.