Context Language Model

Context-aware generation requires the model not only to grasp what has changed, but also to express those changes in an appropriate tone and format while maintaining consistency with the existing documentation system.

Context Language Model trades more flexible context management for higher inference costs, and whether this trade-off is worthwhile depends on the task's sensitivity to context quality and the ability to bear inference costs.

Context Language Model was proposed by a research team from Meta and the University of Washington.

Under the Context Language Model paradigm, an agent can shorten an outdated tool output or replace an entire block of no-longer-important content with a brief note.

Compaction is passive, global, and indiscriminate, whereas self-editing is active, local, and selective.

Context Language Model disrupts prefix caching because it continuously modifies the historical content of the context, and once earlier content is edited, the cached prefix becomes invalid.

In a truck seal code tracking task, the summarization baseline hallucinated 11 out of 12 seal codes in the first summary and fabricated 14 out of 18 codes in the third summary; whereas Py CLM correctly reproduced all 18 codes in the same task.

For CLM to truly work, it is necessary to enable calling only one tool per round and generating a note after each result; otherwise, parallel tool calls will cause the context to overflow.

The location where the model edits the file in CLM affects prefix cache reuse: edits at the end of the file allow most of the cache to be reused, while editing from the middle causes content after that point to lose cache, increasing latency and cost.

CLM does not automatically reduce costs; in actual testing, Py CLM processed approximately twice the token volume as Py in one instance, and in another instance only 10% of the prompt came from the cache.

The 9-billion-parameter model in the paper lagged 6 points behind a clean summarization baseline before being trained with reinforcement learning.

At a 128,000-token budget, CLM and summarization perform roughly on par, with CLM's advantage primarily manifesting in small-budget scenarios such as 32,000 tokens.

CLM introduces a new prompt injection attack surface: injected malicious instructions can persist in the context file and survive across multiple rounds, even actively preventing themselves from being deleted, which the paper's authors list as an unresolved risk.

The paper constructs a small benchmark called ContextBench, on which most existing compression techniques fail directly.

On the BrowseComp-Plus benchmark, CLM paired with Qwen3 27B achieves 59.4%, approximately 11% higher than the best summarization solution, with about one-fifth less computation.

Context Language Model (CLM) is a paper proposed by Meta and the University of Washington, whose core idea is to let the model edit its own context window rather than perform a one-time summary.

CLM transforms the context from an append-only linear sequence into a readable and writable working memory, where the model can retain, shrink, delete, or rewrite any part of it.

CLM's implementation works by having the harness write the current context to a file before each request, with the model modifying that file by generating and running Python scripts, and the remaining content after modification becomes the next round's prompt.

The video author conducted experimental validation of the Context Language Model approach on their own DGX device using a local model, but this segment does not elaborate on the specific conclusions of the experiment.

The authors have released a CLM extension for Py that can be installed with a single command, and typing /CLM within Py opens a panel showing context size changes and edit comparisons.