Context length
The upper limit on the number of tokens a large language model can process in a single inference pass. It determines how much contextual information the model can reference and is a key factor affecting the behavior and effectiveness of AI programming tools.
Timeline (last 90 days)
Context length refers to the upper limit on the number of tokens a model can process at once, and when running locally, the longer the context, the more memory is used.
Context length remains a lever for differentiated competition among models.
When the context extends to 100k or even longer, the per-token price, memory usage, and inference latency all change, and "performance per dollar" conclusions under short contexts cannot be directly extrapolated.
One comment questioned, "What about over 100k context? This chart is misleading," arguing that short-context pricing cannot reflect the real cost in long-context scenarios.
A common cause of Codex getting stuck in a "reconnecting" loop at runtime is that Ollama's default context length is 32K; changing it to 16K or 8K can improve agent speed.
一个7B模型在32K上下文下，内存占用可能接近一个更大模型在短上下文下的水平。
The longer the context length, the greater the VRAM and memory usage.
The context length of traditional models is usually only 4K to 32K; beyond that, the model forgets.
That the model cannot understand overly complex structures, or performs worse when the context is too long, is today's capability floor; this method cannot yet run in all scenarios, but that floor is not the method's permanent boundary.
Context length refers to the number of tokens a model can remember at one time, and a span from 4000 to 64000 means that long-document processing capability and VRAM usage will expand simultaneously.
3 more timeline events
All Facts (13)
Context length refers to the upper limit on the number of tokens a model can process at once, and when running locally, the longer the context, the more memory is used.
50%UnverifiedContext length remains a lever for differentiated competition among models.
50%UnverifiedWhen the context extends to 100k or even longer, the per-token price, memory usage, and inference latency all change, and "performance per dollar" conclusions under short contexts cannot be directly extrapolated.
50%UnverifiedOne comment questioned, "What about over 100k context? This chart is misleading," arguing that short-context pricing cannot reflect the real cost in long-context scenarios.
50%UnverifiedA common cause of Codex getting stuck in a "reconnecting" loop at runtime is that Ollama's default context length is 32K; changing it to 16K or 8K can improve agent speed.
50%Unverified一个7B模型在32K上下文下，内存占用可能接近一个更大模型在短上下文下的水平。
50%UnverifiedThe longer the context length, the greater the VRAM and memory usage.
50%UnverifiedThe context length of traditional models is usually only 4K to 32K; beyond that, the model forgets.
50%UnverifiedThat the model cannot understand overly complex structures, or performs worse when the context is too long, is today's capability floor; this method cannot yet run in all scenarios, but that floor is not the method's permanent boundary.
50%UnverifiedContext length refers to the number of tokens a model can remember at one time, and a span from 4000 to 64000 means that long-document processing capability and VRAM usage will expand simultaneously.
50%UnverifiedContext length refers to the total number of tokens a model can accommodate in a single request, including input and output; when the upper limit is exceeded, different services handle it differently, with some returning an error directly and others silently truncating.
50%UnverifiedVRAM consumption for context length grows approximately quadratically as length increases, because the attention mechanism must maintain a KV Cache storing the intermediate states of historical tokens
50%UnverifiedInference speed drops sharply once the context exceeds 128K, due to the ballooning KV cache size and the quadratic growth of attention computation time with sequence length
50%