Prefill

Time to First Token (TTFT) refers to the time interval between a user sending a request and the model outputting the first token, and is primarily constrained by the duration of the prefill stage.

The KV Cache mechanism can cache repeated prefix content, and the more concise the system prompt, the higher the cache hit rate.

LLM 推理分为预填充阶段（Prefill，计算密集型）和解码阶段（Decode，内存带宽密集型）两个阶段

The original configuration's prefill speed was below 200 tokens/second, partly because the micro-batch size was set to 128; after raising the micro-batch to 1024 and moving some expert layers back to the GPU, prefill speed increased to about 800 tokens/second.

Prefilling refers to when, in using a large language model API, the caller pre-provides a piece of text (or token sequence) for the model to continue generating from, rather than starting from an empty context.

Prefilling refers to pre-filling the beginning of the assistant's reply in a conversation to guide the model to continue in a particular direction.

Without prefilling, this abnormal behavior is rare, but it can be reliably reproduced with a specific prompt combined with prefilling.

The prefill stage is compute-intensive and the decode stage is memory-intensive, and running them together causes mutual interference, leading to latency jitter and wasted resources.

Prefill is the first stage of large language model inference, in which the model reads the entire prompt entered by the user in one pass and computes key-value vectors for each token and stores them in VRAM.

Prompt caching can reuse computation results for long contexts that recur, significantly reducing duplicate billing.

When prefill and decode run on the same engine, a long prompt arriving mid-flight causes every sequence in the decode batch to experience an ITL spike whose duration roughly equals the time taken by that prefill step.

A machine without a dedicated GPU cannot ingest large prompts in a reasonable time, so a second host with a GPU is needed to solve the prefill problem.

When multiple models show anomalies at the same time, it is more likely that the routing, scheduling, authentication, or rate-limiting components they share experienced fluctuations, rather than that a single model's weights or inference logic failed individually.

Prefill is a compute-intensive operation where computation grows quadratically with sequence length; decoding is a memory-bandwidth-intensive operation that can reuse the KV Cache.

The prefill stage is a compute-intensive operation primarily limited by GPU compute, while the decode stage is a memory-bandwidth-intensive operation primarily limited by VRAM bandwidth.

LLM inference requests are divided into two phases—Prefill (input) and Decode (output); Prefill can read inputs in parallel with extremely high throughput, while Decode requires token-by-token serial inference and occupies significantly more GPU time.