ITL
A key metric for measuring user experience in LLM serving, referring to the time interval between two consecutive output tokens. It is mainly determined by the decode phase and is affected by memory bandwidth, batch size and the volume of KV cache reads.
Timeline (last 90 days)
The Decode phase is memory-bandwidth-intensive, requiring model weights and the KV cache to be reloaded from HBM at each step, which determines inter-token latency (ITL).
When prefill and decode run on the same engine, a long prompt arriving mid-flight causes every sequence in the decode batch to experience an ITL spike whose duration roughly equals the time taken by that prefill step.
Inter-token latency (ITL) is the time interval between adjacent tokens during the generation process and determines the smoothness of streaming text output.
Deterministic execution architecture minimizes variance in time-to-first-token (TTFT) and inter-token latency (ITL), enabling SLA-like latency commitments to be offered externally.
TTFT determines how long a user waits from sending a request to seeing the first token, while ITL determines the pace at which subsequent content is generated.
All Facts (5)
The Decode phase is memory-bandwidth-intensive, requiring model weights and the KV cache to be reloaded from HBM at each step, which determines inter-token latency (ITL).
50%UnverifiedWhen prefill and decode run on the same engine, a long prompt arriving mid-flight causes every sequence in the decode batch to experience an ITL spike whose duration roughly equals the time taken by that prefill step.
50%UnverifiedInter-token latency (ITL) is the time interval between adjacent tokens during the generation process and determines the smoothness of streaming text output.
50%UnverifiedDeterministic execution architecture minimizes variance in time-to-first-token (TTFT) and inter-token latency (ITL), enabling SLA-like latency commitments to be offered externally.
50%UnverifiedTTFT determines how long a user waits from sending a request to seeing the first token, while ITL determines the pace at which subsequent content is generated.
50%