Streaming
A transmission method in which a large language model generates a response token by token and pushes it to the client in real time. It substantially reduces time to first token and improves the experience of chat-based interaction.
Timeline (last 90 days)
Streaming can significantly improve perceived user experience by reducing time-to-first-byte, even if the total generation time remains unchanged.
Streaming allows the server to progressively push tokens to the client during generation, achieving a typewriter-like character-by-character appearance effect, thereby reducing perceived wait time.
Streaming responses push model tokens to the client incrementally via Server-Sent Events or WebSocket, which can improve perceived latency.
When streaming output is interrupted, it is acceptable for the streaming output itself to be aborted, but the entire conversation history must be fully preserved, requiring an independent session storage mechanism to be maintained concurrently with the streaming transmission
流式响应的正确计费实现应只在流结束时根据最终usage字段一次性计费，或只累加每个chunk的增量Token数
All Facts (5)
Streaming can significantly improve perceived user experience by reducing time-to-first-byte, even if the total generation time remains unchanged.
50%UnverifiedStreaming allows the server to progressively push tokens to the client during generation, achieving a typewriter-like character-by-character appearance effect, thereby reducing perceived wait time.
50%UnverifiedStreaming responses push model tokens to the client incrementally via Server-Sent Events or WebSocket, which can improve perceived latency.
50%UnverifiedWhen streaming output is interrupted, it is acceptable for the streaming output itself to be aborted, but the entire conversation history must be fully preserved, requiring an independent session storage mechanism to be maintained concurrently with the streaming transmission
50%Unverified流式响应的正确计费实现应只在流结束时根据最终usage字段一次性计费，或只累加每个chunk的增量Token数
50%