Online Inference
An inference mode in which an AI model is embedded within a business pipeline to return predictions in real time for each individual request, as opposed to offline batch processing. This mode places strict requirements on inference latency.
Core Facts
Timeline (last 90 days)
Evaluation with a batch size of 1 is highly representative of interactive services and development/debugging scenarios, but does not apply to high-throughput offline batch processing.
Online inference pursues first-token latency and tail latency, requiring continuous batching, preemptive scheduling and fine-grained KV Cache paging management.
Online inference requires low latency and high concurrency, and vendors must reserve GPU resources to handle burst traffic, resulting in a higher unit cost; batch inference allows tasks to be queued and executed during off-peak periods, resulting in a lower unit price.
International card network standards typically require end-to-end decision latency for fraud detection to be below 100–200 milliseconds.
All Facts (4)
Online inference requires low latency and high concurrency, and vendors must reserve GPU resources to handle burst traffic, resulting in a higher unit cost; batch inference allows tasks to be queued and executed during off-peak periods, resulting in a lower unit price.
75%UnverifiedEvaluation with a batch size of 1 is highly representative of interactive services and development/debugging scenarios, but does not apply to high-throughput offline batch processing.
50%UnverifiedOnline inference pursues first-token latency and tail latency, requiring continuous batching, preemptive scheduling and fine-grained KV Cache paging management.
50%UnverifiedInternational card network standards typically require end-to-end decision latency for fraud detection to be below 100–200 milliseconds.
50%