Inference

NVIDIA has long dominated the GPU computing market required for AI training and inference.

The essence of an agent is to equip a model with the capabilities of planning, tool calling, multi-step reasoning, and environment interaction.

As inference request volumes surge with the spread of applications, small- and medium-scale deployers will choose consumer-grade flagship cards to fill the gap.

Training requires massive memory and inter-GPU interconnect bandwidth, making reliance on professional accelerator cards almost inevitable; inference has lower interconnect requirements and places more emphasis on single-card memory capacity and unit cost.

When the same H100 GPU executes the same inference request and outputs the same number of tokens, the power consumption is basically the same.

Model training is one-time, but inference occurs continuously with user requests, so inference costs often account for the vast majority of total AI costs.

Inference is a machine learning term referring to the process of using a trained model to perform computation on new input data and produce results, as distinct from the time-consuming training phase.

Inference refers to the computation process in which a loaded model generates an answer token by token based on an input prompt, unlike training; when run locally, inference is performed on the user's CPU or GPU.

Pretraining and inference differ fundamentally in engineering requirements: inference is forward computation, tolerates high numerical precision loss, can be quantized to INT8 or even INT4, and can be compiled offline into a fixed computation graph; pretraining requires full forward and backward propagation, optimizer state maintenance, and cross-device gradient synchronization.

Sending a chat message does not immediately retrain the model; training repeatedly adjusts model parameters through large amounts of data, takes weeks or even months, and is usually done offline; inference, by contrast, uses the already-trained fixed parameters to process new input.

Training refers to repeatedly iterating model parameters with large-scale data, an offline, long-cycle task; inference is the forward computation process by which the model responds to each user request after going live, a real-time, highly concurrent online task.

Training can be completed concentrated in a few supercomputing clusters, but inference is an ongoing expense, and lowering the cost per token of inference helps seize the initiative in AI product gross margins.

The industry widely believes that inference will become the main body of data center computing power spending.

Online real-time inference is typically a low-latency task involving small batches or single requests, where the bottleneck is often memory bandwidth rather than compute.

The basic form of reasoning is 'if–then': if x is true, then y must be true.

For pure inference scenarios, model weights typically only need to be loaded once, and the constraint of data gravity is far less significant than in the training phase.

Inference is the process by which a deployed model actually handles user requests, and is a long-term, ongoing cost that scales linearly with user volume.

Training prioritizes maximizing throughput, whereas inference is extremely sensitive to latency and faces unpredictable traffic fluctuations.

High throughput is achieved through batching, which increases the total number of tokens processed per unit of time but raises time-to-first-byte latency; low latency requires small batches or no batching, which reduces hardware utilization and raises the cost per token.

The target customers for Apple's server product are enterprises that want to run AI models on their own devices, with the focus placed on the inference stage.