CPU

Ollama can automatically detect and utilize the computing resources available on the machine, supporting scheduling across both GPUs and CPUs.

For modern CPUs and GPUs, memory bandwidth is often the true bottleneck for inference speed rather than the compute units' arithmetic throughput, which is why quantization can bring speed improvements.

When a model exceeds VRAM capacity, Ollama can automatically offload some model layers to system memory for CPU processing (offloading technology).

In traditional PC architecture, the CPU and GPU have their own separate memory pools, and data transfer between the two creates bandwidth bottlenecks.

Heterogeneous computing refers to combining processing units of different architectures—such as CPUs, GPUs, and dedicated accelerators—within the same system to dynamically allocate computing resources based on task characteristics.

The performance of CPU software rasterization rendering is 10–100 times (1–2 orders of magnitude) slower than a real GPU.

For most CPU inference workloads, small independent chunks paired with parallel worker processes are more efficient than large inference batches.

In scenarios of pure GPU inference combined with low concurrency, the CPU preprocessing time may be only a few tens of milliseconds or even lower, which is negligible relative to the entire request's generation time of several seconds

Pure CPU inference is typically 5–10 times slower than GPU inference.

A CPU is essentially a state machine: the current state is defined by register values and the program counter, and after each instruction executes, the state transitions to the next state according to the instruction's semantics.

The 35-watt power estimate in the article usually covers only standby or light-load states; when the model is actually performing inference, actual power consumption under high CPU/GPU load may rise to 60–100 watts or even higher, depending on the chip model and model scale.

Single-CPU training took about 20 minutes, and the agent has about 475,000 parameters.

Training was conducted entirely on CPU, because the developer found in testing that the GPU brought no speedup.

DeepInfo adopts a pipeline design in which the GPU and CPU work in alternation, achieving heterogeneous scheduling.

Unlike automatic prefetching by caches on CPUs/GPUs, data movement on Spyre must be inserted by the compiler at appropriate positions and synchronized with compute instructions.

With memory bandwidth unchanged, upgrading the CPU or increasing the number of cores will not substantially improve local large model inference speed.

The system executes all three multiplication operations involving frozen weights on the NPU, while keeping the training graph, adapters, and optimizer on the CPU.

The runtime system fuses the remaining CPU operators and overlaps NPU execution with adapter computation, thereby improving overall throughput.

In all CPU runs, the statement permutation method achieved byte-for-byte peak memory and bit-for-bit identical outputs.

On PyTorch Inductor, 22 of the 54 CPU configurations had actual peak memory deviate from the predicted value, with 9 of those configurations even performing worse than the original program.