GPU
A Graphics Processing Unit (GPU) is a specialized parallel computing processor originally designed to accelerate computer graphics rendering by simultaneously handling multiple tasks across a large number of small computational cores. Its highly parallel architecture provides significant performance advantages in graphics display, scientific computing, deep learning training and inference, and data processing, and it is widely deployed in personal computers, servers, and data centers.
Core Facts
Continuous batching allows new requests to be inserted and completed requests to be removed at any iteration step, keeping the GPU running at full load at every step.
NVIDIA CUDA was launched in 2006 and was originally a programming framework for enabling developers to use GPUs for general-purpose computing.
The pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
Timeline (last 90 days)
To support AI infrastructure (data centers, TPU and GPU purchases, power and cooling), Alphabet's capital expenditure has risen sharply in recent years.
In AI training clusters, the communication topology between GPUs is often organized hierarchically by rack, node, and supernode.
When a new generation of GPUs is significantly better than old cards in energy efficiency, the electricity cost and rack-space opportunity cost of continuing to run old cards will exceed their output value, causing old cards to be "squeezed out."
The 35-watt power estimate in the article usually covers only standby or light-load states; when the model is actually performing inference, actual power consumption under high CPU/GPU load may rise to 60–100 watts or even higher, depending on the chip model and model scale.
The Neocloud business model relies heavily on debt or asset-backed financing to purchase GPUs, then locks in revenue with multi-year contracts.
When large compute rental companies sell off equipment in concentrated fashion, the short-term supply curve shifts right and prices often overshoot; however, second-hand cards face issues such as voided warranties, driver support cycles, and lagging energy efficiency, so the buyer pool is limited and liquidity is far lower than for new cards.
Some users on Reddit judge that GPU prices may drop significantly in the coming months, and use "dirt cheap" to describe next year's second-hand and newly built compute market.
High-frequency, low-complexity requests account for the vast majority, and only a few tasks require top-tier compute; this structural change means compute demand no longer grows in sync with model parameter counts.
Second-hand GPU prices are affected by three factors: depreciation speed, generational turnover, and supply concentration.
Unlike the general-purpose parallel computing of GPUs, the neural engine adopts a more specialized dataflow architecture and can deliver higher inference throughput at the same power consumption.
40 more timeline events
All Facts (20)
Continuous batching allows new requests to be inserted and completed requests to be removed at any iteration step, keeping the GPU running at full load at every step.
90%VerifiedNVIDIA CUDA was launched in 2006 and was originally a programming framework for enabling developers to use GPUs for general-purpose computing.
90%VerifiedThe pre-training process of large language models requires thousands of high-end GPUs running in parallel for months, with costs typically ranging from tens of millions to hundreds of millions of dollars.
90%VerifiedTraining a model at the level of GPT-4 requires tens of thousands of A100/H100 GPUs working in concert for several months.
90%VerifiedThe HBM bandwidth of the A100 GPU is approximately 2 TB/s, while the on-chip SRAM bandwidth can reach 19 TB/s, a difference of nearly 10x.
85%VerifiedThe NPU is a hardware accelerator designed specifically for AI inference computation; the Hexagon NPU in Qualcomm's Snapdragon 8 series, Apple's Neural Engine, and MediaTek's APU have all been integrated into mobile chips.
80%VerifiedEfficient operation of Tensor Cores requires matrix dimensions to be aligned to multiples of 16 or 8.
80%VerifiedDuring large model inference, actual GPU utilization is often less than 10%.
80%VerifiedOllama can automatically detect and utilize the computing resources available on the machine, supporting scheduling across both GPUs and CPUs.
80%VerifiedThe massively parallel computing architecture of GPUs is naturally suited for matrix operations in deep learning, and their thousands of computing cores can accelerate the training process by tens to hundreds of times.
80%VerifiedTensorRT unleashes GPU hardware potential through operator fusion, kernel auto-tuning, and memory optimization.
80%VerifiedTensorRT's operator fusion merges multiple consecutive small operators in a computation graph into a single large operator, reducing the number of GPU kernel launches and memory reads/writes of intermediate results; a typical fusion operation can combine convolution, batch normalization, and activation function into a single GPU kernel call.
80%VerifiedTraining a frontier large model requires hundreds of millions of dollars in compute, data cleaning, and human labor.
80%VerifiedMixture of Experts (MoE) is a model architecture composed of multiple expert sub-networks that each specialize in different sub-tasks, together with a gating network that decides which expert receives a given input.
75%VerifiedThe full-stack GPU paradigm keeps environment state, dynamics simulation, policy network forward passes, and gradient computation entirely within video memory, eliminating the data transfer bottleneck between CPU and GPU.
75%VerifiedNVIDIA has long dominated the GPU computing market required for AI training and inference.
75%VerifiedThe bottleneck of traditional autoregressive decoding lies in memory bandwidth rather than compute capability, resulting in low GPU utilization.
75%VerifiedThe bottleneck in large model inference is often not compute but memory bandwidth, and GPU compute utilization during the inference phase is often less than 10% of peak.
75%VerifiedFor technically capable users, fully local deployment on their own GPUs using tools such as vLLM and llama.cpp often yields a per-token cost far lower than cloud-based paid solutions in long-term, large-scale usage scenarios.
75%VerifiedCPU-GPU hybrid inference (offloading) introduces a significant drop in inference speed.
75%