Tensor Core
Tensor Core is a dedicated hardware compute unit introduced by NVIDIA in its GPU architectures, designed for efficient execution of matrix multiply-and-accumulate (MMA) operations. Its key characteristic is support for mixed-precision computation across data types such as FP16, BF16, and INT8, enabling multiple matrix operations to be completed within a single clock cycle and significantly increasing throughput for deep learning training and inference. Tensor Cores are widely used to accelerate neural network model computation and represent a core hardware foundation for modern AI workloads.
Core Facts
The NVIDIA H100 single-card FP64 double-precision floating-point performance reaches approximately 34 TFLOPS.
NVIDIA introduced Tensor Cores starting with the Volta architecture, which are hardware units specifically designed for matrix multiply-accumulate operations.
Efficient operation of Tensor Cores requires matrix dimensions to be aligned to multiples of 16 or 8.
Timeline (last 90 days)
Organizing neighboring atoms into regular tensor batches and aligning matrix dimensions to shapes preferred by Tensor Cores is the first step in improving MLIP inference throughput.
MLIP inference workloads are essentially stacks of large numbers of small-scale matrix operations and tensor contractions, naturally suited to Tensor Cores, but require targeted design of data layout, batching dimensions and kernel fusion.
The H100 supports fourth-generation Tensor Cores and the Transformer Engine.
Magnitude's CUDA kernel autotuning activates the RTX 4090's Tensor Cores for INT4 operations, which is one of the keys to its doubled throughput.
CUTLASS encapsulates shared memory tiling, register-level data pipelining, Tensor Core scheduling, and asynchronous memory copying in CUDA programming into hierarchical, composable templates.
Pascal架构缺乏Tensor Core，对FP16和INT8的加速支持有限
The Streaming Multiprocessor (SM) is the core compute unit of NVIDIA GPUs; each SM contains a number of CUDA cores, Tensor Cores, a register file, and shared memory, and is responsible for executing thread blocks in parallel.
What affects GPU/TPU computational efficiency is whether the dimension is divisible by 8, 16, 32, 64, etc., in order to leverage specialized compute units such as Tensor Cores.
Compared to general-purpose CUDA Cores, Tensor Cores can deliver dozens of times higher throughput when executing neural network inference.
NVIDIA delegates the DLSS inference process to the Tensor Cores on RTX graphics cards and schedules it through a proprietary driver.
10 more timeline events
All Facts (20)
The NVIDIA H100 single-card FP64 double-precision floating-point performance reaches approximately 34 TFLOPS.
90%VerifiedNVIDIA introduced Tensor Cores starting with the Volta architecture, which are hardware units specifically designed for matrix multiply-accumulate operations.
85%VerifiedEfficient operation of Tensor Cores requires matrix dimensions to be aligned to multiples of 16 or 8.
80%VerifiedThe Tensor Core of the A100 can complete one 4×4 matrix multiply-accumulate operation within a single clock cycle.
80%VerifiedTensorRT unleashes GPU hardware potential through operator fusion, kernel auto-tuning, and memory optimization.
80%VerifiedGeneral Matrix Multiplication (GEMM) is the most fundamental and computationally intensive operation in deep learning inference, and NVIDIA Tensor Cores and Google TPU systolic arrays are both specialized optimizations targeting GEMM.
75%VerifiedRDNA 2 is a GPU graphics architecture launched in 2020 that introduced hardware ray tracing support for the first time and was used in the RX 6000 series desktop graphics cards and the custom APUs in the PS5 and Xbox Series X.
75%VerifiedIn mixed-precision training, the forward pass uses FP16 to leverage Tensor Core acceleration while parameter updates retain FP32 to maintain numerical stability, which has become standard practice for large model training.
75%VerifiedThe NVIDIA A100 has 6,912 CUDA cores and 432 Tensor Cores, with a theoretical peak compute of 312 TFLOPS (BF16).
75%VerifiedChip manufacturers such as Qualcomm and MediaTek have integrated dedicated NPUs (Neural Processing Units) into mobile SoCs.
70%VerifiedThe RTX 5090 is based on NVIDIA's Blackwell architecture.
65%VerifiedAda Lovelace架构（RTX 40系列）具备FP8 Tensor Core支持
65%VerifiedNVIDIA's DLSS relies on Tensor Cores in the GPU for AI inference acceleration and can only be used on RTX series graphics cards.
65%UnverifiedConverting from FP32 to FP16 can achieve a 2–4x throughput improvement on the Tensor Cores of modern GPUs.
70%UnverifiedThe design of CUDA Tile naturally aligns with the working mechanism of NVIDIA Tensor Cores, which themselves execute matrix multiply-accumulate (MMA) operations in tile units.
70%UnverifiedThe Blackwell architecture's 5th-generation RT Core improves ray tracing performance by approximately 2x, and Tensor Core support for FP4 precision doubles AI inference throughput.
60%UnverifiedMLIP inference workloads are essentially stacks of large numbers of small-scale matrix operations and tensor contractions, naturally suited to Tensor Cores, but require targeted design of data layout, batching dimensions and kernel fusion.
50%UnverifiedOrganizing neighboring atoms into regular tensor batches and aligning matrix dimensions to shapes preferred by Tensor Cores is the first step in improving MLIP inference throughput.
50%UnverifiedThe H100 supports fourth-generation Tensor Cores and the Transformer Engine.
50%UnverifiedMagnitude's CUDA kernel autotuning activates the RTX 4090's Tensor Cores for INT4 operations, which is one of the keys to its doubled throughput.
50%