cuBLAS

The CUDA Toolkit includes the compiler nvcc, debuggers, performance analysis tools, and math libraries such as cuBLAS and cuFFT.

CUDA（Compute Unified Device Architecture）是NVIDIA推出的并行计算平台和编程模型

The CUDA ecosystem encompasses core components such as cuDNN, cuBLAS, and NCCL; mainstream frameworks including PyTorch and TensorFlow all use CUDA as their primary acceleration backend.

The barriers of the CUDA ecosystem are deeply rooted in operator libraries (cuBLAS, cuDNN), compilation optimization toolchains (NVCC, TensorRT), and distributed training frameworks (NCCL), forming NVIDIA's deep moat.

Since its release in 2007, CUDA has accumulated nearly two decades of development, with its moat reflected in low-level optimization libraries such as cuDNN, TensorRT, and NCCL.

CompileIQ 可用于优化 cuBLAS、cuDNN、CUTLASS 等底层库的内核级编译

Setting CUBLAS_WORKSPACE_CONFIG=':4096:8' forces cuBLAS to allocate a deterministic workspace; this configuration was introduced starting from CUDA 10.2.

rocBLAS is AMD's BLAS library (counterpart to NVIDIA cuBLAS), providing basic linear algebra routines such as matrix multiplication; composable_kernel (CK) is AMD's high-performance operator template library, commonly used to implement fused operators and low-precision GEMM.

CUDA's moat includes optimized libraries such as cuBLAS, cuDNN, and NCCL, the Nsight series of profilers and debuggers, and more than a decade of accumulated tutorials, Q&A, and third-party framework adaptations.

The request bytes the author initially measured exceeded the theoretical formula by a fixed offset of 8,519,688 bytes, of which 8,519,680 bytes come from the workspace lazily allocated by cuBLAS and the remaining 8 bytes are the next-token tensor.

The cuBLAS workspace is a GPU memory region reserved by cuBLAS for temporary results or internal algorithms, whose size is determined by the CUBLAS_WORKSPACE_CONFIG environment variable or the library's default policy, and which is usually allocated only on the first call to the relevant kernel; it is not a user tensor but is counted in the memory pool managed by PyTorch's caching allocator.

Achieving bit-wise determinism in LLM inference requires fixing all reduction orders, disabling non-deterministic GPU operators (such as some cuBLAS/cuDNN kernels and atomic adds), disabling dynamic batching, and padding inputs to a fixed length; these constraints significantly reduce throughput and memory efficiency.

From an engineering perspective, training a production-grade model completely without libraries is almost unrealistic, because it requires low-level dependencies such as CUDA, cuBLAS, and distributed communication libraries.

When torch.matmul is called in PyTorch, what actually executes is often a highly tuned cuBLAS kernel.

OjasX is called open-source CUDA by developers, but its logic is to reduce the amount of data that needs to reside in memory, rather than to beat cuBLAS in pure compute power.

Common reference baselines in the kernel optimization field include NVIDIA's cuBLAS/cuDNN, code generated by Google's XLA compiler, and hand-written Triton kernels.

NVIDIA's official cuBLAS has long been the de facto standard for GPU matrix operations.

DeepGEMM corresponds to cuBLAS, responsible for optimizing the matrix multiplication calculations that are core to AI.

Tile blocking maximizes data reuse and hides memory access latency by dividing large computational tasks into small blocks that fit in shared memory or registers, and is a key technique by which high-performance libraries such as cuBLAS and FlashAttention approach peak hardware performance.

The reconstructed ThunderKittens kernels can now compete head-to-head with officially highly optimized solutions such as cuBLAS and the CuTe DSL.