Rust Framework DenseCore Achieves Bit-Exact Parity with PyTorch, Zero-Error nanoGPT Training

A CS student on Reddit showcased his deep learning framework DenseCore, written in Rust, claiming that when running Karpathy's nanoGPT on an Apple Silicon Mac, its training results are bit-for-bit identical to PyTorch — same initial weights, losses, gradients, weight updates, fully consistent for 50 consecutive steps, even with dropout enabled. This is not "approximate matching" but a literal bit-level reproduction.

Engineering Challenges of Bit-Exact Reproduction

Achieving this goal is far from trivial. The author dug deep into PyTorch's CPU implementation details and discovered many designs that differ from textbook formulas. For example, PyTorch's CPU attention mechanism is not the standard formula but a flash attention kernel, and it switches to a different path when dropout is enabled. The Adam optimizer rounds differently on the last few elements per thread block, due to the compiler's fusion of tail loops. clip_grad_norm_ internally actually performs reciprocal() * max_norm rather than division. These subtle differences are everywhere, and any single mismatch breaks bit-exact consistency.

More complex is that PyTorch itself cannot guarantee bit-exact consistency across different machines. The author tested 26 operators and found that the same version on Mac and Linux x86 produced identical bits for only 1 operator. Therefore, DenseCore can currently only match PyTorch on Apple Silicon Macs.

What is bit-exact?

Bit-exact means two computations produce exactly the same bits at the floating-point level. Although floating-point operations (IEEE 754) are standardized, the actual execution order differs — for example, reduction order, vectorization width, use of fused multiply-add (FMA) instructions — all change the rounding accumulation, leading to differences in the last bits. In deep learning training, forward pass, backward pass, and optimizer updates are interlinked; any deviation in one step is amplified. Thus, "50 consecutive steps with identical loss, gradients, and weights" means the entire computation graph is aligned with PyTorch's CPU implementation at the operator level. This is precisely the technical difficulty of this project: it is not "numerically close" but requires every operator and every loop's floating-point summation order to match PyTorch exactly.

Why is PyTorch itself inconsistent across platforms?

PyTorch's CPU operators heavily rely on underlying backends (such as oneDNN, MKL, Apple Accelerate) and SIMD instruction sets (AVX2, AVX-512, NEON). Different platforms have different vector widths, which change the blocking of reductions; different compiler optimizations (such as auto-vectorization, loop unrolling, FMA fusion) also change the rounding path of intermediate results. This explains the author's observation that "only 1 out of 26 operators is bit-identical on Mac and Linux x86" — most operators' floating-point summation orders are not consistent across the two. Therefore, bit-exact consistency is inherently platform-bound; DenseCore's choice to align only with PyTorch on Apple Silicon Macs is a reasonable trade-off for the consistency goal.

Lightweight Potential: 2MB vs 580MB

DenseCore's size advantage is extremely significant: about 2MB, compared to PyTorch's about 580MB. This comparison reveals a key contradiction in deep learning frameworks — the trade-off between feature comprehensiveness and deployment lightness. For resource-constrained environments like serverless and CI, a 2MB framework that can provide core training capabilities would greatly reduce cold start time and memory overhead.

The author proposed three possible development directions:

• Continue replicating PyTorch until most features are usable;
• Become a lightweight drop-in replacement, targeting small CPU scenarios like serverless/CI;
• Transform into a diagnostic tool to help users understand why two PyTorch runs produce different results.

Why is PyTorch so large?

PyTorch's ~580MB size mainly comes from three aspects: first, bundled precompiled kernel libraries (MKL, oneDNN, cuDNN, NCCL, etc.) covering the complete operator set from CPU to multi-GPU; second, code for numerous subsystems like automatic differentiation, distributed communication, quantization, TorchScript, and compilers; third, binary support for multiple platforms and backends. In contrast, DenseCore implements core operators only for a single platform and single-batch scenarios, allowing it to be compressed to about 2MB. This "minimize on demand" approach is attractive in scenarios sensitive to cold start and image size, such as serverless, CI, and edge devices, but it also means it inherently cannot cover PyTorch's full functionality.

Practical Value and Community Response

From an engineering perspective, bit-exact consistency has unique value in debugging and verification. When model behavior is abnormal, eliminating framework differences can locate problems faster. But this also limits DenseCore's applicability — currently only Apple Silicon Macs and only small models like nanoGPT.

On Reddit, the author asked: "Is this useful to anyone? Or is it just a cool learning project?" and invited the community to give "brutal answers." This candor reflects the common tension in personal projects between practicality and learning value.

For the AI infrastructure field, DenseCore's exploration offers at least two insights: PyTorch's CPU implementation contains many non-intuitive optimizations that, while improving performance, sacrifice cross-platform reproducibility; and building a lightweight training framework in Rust is technically feasible, but ecosystem compatibility remains a major obstacle.

Future Direction: Tooling May Be the Best Path

Among the three options, transforming DenseCore into a "PyTorch difference diagnostic tool" may be the most practical. Cross-platform reproducibility issues have long plagued researchers, and a tool that can automatically identify and explain sources of bit-level differences could save significant debugging time. Continuing to replicate PyTorch or making a lightweight alternative faces challenges such as high maintenance costs and low user migration willingness.

Whichever path the author chooses, this project has already demonstrated Rust's potential in deep learning low-level implementations and the astonishing details involved in bit-exact reproduction. Interested readers can visit the GitHub repository (https://github.com/lncch/DenseCore) and run python scripts/demo_bit_exact.py to personally verify the nanoGPT comparison results.

Why might bit-level difference diagnosis become a necessity?

When debugging issues like "the same code produces different results on two runs" or "accuracy drops after switching machines," researchers often need to manually investigate multiple layers of frameworks, libraries, and compilers. If a tool could automatically compare operator-level outputs of two runs and pinpoint which operator first deviates at the bit level, it could compress days of investigation into minutes. The value of such a "difference locator" does not depend on whether DenseCore can replace PyTorch — even as an independent diagnostic layer, it can serve the broader community. From a product positioning perspective, the maintenance cost and ecosystem compatibility threshold for a diagnostic tool are far lower than for a general training framework, making it possibly the most balanced landing point between practicality and learning value.

One-line Summary

Rust framework DenseCore achieves bit-exact training of nanoGPT with PyTorch on Apple Silicon.

Paragraph Summary

A CS student built the deep learning framework DenseCore in Rust. When running nanoGPT on an Apple Silicon Mac, its initial weights, losses, gradients, and weight updates are bit-for-bit identical to PyTorch for 50 consecutive steps, even with dropout enabled. To achieve this, the author delved into PyTorch's CPU implementation, handling many non-intuitive details such as flash attention kernel path switching, Adam tail rounding, and clip_grad_norm_'s reciprocal multiplication. Since PyTorch itself is not bit-exact across platforms (only 1 out of 26 operators is identical on Mac and Linux x86), DenseCore can currently only match Mac. The framework is about 2MB, far smaller than PyTorch's 580MB, suitable for serverless and CI scenarios. The author sought "brutal answers" from the community on Reddit and proposed three possible directions: continuing to replicate PyTorch, making a lightweight alternative, or transforming into a bit-level difference diagnostic tool.