NCM Hands-On Test: 4x Faster Than an LLM, or Slower? Local Benchmarks Reveal the Truth

Title

NCM Hands-On Test: 4x Faster Than an LLM, or Slower? Local Benchmarks Reveal the Truth

Meta Description

Stanford's NCM model claims to be 31x faster than diffusion models and solves 86% of hard Sudoku. Local testing shows it is 4x faster than a same-size LLM on single requests, but falls behind at 64 concurrent requests; the Sudoku edge comes mainly from fill order, not the architecture itself.

Article Content

A new Stanford study has led many to proclaim the "LLM killer" — a model called NCM (Next Concept Modeling) that is said to be 31x faster than diffusion models and can enable a very small model to solve 86% of hard Sudoku puzzles, while traditional language models of the same scale fail almost entirely. But after running a series of controlled experiments on local hardware, a YouTube creator reached a more nuanced conclusion: NCM is indeed a genuine architectural innovation, but its advantages have strict boundaries of applicability.

What exactly is NCM doing?

To understand NCM's value, you first need to know why today's mainstream models are slow. The ChatGPT and Gemini you use today are all autoregressive models — they spit out one token at a time. 1,000 tokens means 1,000 serial steps. The real bottleneck is not computation but memory bandwidth: at every step, the GPU has to read the entire model's weights from VRAM just to emit one more word. According to the video creator's analysis, when a typical local model writes a long answer, it means terabytes of data are shuttled back and forth inside the GPU. This is the root cause of the "writing is slower than reading" experience you feel on your own machine.

Diffusion models try to bypass this problem by guessing a whole page at once and repeatedly erasing and revising. But there is a cost: every iteration has to reprocess all positions, and achieving good quality usually requires dozens of rounds; cutting the number of rounds hurts quality. This is exactly where NCM comes in — can you write in parallel without paying the cost of "redrawing the whole page"?

Its approach borrows the coarse-to-fine idea from image generation. NCM contains two core components: a compressor that encodes an entire passage into multiple layers of "concept" codes (each layer's codes come from a learned codebook of about 4,000 entries); the top layer has only a handful of codes, equivalent to a blurry outline of the whole passage, and the lower the layer, the more codes there are, responsible only for filling in missing details, with the bottom layer having one code per word. The other component is an ordinary Transformer generator, but what it predicts is not "the next word" but "the next layer of concepts," one forward pass per layer. So a 9-layer model means 9 rounds, regardless of output length; the German translation example at the beginning of the video used 11 rounds because that translation model has 11 layers. This is where the name "Next Concept Modeling" comes from — the "concept" here is not a human semantic unit, but merely a learned compression code.

Autoregressive models and the VRAM bandwidth bottleneck

Autoregressive models are the foundation of current mainstream large language models (such as the GPT series and Llama), whose core characteristic is generating output token by token: each time a word is generated, the entire model's weights must be loaded from VRAM into the compute units. For models with a huge number of parameters, a single weight load can involve tens of GB of data, and generating 1,000 tokens means repeating the load 1,000 times, cumulatively transferring terabytes of data. This process is limited by the GPU's memory bandwidth rather than floating-point compute capability, and is therefore called the "VRAM bandwidth bottleneck." NCM's parallel generation approach is precisely intended to bypass this serial bottleneck and reduce the number of repeated weight loads.

Diffusion models and iterative denoising

Diffusion models first shone in the image generation field (such as Stable Diffusion), and their core idea is to generate the target through multiple rounds of iterative denoising: starting from pure noise, each step predicts and removes part of the noise based on the current entire "canvas," gradually approaching a clear result. In text generation, diffusion models likewise use parallel decoding — generating a draft of the entire sequence at once, then repeatedly erasing and revising. However, each iteration requires reprocessing all positions, and usually dozens of rounds are needed to reach acceptable quality, resulting in high computational overhead. NCM borrows the parallel approach of diffusion models but greatly reduces the number of iterations through coarse-to-fine hierarchical concept codes.

Two things the paper didn't test

The video creator's skepticism starts from two gaps. First, all speed numbers in the paper are compared with diffusion models; none are compared with autoregressive models of the same scale — and the latter is the type of model you actually run every day. Second, the 86% result on Sudoku — does it come from the NCM architecture itself, or from some simpler factor?

The creator reproduced the released model on his own DGX, built a same-size ordinary LLM and a self-trained diffusion model, and ran three head-to-head tests.

Test 1: Visualizing the Sudoku process

The paper's numbers hold up — on hard Sudoku, the creator got almost exactly the same 86%. But after printing out the board at each round, he found that NCM does not fill in one cell at a time; instead, it rewrites the entire board every round, and in early rounds it even gets the initial givens wrong, only converging to the correct solution as a whole at the end.

This directly exposes a hard flaw at the user-experience level: NCM's answers cannot be streamed like those of an ordinary model, because until the final round ends, you get no readable content at all.

Test 2: Fast on single requests, slow on batches

The creator found an autoregressive model whose size and shape exactly matched the NCM model, gave it the same prompts, and required the same word count. When sending only one request at a time, NCM was 4x faster on the translation task and nearly 2x faster on news summarization (where the prompt is a full long article). This does show that it is faster than an ordinary autoregressive model.

But when concurrency was raised to 64 requests — the load pattern of a real server — the result completely flipped, and the ordinary autoregressive model came out ahead. The reason again comes back to that VRAM-versus-compute bottleneck: with a single request, the GPU spends most of its time waiting for weight loads, so 9 to 11 forward passes is a huge advantage; with batched requests, the weights are loaded only once and the GPU is doing math the whole time, while NCM processes about 4x as many positions across all layers as the number of words it writes, so the computational load is actually larger. To make matters worse, in the released code, NCM reprocesses the entire prompt through the model at every round, which hurts long-text prompts the most.

This conclusion is crucial for users who want to run multi-agent parallelism: the creator states plainly that running multiple agents simultaneously on NCM will slow things down.

Concurrent batching and GPU utilization

In real server deployments, requests are usually executed concurrently in batches: the GPU loads the model weights once and then computes on multiple inputs simultaneously, thereby amortizing the weight-loading overhead and making the GPU spend most of its time on matrix operations. At this point, the bottleneck becomes compute (FLOPs) rather than memory bandwidth. NCM's cross-layer processing mechanism means that the number of positions processed per output word is about 4x that of an ordinary autoregressive model, so the computational burden is heavier in batched scenarios. In addition, NCM reprocesses the entire prompt at every round, which is especially disadvantageous for long-text prompts. This explains why NCM wins on single requests thanks to its low number of rounds, but is overtaken by autoregressive models under high concurrency.

Test 3: The real source of the Sudoku advantage

The creator trained a diffusion model with the same size, the same training data, and the same test questions. Run in the standard way (filling cells in random order), it solved about a quarter, consistent with what the paper reported. But change one decoding rule — fill the cells the model is most confident about first — and its score approached NCM's. With a few more iterations, the score climbed to 95%.

The creator carefully emphasizes that this is only a simple experiment, and a diffusion model is ultimately not NCM. But the result points to a judgment: the huge Sudoku gap in the paper comes mainly from the cell-filling order, not from the NCM architecture itself. That said, NCM still has something unique — it builds "coarse-to-fine" into the model, whereas diffusion models need additional decoding rules to achieve a similar effect.

Decoding strategy and cell-filling order

In constraint satisfaction tasks such as Sudoku, a model's score depends not only on the architecture but is also deeply affected by the decoding strategy. Standard diffusion models usually fill cells in random order, whereas NCM implicitly prefers "fill the most confident cells first" through the hierarchical structure of its concept codes. Experiments show that merely changing the diffusion model's decoding rule to prioritize filling the cells with the highest model confidence raises its accuracy from about 25% to close to NCM's 86%, and with more iterations it can even reach 95%. This shows that NCM's Sudoku advantage in the paper comes mainly from the optimization of decoding order, not from a fundamental breakthrough in the architecture itself. However, NCM builds this "coarse-to-fine" strategy into model training, achieving a similar effect without additional rules, which still reflects the value of the architectural design.

Unresolved limitations

In addition to the above tests, the creator also listed several observed limitations:

• Fixed output box: Each NCM model writes into a fixed-size box; even if the answer is only one line, it still has to run through all layers, and you pay the full cost.
• Prone to errors on proper nouns and numbers: When converting between text and codes, the compressor sometimes confuses names and numbers.
• Scale unknown: The largest model the creator trained was under 1 billion parameters; whether it holds at 27B or even larger scales has not been verified.
• Code is evaluation-only: The released code is currently for evaluation only, with no training code, and in the creator's experiments, prompt caching behaved problematically.

How to view NCM

Taken together, these local tests show that NCM is not an LLM killer, but a genuinely effective complementary approach in specific scenarios. For a single user running a model on a local GPU, a speedup of up to 4x is a real gain; for scenarios requiring batch serving or parallel agents, it is currently a burden. The attribution of the Sudoku result also needs correction — the architectural innovation deserves recognition, but the exaggerated part comes mainly from the decoding strategy.

The creator's advice to the community is direct: it is worth trying for yourself; it is a very clever idea, but its limitations are equally obvious. As for whether the "sketch first, then fill in details" path can scale to larger model sizes, we will have to wait for more people to verify it on their own hardware.

One-line Summary

NCM is a genuine architectural innovation but not an LLM killer: fast on single requests but slow on batches, with its Sudoku edge coming mainly from decoding strategy.

Paragraph Summary

Stanford's NCM (Next Concept Modeling) generates text in parallel through multi-layer concept codes from coarse to fine, with the number of layers determining the number of iterations rather than output length, making it 2-4x faster than same-size autoregressive models on single requests. However, local controlled experiments reveal strict boundaries: at 64 concurrent requests it is overtaken because of higher computational load and repeated prompt processing each round; the 86% Sudoku score is reproducible, but a diffusion model can approach it merely by changing the cell-filling order, and with more rounds can reach 95%, showing that the advantage comes mainly from decoding strategy rather than the architecture itself. NCM also has limitations such as a fixed output box, errors on proper nouns and numbers, and unverified scale. Conclusion: NCM is an effective acceleration solution for single-user local inference, but is not suitable for batch serving or multi-agent parallelism, and is not an LLM replacement.