Pushing the Limits of Extreme Sudoku on a 16GB GPU: Insights from Depth and Flow Matching

Starting from Extreme Sudoku: A Personal Experiment on Depth, Flow Matching, and Test-Time Compute

Training a 16.54M-parameter model on a single 16 GB GPU to solve the full training set of sapientinc/sudoku-extreme—this sounds like a self-indulgent pastime under compute constraints, but an experiment log shared by a researcher on Reddit actually touches on several quite central questions in current reasoning model design: effective depth, iterative refinement, and how inference-time compute budgets are allocated.

The author explicitly states that this is a personal research record, not a controlled benchmark paper. Some implementation and evaluation details still need to be filled in before it can be reproduced. But it is precisely this "informal" sharing that preserves the more genuine twists and surprises of the experimental process.

The Illusion of Early Acceleration: Learning Fast Doesn't Mean a High Ceiling

The author's initial goal was to get the model to complete more reasoning in a single forward pass. He tried intermediate supervision, Sudoku-specific losses, architectural adjustments, and RL-inspired objectives—such as encouraging shallow layers to produce hidden states that help deep layers solve the puzzle. Some changes made the early training curves look very good: within one epoch, accuracy could rise by 10 or even 20 percentage points, rather than the usual few points.

But the problem was that most of these models eventually stalled around 60% validation accuracy. Changing the loss function, adding soft intermediate targets, increasing depth and parameter count, letting some cells stop updating early, repeatedly training on hard examples—the early learning curves all differed, but after about ten epochs, the final results tended to converge to the same level.

The author's summary is quite restrained: improving how fast a model learns does not necessarily improve the accuracy it can ultimately reach. That 60% ceiling may simply reflect the architecture, optimization choices, or available compute at the time, rather than some fundamental limit on single-pass inference.

Effective Depth: The Key to Breaking Through the Plateau

The turning point came from a paper on thousand-layer networks for self-supervised reinforcement learning. The residual layer updates the hidden state in the form h[l+1] = h[l] + Δh[l], which reminded the author of iterative optimization: a sequence of small update steps might solve problems that short computations cannot handle. Of course, this is only an analogy; residual updates are not automatically equivalent to gradient descent.

He scaled the network up to 120 layers, and accuracy finally crossed 60%, reaching about 76%. He then tried removing the many tricks he had previously added, and the model still maintained about 76%. More importantly, a design with a 12-layer network repeated 10 times, also giving an effective depth of 120 layers, was roughly comparable in training progress and final accuracy to a 120-layer network without weight sharing.

This result changed the interpretation of the earlier experiments: those elaborate losses and architectural changes may have helped optimization, but sufficient effective depth was the more critical factor in breaking the plateau. Going deeper might still yield gains, but training costs become unbearable on limited hardware.

Effective depth refers to the number of nonlinear transformation layers the model actually executes during inference, not the number of static parameter layers. For weight-shared recurrent networks, the same set of weights is applied multiple times, equivalent to a very deep network: for example, a 12-layer backbone repeated 10 times has an effective depth of 120 layers. Effective depth determines how much serial computation the model can perform in a single forward pass, directly affecting its ability to solve problems requiring multi-step reasoning. There are several ways to increase effective depth: stacking more independent layers (parameter count grows accordingly), recurrently applying the same module (parameter count is controllable but an iterative mechanism must be designed), or dynamically increasing computation at inference time through an iterative sampling process. The breakthrough past the 60% plateau in this article was achieved precisely by raising effective depth from shallow layers to 120 layers, and 12 layers × 10 repetitions being equivalent to an independent 120-layer network shows that what matters is computational depth, not parameter count.

A Detour into Image Generation, Encountering Flow Matching

The author realized that Sudoku can be treated as a structured image generation problem: input an incomplete grid, output a complete grid. This led him to spend considerable time on generative models—discrete latent representations and quantization schemes, reconstruction losses and adversarial losses, CNNs, Transformers, MLP-Mixers, and column-wise or block-wise autoregressive generation.

Among these attempts, flow matching performed best in small-scale image experiments. Later, reading the GRAM work rekindled his belief that neural networks are capable of handling this kind of difficult Sudoku. A thought experiment made the connection clear: if you want the model to have an effective depth of 10,000 layers, how should you train it? Supervising only the final output is obviously difficult; intermediate supervision might help; weight sharing can keep the parameter count controllable; a small network applied repeatedly can provide depth. And flow matching happens to be a natural way to train this kind of iterative state refinement process.

Flow Matching is a class of methods for training continuous-time generative models. The core idea is to learn a velocity field that transports a simple prior distribution (such as Gaussian noise) along ordinary differential equation (ODE) trajectories to the data distribution. Unlike diffusion models, which need to simulate a complex stochastic forward process, flow matching directly regresses the conditional velocity field, making the training objective simpler and more stable. At inference time, one samples from the prior and iteratively solves the ODE with a numerical integrator (such as Euler's method); the number of integration steps is the number of function evaluations (NFE), which constitutes an adjustable compute budget. In structured generation tasks like Sudoku, flow matching naturally fits iterative refinement: the model can be viewed as performing a series of small update steps on the current grid state, each step adding effective depth, and training only requires supervising the final output without complex intermediate labels.

Inference Compute Becomes an Adjustable Knob

After implementing the flow matching method, the author evaluated every 5,000 training steps. Under a fixed inference budget, accuracy climbed rapidly, once again reaching about 76%. The truly bigger change happened after training, with weights fixed.

He tried several things: increasing the number of integration steps (i.e., the number of function evaluations, NFE); sampling multiple trajectories; and moving an intermediate result back to an earlier flow time point for further refinement—because simply extending the trajectory does not always continue to improve results. Together, these changes pushed the success rate past 99%.

The model size is 16.54M parameters with a 12-layer backbone. The recorded results are as follows:

Max attempts	Steps per attempt	Puzzle-solving accuracy
1	10	73.23%
1	20	75.98%
1	50	78.32%
1	100	79.87%
1	200	81.39%
5	10	79.76%
10	10	82.56%
20	10	85.21%
50	10	88.81%
50	20	92.19%
100	10	91.54%
100	20	94.28%
100	50	96.26%
1,000	20	98.22%
1,000	50	99.29%

A distinction needs to be made: 99.29% used up to 1,000 attempts with 50 steps each; while a single attempt with 50 steps gives 78.32%. Multi-attempt results are conceptually similar to allowing multiple candidate solutions in pass@k, but they should not be read as single-trajectory accuracy, nor as a formal pass@k estimate. The author also trained the model for a few more days hoping to see a late-stage performance jump, but it did not occur in that run.

Number of Function Evaluations (NFE) is a commonly used metric for measuring the inference compute of iterative generative models, referring to the number of times the model's forward pass is invoked during ODE solving or iterative refinement. In flow matching, NFE directly corresponds to the number of integration steps; more steps mean finer discretization of the trajectory and usually higher generation quality, but compute cost also grows linearly. In this article, NFE is treated as an adjustable knob: increasing steps per attempt from 10 to 200 raises accuracy from 73.23% to 81.39%; and by making multiple attempts combined with step adjustments, compute can be flexibly allocated under an overall budget constraint. Note that NFE differs from the number of backbone layer evaluations—the latter equals NFE multiplied by the number of layers per forward pass, and better reflects the actual computational depth. The author uses "nominal maximum layer-evaluation budget" when comparing sampling improvements precisely to unify the accounting.

The Sampling Process Itself Is Hugely Valuable

The initial configuration for high accuracy is expensive. The author then improved inference efficiency while keeping the same weights. Before changing the sampling process, 96.26% corresponded to:

100 attempts × 50 steps × 12 layers = 60,000 layer evaluations

After changing the sampling process, about 96% accuracy corresponded to:

5 attempts × 20 steps × 12 layers = 1,200 layer evaluations

This is roughly a 50× reduction in nominal maximum layer-evaluation budget, not a measured 50× wall-clock speedup. These products are only budget estimates; they do not indicate the average compute per solved puzzle or throughput, which also depend on early stopping, batching, and sampling overhead.

The author then further simplified the architecture. In the experiments, explicit Sudoku-specific row, column, and box mechanisms seemed unnecessary: 2D RoPE + Transformer + flow matching gave essentially the same performance. This does not mean the model has no inductive bias—grid representation, positional encoding, architecture, and training objective still exist; what was removed were the additional Sudoku-specific mechanisms previously built into the network.

Related Work and the Boundaries of Comparison

While preparing the paper, the author found two closely related works: Flow Reasoning Models and Thinking with Looped Flows, both exploring flow-based reasoning methods, including extreme Sudoku. Given the overlap, he decided to present this record as an experimental sharing rather than claim the overall idea as entirely new.

He is particularly interested in reducing the depth required for inference. Some later experiments suggested that an effective depth of about 240 layers or fewer per trajectory might be useful. This number should be distinguished from the total compute of multiple attempts: a 12-layer backbone running 20 steps is itself 240 layer evaluations per trajectory.

The author does not claim to outperform these papers on benchmarks. His experiments used the full training set, whereas FRM used a 1,000-sample setting; training data volume, sampling, and evaluation conditions would all need to be aligned for a meaningful comparison. More training data alone also cannot prove generalization or rule out memorization. He also tried the flow matching method on graph coloring and N-Queens, with encouraging results, but detailed metrics are lacking, so these can only be regarded as preliminary observations.

pass@k is a commonly used metric for evaluating a generative model's problem-solving ability under multi-candidate sampling: the model is allowed to generate k candidate solutions, and success is counted if any one of them is correct. Its estimate is usually obtained by sampling n solutions per problem (n ≥ k) and computing the probability that at least one is correct, with a combinatorial correction. In this article, the author explicitly distinguishes multi-attempt results from pass@k: multi-attempt means independently sampling multiple trajectories under fixed weights and taking the best, which is conceptually similar to pass@k, but because sampling strategy, early stopping, and evaluation conditions differ, it should not be read directly as single-trajectory accuracy or a formal pass@k estimate. This distinction is crucial, because a high pass@k can come from an extremely low single-attempt success rate combined with massive sampling, masking the model's weakness in single-pass inference, whereas in real deployment the sampling budget is often limited.

Rethinking How Reasoning Models Scale

This project changed the author's understanding of how reasoning models scale. There are several ways to add compute: increasing parameter count to boost capacity; generating longer sequences to allow more explicit intermediate computation, such as chain-of-thought; improving effective depth through recurrence or iterative refinement; and sampling more candidate trajectories to explore different solutions.

These approaches are related but not interchangeable, and their value depends on the task, training objective, and inference process. For the Sudoku experiments, the most useful lesson is: learning a good iterative update, and deciding how to apply it at inference time, are both central to performance. Architectural and loss changes helped, but depth and sampling were what pushed the results past the plateau.

The author suspects that combining these scaling forms with reinforcement learning could produce stronger reasoning systems—but this is a hypothesis, and the Sudoku experiments themselves do not establish it. He also does not believe one can infer the internal architecture of proprietary models from these results or from rumors about them. Finally, he poses an open question: when comparing such methods, which compute accounting is most informative—total number of function evaluations, backbone layer evaluations, FLOPs, or measured latency at a fixed solve rate?

One-line Summary

A personal experiment shows that effective depth and inference-time compute allocation are the keys to breaking through Sudoku accuracy plateaus.

Paragraph Summary

The author trained a 16.54M-parameter model on a single 16GB GPU to solve extreme Sudoku, finding that early training acceleration tricks improved learning curves but could not break through a validation accuracy ceiling of about 60%. After raising effective depth to 120 layers, accuracy rose to about 76%, and a 12-layer network repeated 10 times was equivalent to an independent 120-layer network. After introducing flow matching training, by increasing integration steps, multi-trajectory sampling, and intermediate-result fallback refinement, the success rate was pushed past 99% with fixed weights. Improving the sampling process reduced the nominal layer-evaluation budget required for about 96% accuracy by roughly 50×. After architecture simplification, 2D RoPE + Transformer + flow matching achieved similar performance, and explicit Sudoku-specific mechanisms were not necessary. The author emphasizes that these findings are a personal experimental record, and comparisons require aligned conditions.