Building a Neural Network from Scratch in C++: Why Does Digit Recognition Always Output the Last Digit?

A developer shared their experience on Reddit: they implemented a neural network from scratch in C++, and it performed well on simple linear regression tasks, able to learn and minimize error. But when they tried to implement digit recognition, the network always output the last digit it had learned, regardless of the input—even when the input was randomly selected. They suspect the problem may lie in their code rather than the neural network itself.

The Symptom: The Network Only Remembers the Last Class

According to the description, the network can converge on regression tasks, suggesting the basic logic of forward and backward propagation may be fine. But after switching to multi-class digit recognition, the model gives the same output for any input—the last digit seen during training. This is usually not random guessing, but rather the model completely failing to learn features and degenerating into a constant predictor.

The developer themselves realized this may not be a problem with neural network theory, but more likely a flaw in implementation details. They admitted in the post: "I don't think this problem is particularly related to neural networks; I probably messed up the code somewhere."

Possible Causes: Troubleshooting from Data to Training

Although the original post didn't provide code, this kind of "constant output" phenomenon is very common when implementing networks from scratch. Here are several directions worth checking first.

1. Label or Data Preprocessing Errors

If the input data is not properly normalized, or the label encoding is incorrect, the network may be unable to distinguish between different classes. For example, all inputs might be incorrectly flattened or shuffled, causing the correspondence between features and labels to be lost. Another common case is inconsistent preprocessing between training and test sets, where patterns learned during training completely fail at test time.

2. Improper Learning Rate or Initialization

A learning rate that is too large may cause weight updates to oscillate violently and eventually saturate to a constant output; a learning rate that is too small may leave the network barely updating. If weight initialization is all zeros or too large, neuron symmetry cannot be broken, and all neurons learn the same thing.

3. Mismatch Between Output Layer and Loss Function

Digit recognition is typically a multi-class problem, where the output layer should use Softmax activation with cross-entropy loss. If mean squared error, commonly used for regression, is mistakenly used, or the output layer lacks the correct activation function, gradients may not propagate effectively, and the network will tend to output the mean or the bias of the last class.

Softmax and cross-entropy loss are the standard combination for multi-class tasks. Softmax transforms the raw scores (logits) from the network's final layer into a probability distribution, where each class's output value is between 0 and 1 and sums to 1; cross-entropy loss measures the difference between the predicted distribution and the true label (usually one-hot encoded). Unlike mean squared error, cross-entropy penalizes incorrect predictions sharply as confidence deviates, and its gradient form is simple ((\hat{y} - y)), effectively avoiding vanishing gradients. If mean squared error is mistakenly used with Softmax when implementing from scratch, the gradient will include an additional Softmax derivative term, causing the update signal to be weak and prone to saturation, and the network will tend to output a constant close to the class prior—this is a common trigger for "always outputting the last class."

4. Backpropagation Implementation Flaws

When implementing from scratch, the chain rule in backpropagation is easy to get wrong. For example, failing to zero gradients during accumulation, incorrect derivative calculations for activation functions, or reversed signs during weight updates can all prevent the network from learning. Regression tasks may converge by luck due to low output dimensionality, but multi-class tasks are more sensitive to gradient correctness.

Gradient checking is the gold standard for locating backpropagation errors when implementing neural networks from scratch. Its principle is to approximate analytical gradients using numerical differentiation: for each parameter (\theta_i), compute (\frac{J(\theta_i+\epsilon)-J(\theta_i-\epsilon)}{2\epsilon}) and compare it with the analytical gradient from backpropagation using relative error (typically requiring less than (10^{-7})). If the error is too large, some step in the chain rule is wrong. In practice, it is recommended to first perform gradient checking on a very small network (e.g., a single hidden layer with few parameters) and a small number of samples, then expand to the full model once confirmed. Additionally, separately check activation function derivatives (e.g., handling of ReLU at 0, saturation regions of Sigmoid) and whether the weight update direction is consistent with gradient descent.

Community Perspective: The Value and Pitfalls of Building from Scratch

In Reddit discussions, such problems often attract experienced developers to share debugging tips. Although the original post didn't provide a follow-up solution, common advice for similar cases includes: overfit on a very small dataset (e.g., a few samples) as a test—if the network cannot even distinguish a few samples, there is an implementation bug; print the gradient norm of each layer to see if it vanishes or explodes; and check layer by layer whether the forward propagation outputs are reasonable.

Writing neural networks from scratch is a great way to deeply understand deep learning, but debugging costs are high. Modern frameworks encapsulate details but also provide tools like gradient checking and visualization. For learners, adding assertions and logging at key steps is more effective than blindly tuning hyperparameters.

Small-scale overfitting testing is a quick way to determine whether it's an "implementation bug" or "insufficient training." The specific approach is: take 5 to 10 training samples (or even fix the same batch and train repeatedly), disable regularization and data augmentation, adjust network capacity to be sufficient to memorize these small samples, and then observe whether the training loss can drop to near zero. If the loss remains high or the output is constant, it is almost certainly a code implementation error (e.g., reversed gradient signs, incorrect loss function, misaligned data and labels); if the loss drops smoothly but test performance is poor, it is a generalization problem. This test can eliminate many low-level errors in minutes and is much more efficient than blindly tuning parameters.

Summary

The problem this developer encountered—the network outputting the same result for all inputs—is a typical symptom of "model not learning." Based on their description, the problem is most likely at the code implementation level, not neural network theory itself. Troubleshooting should start with data preprocessing, loss function matching, and gradient calculation correctness, and use small-scale overfitting experiments to quickly locate the issue.

Original source: Reddit community discussion "Neural Net Learning problem."