H²R: Boosting Mathematical Reasoning with Hierarchical Hint Rewards

Reinforcement learning is becoming a key pathway for improving the mathematical reasoning capabilities of large language models, but existing methods remain weak in supervising intermediate steps. A new paper proposes the Hierarchical Hint-based Reward (H²R) framework, which attempts to use the sub-problem structure inherent in mathematical problems as structured intermediate signals to provide denser guidance for complex reasoning.

The Problem: Sparse Rewards Cannot Support Multi-Step Reasoning

Reinforcement Learning with Verifiable Rewards (RLVR) has shown great potential on mathematical reasoning tasks, but it mainly relies on the correctness of the final answer to issue rewards. For complex problems requiring multi-step derivation, this sparse reward makes it difficult for the model to judge whether intermediate steps are reasonable, and the exploration efficiency of reasoning paths is therefore limited.

The paper points out that existing methods have limited intermediate supervision for complex, multi-step reasoning tasks. In other words, the model only knows "whether the final answer was correct," but not "which step was right and which step went astray." This is exactly the gap that H²R aims to fill.

RLVR (Reinforcement Learning with Verifiable Rewards) is a training paradigm developed in recent years for tasks with standard answers, such as mathematics and code. Its core feature is that the reward signal comes from an automatically determinable verifier—for example, using a symbolic solver to compare the final answer for math problems, or using unit tests to determine whether code passes—rather than relying on human preference annotation or training a reward model. This characteristic gives RLVR the advantages of scalability and low cost, and it has been widely adopted after works such as DeepSeek^Ad-R1. But its cost is also obvious: the verifier usually gives only a binary judgment on the entire response, the reward granularity is too coarse, there is a lack of feedback on local errors in the multi-step reasoning process, and exploration efficiency is limited. H²R is an improvement that adds intermediate signals on top of this paradigm.

The Core Idea of H²R: Turning Sub-Problems into Milestones

The key design of H²R is to use the sub-problem structure intrinsic to mathematical problems to introduce structured intermediate conditioning signals during RLVR training. Unlike chain-of-thought prompting, these hierarchical hints are positioned as "logically grounded intermediate milestones," whose role is to effectively shorten the model's reasoning span toward the final answer.

It can be understood this way: a multi-step math problem can often be decomposed into several progressive sub-problems, and the answer to each sub-problem naturally constitutes a verifiable checkpoint. H²R incorporates these checkpoints into the reward design, so that during training the model aligns not only with the final answer but also with the key nodes of intermediate reasoning. The paper emphasizes that incorporating reliable intermediate checkpoints into reward design is the key to maintaining complex mathematical reasoning capabilities.

The "Oracle" in Oracle Hierarchical Hints originates from the concept of an oracle in reinforcement learning and sequential decision-making, referring to a reference agent with complete information or ideal judgment capability. In this context, Oracle hierarchical hints mean sub-problem checkpoints provided by a stronger model, a standard solution, or human verification, with credibility close to the upper limit, rather than generated by the policy model itself. It is not used in the deployment stage, but serves as an upper-bound reference: by comparing performance under ideal hints with performance under automatically generated hints, one can quantify the contribution of the variable "hint quality" to reward effectiveness and final accuracy, thereby determining whether the performance bottleneck comes from reward design or from hint generation. This kind of upper-bound analysis is a commonly used diagnostic tool when evaluating intermediate supervision methods.

The HiMath Benchmark: Providing Evaluation Ground for Hierarchical Structure

To validate the framework, the authors also constructed HiMath, a benchmark composed of hierarchically structured mathematical problems. Its value lies in turning "sub-problem structure" from an abstract concept into an evaluable object, so that the effects of hierarchical hints can be systematically observed rather than remaining only empirical judgments at the level of training techniques.

Experimental Findings: Hierarchical Hints Alone Already Provide Gains

The paper reports a noteworthy conclusion: across a variety of large language models, even without task-specific training, hierarchical hints consistently improve reasoning performance. This shows that the guidance information carried by sub-problem structure has cross-model generality rather than depending on fine-tuning adaptation for a specific model.

When hierarchical hints are integrated into RL training to form H²R, the gains are further amplified. Taking Qwen2.5-Math-7B as an example, after using hierarchical hints its accuracy on HiMath improved markedly; and when using Oracle Hierarchical Hints (that is, providing more reliable hierarchical hints), accuracy rose another notch. The paper regards this significant improvement as direct support for its core argument: reliable intermediate checkpoints are indispensable for sustaining complex mathematical reasoning.

It should be noted that the Oracle setting represents an idealized upper-bound reference; it indicates the direction in which the quality of hierarchical hints affects final benefits—the more reliable the hints, the more effective the reward signal.

Chain-of-thought prompting requires the model to explicitly write out step-by-step derivations before giving an answer. Its role is to guide the model to spread computation across multiple tokens and reduce the burden of one-shot reasoning, but the prompt itself does not carry supervision signals about "which step is correct." The hierarchical hints discussed by H²R depend on the sub-problem decomposition inherent in the problem: the answer to each sub-problem is an independently verifiable milestone, and thus a long reasoning chain can be cut into several scorable segments. The difference between the two can be summarized as follows: chain-of-thought provides the format for unfolding reasoning, while hierarchical hints provide the locations for checking reasoning; the former is a technique at the output level, while the latter is a structure at the reward level. This also explains why hierarchical hints can independently bring cross-model gains without task-specific training.

Implications: Reward Design Is Shifting from "Looking at Results" to "Looking at Process"

The approach of H²R reflects a broader trend: reinforcement learning for mathematical reasoning is shifting from outcome-oriented sparse rewards to process-aware dense rewards. Sub-problem structure is a low-cost, verifiable, naturally occurring intermediate signal. Using it in reward design amounts to providing the model with more alignable anchors without introducing additional human annotation.

Of course, this path still leaves questions to be answered: how to automatically generate hierarchical hints, how to ensure their quality, and where the generalization boundaries lie across different problem types are all directions that future research needs to address. But at least from the experimental results on HiMath, writing intermediate checkpoints into the reward function is a route worth continued investment.

One-line Summary

H²R uses the sub-problem structure of math problems as intermediate rewards, making up for RLVR's insufficient supervision of multi-step reasoning due to sparse rewards.

Paragraph Summary

To address the problem that RLVR issues rewards only based on the final answer and struggles to supervise intermediate steps in multi-step reasoning, the paper proposes the Hierarchical Hint-based Reward (H²R) framework. It uses the sub-problem structure inherent in math problems as structured intermediate conditioning signals, turning the answer to each sub-problem into a verifiable checkpoint and thereby shortening the model's reasoning span toward the final answer. The authors also construct the hierarchical math benchmark HiMath, enabling systematic evaluation of the guidance effects of sub-problem structure. Experiments show that even without task-specific training, hierarchical hints consistently improve reasoning performance across multiple large models; after integration into RL training, Qwen2.5-Math-7B's accuracy on HiMath improves significantly, with even greater gains when using Oracle hierarchical hints. The paper therefore argues that incorporating reliable intermediate checkpoints into reward design is key to maintaining complex mathematical reasoning capabilities.