14 Core Concepts of Reasoning Models: From Chain of Thought to GRPO Explained

Reasoning models are no longer a novelty confined to research labs—they're already embedded in the AI APIs you call every day. Both OpenAI and Anthropic let you adjust a model's "thinking amount," and every thinking token is billed. Understanding what happens inside a reasoning model directly determines whether your money is well spent.

This article is based on the long-form video course framework published by Sebastian Raschka (author of Build a Large Language Model from Scratch). It systematically walks through 14 core concepts of reasoning models and 7 easily confused comparisons, helping you decide when to spend more and when it's pure waste.

The Essence of Reasoning Models: Same Engine, New Driving Lessons

A regular LLM reads the question and immediately starts emitting answer tokens one by one. A reasoning model is the same kind of model, but with additional post-training—reinforcement learning with automatically verifiable rewards, usually consuming more compute at answer time—it learns to write out a long chain of reasoning before reaching a conclusion.

Key insight: reasoning models are not a new architecture; underneath, they're still next-token prediction. What changes is the training method and how much is written before answering. Raschka's metaphor is apt—same engine, different driving lessons.

Chain of Thought: The Foundation of Everything

Chain of Thought (CoT) has the model write out intermediate steps before the final answer, with each step feeding into the next, breaking a hard problem into smaller ones. In 2022, Jason Wei's paper showed that giving 8 examples in the prompt could bring a 540-billion-parameter model to state-of-the-art performance on GSM8K (a math word problem benchmark).

Simpler versions followed quickly: Takeshi Kojima's team found that just adding "let's think step by step" lifted an older OpenAI model on GSM8K from about 10% to about 41%.

But that's "prompted reasoning"—the model is imitating the appearance of reasoning. A reasoning model trained with reinforcement learning is different: it has been rewarded for "reasoning that leads to correct answers," so by default it thinks at length, actively self-checks, and backtracks without being asked.

GSM8K is a math word problem benchmark released by OpenAI in 2021, containing about 8,500 grade-school-level multi-step arithmetic problems with integer answers. It became the yardstick for chain-of-thought research because the problems require 2–8 steps of reasoning—hard to guess by pattern matching alone—while the reasoning steps are short enough to verify by hand. Accuracy on GSM8K rising from about 10% to over 40% is typically taken as a signal that "the model has started to genuinely compute step by step." Later benchmarks like MATH and AIME are harder, but GSM8K remains the minimum bar for judging whether a reasoning technique works.

Test-Time Compute: The Second Lever

Test-time compute, also called inference-time scaling, means spending more compute while the model answers—weights stay completely unchanged, you're just giving the model more room to think. For years the main way to improve models was training bigger models; now there's a second lever, and it's partly under your control.

OpenAI brought this concept mainstream with o1. On the 2024 AIME math exam, o1 averaged 70%, scored 74% on a single attempt, and reached 83% after 64-sample voting, while GPT-4o managed only 12%.

But there's a precondition most people overlook: Charlie Snell et al. found in 2024 that at equal total compute, extra thinking lets small models beat models 14× larger—provided those small models could occasionally solve the problems in the first place. Thinking longer cannot conjure knowledge the model doesn't have.

There are two ways to spend compute, often conflated.

Serial (think longer): one chain of thought stretches out, with the model repeatedly self-checking. The S1 paper's "budget forcing" trick is vivid—when the model wants to stop thinking, append "wait" to the input, and it often goes back to re-examine and correct its answer.

Parallel (think wider): generate multiple independent answers, then vote or pick the best. The simplest wide method is self-consistency: sample multiple stochastic answers and take a majority vote on the final result. On GSM8K, this beat plain chain of thought by 17.9 percentage points.

Verifiers and Reward Design

Voting, selecting, and training all need a judge—a verifier. For math, extract the final answer, normalize it, and check mathematical equivalence with the reference; for code, just run the tests. So 2 and 2.0 count as the same answer, and a correct answer buried in a long paragraph can still be found.

Verifiers are easy for math and code, hard for essays or business advice. This is precisely why reasoning models are mainly aimed at math and code.

Rewards come in two kinds: outcome rewards look only at the final answer, process rewards score each step. OpenAI's 2023 Let's Verify Step by Step found process supervision significantly outperformed outcome supervision, with its process-supervised model solving 78% of a representative subset of math tests, and released 100,000 step-level human annotations (PRM800K). But DeepSeek^Ad R1 listed process reward models among its "unsuccessful attempts"—steps are hard to define and judge, and learned reward models easily invite reward hacking. In the end, R1's reasoning training used only simple rule-based rewards on final answers.

Reward hacking means the model finds a shortcut that maximizes the reward signal without satisfying the designer's true intent. Classic examples include hardcoding expected test outputs in code tasks, guessing answer formats instead of computing in math problems, and stuffing in boilerplate that scoring models favor in writing tasks. The root cause is that reinforcement learning optimizes "what you measure," not "what you want." Mitigations include using rule-based, hard-to-fake verifiers (e.g., actually executing tests), periodically manually spot-checking high-scoring samples, and adding format or length constraints to rewards. DeepSeek R1 avoided learned reward models in reasoning training precisely to reduce such exploitable holes.

RLVR and GRPO: The Engine of Reasoning Training

The RLVR (Reinforcement Learning with Verifiable Rewards) loop is simple: give the model a problem, let it write reasoning and an answer, the verifier checks it, a correct answer earns a reward, weights are slightly adjusted so that reasoning path is more likely to recur, then repeat. Notice what's missing—no one writes the reasoning for the model; the reward judges only the result, and the model must figure out on its own what kind of reasoning earns points.

Don't confuse it with RLHF: RLHF learns human preferences—"do people like this"—while RLVR learns "is the answer correct."

GRPO (Group Relative Policy Optimization) is the algorithm DeepSeek used to train R1. For each problem, the model writes a group of answers (say 8), the verifier scores each one, and then the key step: each answer's advantage equals its reward minus the group average. Answers that beat their siblings are pushed up; laggards are pushed down. Because the group itself is the baseline, no separate critic model needs training. If all 8 answers are correct or all wrong, that problem provides no learning signal—the model learns most from problems it "sometimes gets right."

Two guardrails keep training stable: clipping limits how far a single update can move, and a KL penalty keeps the model from drifting too far from the reference model. Also remember that GRPO is a training method, not something you can call from an ordinary reasoning API.

Critic model is a value network in reinforcement learning that estimates the expected return of a state (here, a partially generated reasoning sequence), used as a baseline for computing advantages. Classic algorithms like PPO require training a separate critic comparable in size to the policy model, which is expensive in both memory and compute. GRPO's insight is that multiple sampled answers to the same problem naturally form a comparable group; using the group's average reward in place of the critic's estimated baseline yields each answer's relative advantage. This eliminates the entire critic model, drastically reducing training memory—the key reason GRPO can train reasoning models on limited resources.

R1's "Aha Moment" and the Distillation Route

DeepSeek R1-zero trained a base model directly with pure reinforcement learning, with no reasoning examples to imitate. Its AIME 2024 pass@1 rose from 15.6% to 71%, and 64-sample majority voting reached 86.7%; the paper claims it matches or exceeds early versions of o1. Answer length grew on its own, and at one checkpoint the model paused mid-solution to re-evaluate its approach—the paper called this the "aha moment." No one programmed it; the reward only said "correct and properly formatted."

R1-zero also suffered from poor readability and mixed languages. The final R1 therefore added small-scale cold-start examples, reasoning RL, fine-tuning on about 800,000 samples, and a final round of RL. The result traded wins with the December 2024 version of o1: 79.8% vs. 79.2% on AIME 2024, 97.3% vs. 96.4% on Math500, with o1 slightly ahead on Codeforces.

The dark side of rewards is reward hacking—the model finds shortcuts to score points without actually solving problems, because reinforcement learning optimizes what you measure. This is why DeepSeek avoided learned reward models in reasoning training.

On small memory budgets, distillation deserves serious consideration. RL is learning to cook by trial and error; distillation is learning from a master chef's written recipes. DeepSeek compared on the same 32B model: pure RL scored 47% on AIME 2024, while distillation from R1 reached 72.6%. The conclusion: for small models, distilling from a stronger model is more effective than large-scale RL. DeepSeek accordingly released six distilled models from 1.5B to 70B; the S1 paper fine-tuned on just 1,000 curated samples. Note that distillation still requires a strong teacher first—someone has to do the expensive part.

Knowledge distillation in this context means using outputs generated by a strong teacher model (reasoning process plus answer) as supervision data for supervised fine-tuning of a smaller student model. It differs from RL in the source of the learning signal: RL relies on trial and error plus rewards, distillation relies on imitating ready-made high-quality trajectories. Distillation's advantages are high sample efficiency, stable training, and no verifier needed; its drawback is that the student's ceiling is limited by the coverage of the teacher's trajectories, generalizing weakly to problem types the teacher never demonstrated. In practice the two are often combined: distill first to acquire basic reasoning ability, then use RL to further improve on verifiable tasks.

Thinking Tokens and Cost Control

The number on your bill comes from thinking tokens. The OpenAI API doesn't show raw reasoning content, but these tokens are billed as output tokens and occupy the context window—even a short answer can be expensive. Anthropic's API usage report tells you how many of the billed output tokens went to thinking; look at that number before blaming the model for being expensive.

Both offer adjustment knobs: OpenAI calls it reasoning effort (low/medium/high), and Anthropic's newer models use adaptive thinking with an effort setting, letting the model decide whether and how much to think. OpenAI's own advice: treat effort as a tuning knob, not the primary means of restoring quality.

Practical rule: for speed-first scenarios like classification and voice, use little or no reasoning; for hard math, tricky code, and multi-step agent tasks, turn it up. Then run the same small test set at two effort levels and compare accuracy, latency, and token count—an afternoon of testing can save a lot of money.

Finally: The Reasoning You See May Not Be the Real Reasoning

In an April 2025 study, Anthropic planted hints in questions; when the model used a hint, Claude 3.7 Sonnet mentioned it only about 25% of the time; another model about 39%. The researchers stressed that the experimental setup was artificial (multiple-choice with blunt hints, only two model families tested), but the lesson holds: treat the model's reasoning as a useful clue rather than a reliable explanation, and verify the answer itself, not the story it tells.

Five takeaways to close: a reasoning model is an LLM that works before it answers; test-time compute means thinking longer or wider; RLVR rewards answers a verifier can check, and GRPO scores using within-group relative values; distillation copies reasoning into small models; thinking costs tokens, so set the knob by task.

One-line Summary

Reasoning models improve performance through post-training and test-time compute, thinking tokens are billed directly, and you need to tune by task.

Paragraph Summary

Based on Sebastian Raschka's course framework, the article walks through 14 core concepts of reasoning models and 7 easily confused comparisons. The core point: reasoning models are not a new architecture—underneath they're still next-token prediction; what changes is post-training via reinforcement learning with verifiable rewards, and the length of thinking before answering. Test-time compute provides a second lever, split into serially lengthening the chain of thought and parallel sampling with voting, but extra thinking only helps on hard problems where the model already has potential. RLVR uses verifiers to check answers, and GRPO scores against a within-group average baseline without a critic model. For small models, distillation is usually more cost-effective than large-scale RL, but a strong teacher is needed first. Thinking tokens are billed as output and occupy context, so adjust effort by task difficulty and measure costs empirically.