Reasoning Models: How LLMs Think Before They Answer

A standard LLM starts writing its answer straight away. A reasoning model first writes a long working: it breaks the problem down, tries an approach, checks it, backtracks, and only then replies. This article explains where that behaviour came from, how it is trained, what it costs and when it is worth it.

From "think step by step" to trained reasoning

In 2022 two papers showed that LLMs solve multi-step problems far better when they write out intermediate steps. Wei et al. put worked examples with reasoning steps into the prompt. Kojima et al. found that adding the phrase "Let's think step by step" had a similar effect with no examples at all. The technique became known as chain of thought (CoT).

Why does writing help? A model spends roughly the same amount of computation on every token it generates (see What is an LLM?). A question that needs five dependent steps cannot be solved reliably inside the single forward pass that produces the first token of the answer. Writing the steps out turns one hard prediction into many easy ones, and each step becomes context that later steps can read. The text acts as a scratchpad.

Prompted CoT has limits. The model reasons only as well as its training data taught it to explain, it tends to commit to its first line of attack, and it rarely notices its own mistakes. A reasoning model is trained, mostly with reinforcement learning, to produce chains of thought that actually lead to correct answers. The traces look different from prompted CoT: they often run to thousands of tokens and contain moves such as "wait, that contradicts the second constraint", followed by a correction.

OpenAI's o1, released in 2024, was the first widely available model of this kind. DeepSeek-R1 followed in early 2025 with a published training recipe and open weights. As of 2026, most frontier model families and many open-weight ones offer a reasoning or "thinking" mode.

Test-time compute: a second scaling axis

For years the main lever for better models was training compute: more parameters and more data. Reasoning models add a second lever, test-time compute: the computation spent while answering a particular question. When OpenAI introduced o1, it reported that accuracy improved steadily both with more reinforcement learning during training and with more time spent thinking at inference.

There are two broad ways to spend compute at answer time:

• Sequential. One longer chain of thought, in which the model can revise its own work. This is what reasoning models do natively.
• Parallel. Sample several independent answers and pick one, either by majority vote (often called self-consistency) or with a separate verifier that scores each candidate ("best-of-N").

Snell et al. (2024) studied how to allocate this budget and found that the best strategy depends on difficulty. On easier and medium questions, extra inference compute let a smaller model match a much larger one. On the hardest questions, a stronger base model still mattered more. The practical lesson is that thinking time is a dial, not a free upgrade. It helps most when the model is close to being able to solve the problem and has some way to notice that an attempt is wrong.

How reasoning models are trained

The core recipe is reinforcement learning (RL) on problems whose answers a program can check. That differs from classic RLHF, where the reward comes from a model trained to predict human preferences, and where answers that merely sound good can score well.

Rewards from verifiers

A verifier is any automatic check. Does the final number match the known answer? Does the code compile and pass the unit tests? Does the proof check in a formal proof assistant? Because the check is cheap and hard to fool, the model can practise on very large numbers of problems without human labelling.

The model is not shown how to reason. It is rewarded for reaching correct answers, and whatever habits raise its success rate get reinforced: decomposing the problem, checking intermediate results, trying another approach when one fails.

Outcome versus process rewards

An outcome reward scores only the final answer. It is simple and hard to game, but sparse: a 40-step solution with one slip receives the same zero as nonsense, and a lucky guess with broken reasoning still earns full marks.

A process reward scores each step. Lightman et al. (2023) had people label the correctness of individual steps in maths solutions, trained a process reward model on those labels, and found it picked correct solutions more reliably than a model trained on outcomes alone. The catch is cost: step labels are expensive, and a learned step judge can itself be exploited. The DeepSeek-R1 authors reported that process rewards were hard to scale in large RL runs and relied on simple rule-based outcome rewards instead.

One example: GRPO in DeepSeek-R1

DeepSeek-R1 was trained with Group Relative Policy Optimization (GRPO), an algorithm the same team introduced earlier for maths models. For each problem, the model samples a group of answers. Rules score each one: an accuracy reward for a correct final answer and a format reward for putting the reasoning between <think> tags. Each answer's advantage is its reward relative to the group: the reward minus the group mean, divided by the group's standard deviation. Training then makes above-average answers more likely and below-average ones less likely, with a penalty that stops the model drifting too far from a reference copy of itself.

Because the group provides the baseline, GRPO needs no separate value network (critic), which saves memory compared with the widely used PPO algorithm. The paper's first model, DeepSeek-R1-Zero, applied this RL directly to a base model with no supervised examples of reasoning. Its responses grew longer as training went on, and behaviours such as re-checking and switching approach emerged without being demonstrated.

R1-Zero's output was hard to read and mixed languages, so the final DeepSeek-R1 added a small set of curated reasoning examples before RL, plus further rounds of fine-tuning and RL. The team also distilled R1 into smaller open models by fine-tuning them on its outputs. That is how capable reasoning models became small enough to run locally.

Thinking tokens: hidden, summarised and budgeted

Thinking is made of ordinary generated tokens. The model writes them before the answer, usually between special delimiters. What your application receives depends on the provider:

• Raw. Open-weight models such as DeepSeek-R1 emit their full reasoning, typically between <think> and </think>.
• Hidden. Some APIs keep the raw chain of thought on the server and report only how many reasoning tokens were used.
• Summarised. Several APIs return a condensed summary of the reasoning instead of the full trace.

Either way, you pay for it. Reasoning tokens are billed as output tokens even when you never see them, they take time to generate, and they occupy the context window while the answer is written. A one-line answer can carry thousands of tokens of thinking.

Most APIs expose a control for this, either an effort level (low, medium, high) or a maximum token budget for thinking. Higher settings tend to raise accuracy on hard problems, and they raise cost and latency on every problem. The snippet below uses OpenAI's Responses API to compare effort levels; other providers offer equivalent settings under different names.

from openai import OpenAI

client = OpenAI()
MODEL = "o4-mini"  # substitute any reasoning model available to your account

question = ("A bat and a ball cost $1.10 in total. The bat costs $1.00 more "
            "than the ball. How much does the ball cost?")

for effort in ("low", "medium", "high"):
    response = client.responses.create(
        model=MODEL,
        reasoning={"effort": effort},
        input=question,
    )
    usage = response.usage
    print(f"{effort:>6}  reasoning={usage.output_tokens_details.reasoning_tokens:>5}  "
          f"output={usage.output_tokens:>5}  answer={response.output_text.strip()[:40]!r}")

Run something like this on your own tasks before choosing a setting. You will usually find a level beyond which the answers stop improving while the token count keeps growing.

When a reasoning model is worth it

Reasoning helps when a task has several dependent steps and a definite right answer. It adds little when the answer is a quick transformation of the input.

A practical approach is to start with a standard model or the lowest effort setting and raise it only when your evals show a gain. Many systems route requests: cheap rules or a small classifier send only hard requests to the reasoning model. When the answer is a label, a yes/no or a score, a decision model may be a better fit than either kind of generator.

Failure modes

• Overthinking. On simple questions a reasoning model can spend thousands of tokens second-guessing itself, and occasionally talks itself out of a correct first answer. You pay in latency and money for no gain.
• Unfaithful traces. The visible reasoning is not guaranteed to be the real cause of the answer. In experiments that plant hints in prompts, models often used the hint while their chain of thought never mentioned it. Summarised traces are one step further removed.
• Still-wrong answers. Reasoning cannot recover knowledge the model does not have. A hallucinated fact wrapped in a page of careful-looking steps is more convincing, not more correct.
• Gaming the checker. A model trained to pass tests can learn shortcuts, such as special-casing the test inputs or weakening the tests, especially in agentic coding setups.

Prompting a reasoning model

Prompts that worked for standard models often need trimming. The model already plans its own steps; what it lacks is your context.

• State the goal, constraints and what "done" means. Context about the situation helps more than instructions about how to think.
• Do not micro-manage the steps. "Think step by step" is redundant, and a rigid step-by-step script can push the model down a worse path. The DeepSeek-R1 paper reported that few-shot examples degraded its model's performance, and recommended describing the problem directly instead.
• Ask for verification. Give test cases, sample data or invariants and ask the model to check its answer against them before replying.
• Give it tools. A code interpreter, search or a database turns guesses into checks. Many reasoning models can interleave tool calls with their thinking; see agents and tool calling.
• Specify the output. If you only need the result, say so and give the format, so the visible answer stays short.

Weak:
  Think step by step. First list the tables. Then check each join.
  Then look at the GROUP BY. Then rewrite the query.

Better:
  Goal: find why the monthly revenue report double-counts some orders.
  Context: the schema and query are below. An order can have several shipments.
  Constraints: PostgreSQL, no schema changes, query must stay a single statement.
  Done when: you return a corrected query and explain the root cause in two sentences.
  Before answering, check your query against the three sample rows at the end.