RLHF, chain of thought and tree of thought

Meera’s question from last week had stuck with Arjun: what makes a model’s words good? He decided to find out the hard way. He downloaded the base version of Crumb’s open model, the raw output of pretraining, before any of the extra training that turns it into a helpful assistant. He gave it a real customer message:

“My cake arrived squashed. What can you do?”

The base model replied:

“My cake arrived squashed. What can you do? My order was late too. I’ve emailed three times and nobody answers. Posted by priya_k on 14 March…”

It had not answered the customer. It had continued the document, writing the next likely lines of what looked like a complaint forum. Arjun tried the instruct version of the same model, the one Crumb actually used. It answered politely, apologised once, asked for a photo, and offered a replacement or a refund. Same architecture. Same pretrained knowledge. Completely different behaviour.

“What happened between those two files?” he asked Priya.

“Two stages of training that teach the model what people prefer,” she said. “And once you understand those, you’ll understand the other thing Meera asked about: why some models now think before they answer.”

Why a pretrained model is not an assistant

Pretraining, from Module 30, teaches a model to predict the next token over trillions of tokens of internet text. It learns grammar, facts and styles. But its single skill is “continue this text plausibly”. On the internet, a complaint is often followed by more complaints, so that is what it writes.

The usual fix has three stages, laid out in OpenAI’s 2022 InstructGPT paper and used, in some form, by almost every chat model since:

1. Supervised fine-tuning (SFT): train on example conversations written by people, showing the assistant behaviour you want.
2. Reward modelling: collect human comparisons of model answers and train a model to predict which answer people prefer.
3. Reinforcement learning: adjust the assistant to produce answers the reward model scores highly, without drifting too far from where it started.

Stages 2 and 3 together are reinforcement learning from human feedback, or RLHF.

Arjun’s first attempt: show it good replies

Arjun’s instinct was stage 1. He collected 500 of Crumb’s best support replies, written by the team’s most experienced agent, and fine-tuned the base model on them. You will do exactly this kind of fine-tuning in Module 33.

It helped a lot. The model now answered customers instead of continuing forums. But testing it on 100 new messages, he found trouble:

• It promised refunds on orders that were not eligible, because many example replies happened to include refunds.
• It apologised in almost every reply, even to “What time do you open?”
• When a question had no good example in the 500, it produced a confident, plausible, made-up policy.

“I can’t write an example for every situation,” he said. “There are thousands.”

“No,” said Priya. “But look at these two replies to the squashed cake. Which is better?”

A: “I’m so sorry your cake arrived squashed. Could you send a photo? We’ll send a replacement today or refund you in full, whichever you prefer.”

B: “Sorry. Refund processed. You’ll also get a free cake next week.”

“A. B gives away a free cake we don’t offer and doesn’t check anything.”

“How long did that take you?”

“Two seconds.”

“Writing the perfect reply is hard. Recognising the better of two replies is easy. RLHF is built on that asymmetry.”

Priya’s mystery shopper

“Picture training a new counter assistant,” Priya said. “You can’t script every conversation. So you hire a mystery shopper who visits every day and, for each interaction, tells you which of two ways of handling it they liked better. After a few hundred visits, you’ve learned what this shopper likes. Now you could predict their verdict without them.”

“That’s the reward model.”

“Right. A model trained to predict the human’s preference. Then the assistant practises on thousands of new customers, gets a predicted score from your imitation shopper for each reply, and gradually learns to earn high scores.”

“And if it finds a trick the shopper over-rewards? Like apologising a lot?”

“Then it will apologise a lot. That’s the danger, and the reason for a leash.”

Where the analogy breaks: a real mystery shopper has consistent taste. A reward model is a neural network trained on limited comparisons, and it has blind spots that an optimiser will find and exploit.

Following one comparison through RLHF

Step 1: collect a comparison. A human labeller sees the prompt “My cake arrived squashed. What can you do?” and replies A and B, and picks A.

Step 2: train the reward model. The reward model is usually a copy of the language model with its next-token output replaced by a single number: a score $r$ for how good a reply is. Suppose it currently scores $r_A=1.8$ and $r_B=-0.4$. The probability it assigns to “A beats B” comes from the Bradley-Terry model, a classic way of turning two scores into a win probability:

$$P(A\succ B)=\sigma (r_A-r_B)=\sigma (2.2)=0.90$$

Here $\sigma$ is the sigmoid function, $\sigma (x)=1/(1+e^{-x})$, which squashes any number into a probability between 0 and 1. The loss is $-\log 0.90=0.105$: small, because the reward model already agrees with the human.

If instead it had scored $r_A=0.2$ and $r_B=0.5$, then $P(A\succ B)=\sigma (-0.3)=0.43$, the loss would be $-\log 0.43=0.85$, and training would push $r_A$ up and $r_B$ down. Tens of thousands of such comparisons teach the reward model what labellers like.

Step 3: optimise the assistant. Now the assistant (called the policy in reinforcement-learning language) writes replies to many new prompts, the reward model scores them, and the policy is updated to make high-scoring replies more likely. InstructGPT used an algorithm called PPO (proximal policy optimisation) for this step.

Step 4: the leash. The objective is not “maximise reward”. It is “maximise reward, minus a penalty for drifting away from the starting model”:

$$\underset{\theta }{\max }{\mathbb{E}}_{y\sim {\pi }_{\theta }(\cdot \mid x)}[r(x,y)]-\beta \text{KL}({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{\text{ref}}(\cdot \mid x))$$

${\pi }_{\theta }$ is the policy being trained, ${\pi }_{\text{ref}}$ is the frozen SFT model it started from, and KL (the Kullback-Leibler divergence, a measure of how different two probability distributions are) measures how far the policy has drifted. $\beta$ sets the length of the leash. Without it, the policy finds the reward model’s blind spots, a failure called reward hacking: replies that score brilliantly and read terribly, such as three apologies and a refund in every message.

The InstructGPT paper’s headline result shows how much this matters: human labellers preferred the outputs of a 1.3-billion-parameter InstructGPT model over those of the original 175-billion-parameter GPT-3, a model over 100 times larger. Size gave knowledge; preference training gave usefulness.

DPO: the same goal without the reward model

RLHF works but is fiddly: three models in memory (policy, reference, reward model, often plus a value model for PPO), unstable training, and many settings to tune. In 2023, researchers introduced direct preference optimisation (DPO), which reaches a similar goal with an ordinary supervised loss and no separate reward model or RL loop.

The insight is mathematical: for the KL-leashed objective above, the best possible policy has a known form, so the reward can be expressed in terms of the policy itself. That lets you train directly on preference pairs. For a prompt $x$ with a chosen reply $y_w$ (winner) and a rejected reply $y_l$ (loser):

$${\mathcal{L}}_{\text{DPO}}=-\log \sigma \left(\beta \left[\log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}\right]\right)$$

In words: make the chosen reply more likely relative to the reference model, and the rejected reply less likely relative to it. Follow it with numbers. The log-probability of a whole reply is the sum of its tokens’ log-probabilities. Suppose:

	Policy $\log {\pi }_{\theta }$	Reference $\log {\pi }_{\text{ref}}$	Difference
Chosen reply A	-12.0	-13.0	+1.0
Rejected reply B	-15.0	-14.0	-1.0

With $\beta =0.1$, the bracket is $0.1\times (1.0-(-1.0))=0.2$, so the loss is $-\log \sigma (0.2)=-\log 0.55=0.60$. Training pushes A’s relative likelihood further up and B’s further down, until the margin is comfortably positive and the loss shrinks.

import torch
import torch.nn.functional as F

def dpo_loss(pi_chosen, pi_rejected, ref_chosen, ref_rejected, beta=0.1):
    """All inputs are summed log-probabilities of whole replies."""
    margin = beta * ((pi_chosen - ref_chosen) - (pi_rejected - ref_rejected))
    return -F.logsigmoid(margin).mean()

print(dpo_loss(torch.tensor([-12.0]), torch.tensor([-15.0]),
               torch.tensor([-13.0]), torch.tensor([-14.0])))   # tensor(0.5981)

logsigmoid computes $\log \sigma$ in a numerically safe way. In practice, libraries such as Hugging Face TRL provide ready-made DPO trainers. DPO and its variants are now widely used, especially for open models, because they are so much simpler to run. Some labs also replace human labellers with AI feedback guided by written principles; Anthropic’s version of this is called Constitutional AI.

Chain of thought: giving the model room to think

Preference training shapes how a model answers. A different question is how well it reasons. Meera gave Arjun a test:

“A customer orders 3 Diwali hampers at ₹1,250 each. Orders above ₹3,000 get 10% off. Delivery is ₹60. What’s the total?”

Asked to reply with only the final number, a small model answered ₹3,435 sometimes and ₹3,375 or ₹3,810 other times. Asked to “work through it step by step, then give the total”, it wrote:

3 hampers × ₹1,250 = ₹3,750. That is above ₹3,000, so 10% off: ₹3,750 − ₹375 = ₹3,375. Add ₹60 delivery: ₹3,435.

And it got ₹3,435 almost every time.

Why should writing more help? Remember how the model works. Each token is produced by one forward pass through a fixed number of layers: a fixed amount of computation per token. Answering “₹3,435” immediately means doing multiplication, a discount and an addition inside the passes that produce those few tokens. Writing the steps out gives the model more passes, and every intermediate result it writes becomes part of the context that later tokens can attend to. The written steps are its scratchpad.

This is chain-of-thought (CoT) prompting. A 2022 Google paper showed that including a few worked examples with written reasoning in the prompt sharply improved large models on maths and logic problems. A follow-up found that adding “Let’s think step by step” helped too, with no examples at all. You used this in Module 03 as a prompting technique; now you know why it works.

Here is the comparison with the Claude API, using the pattern from Module 01:

import anthropic
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

question = ("A customer orders 3 Diwali hampers at ₹1,250 each. Orders above ₹3,000 "
            "get 10% off. Delivery is ₹60. What is the total?")

def ask(prompt, effort="low"):
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=1000,
        output_config={"effort": effort},
        messages=[{"role": "user", "content": prompt}],
    )
    return next(b.text for b in response.content if b.type == "text")

print(ask(question + " Reply with only the number."))
print(ask(question + " Work through it step by step, then give the total on its own line."))

A frontier model like this will usually get even the terse version right; the effect is far more visible with small open models, which you can try through Ollama from Module 20. The effort setting is a modern descendant of the same idea: higher effort lets the model spend more of its own reasoning before it answers.

Self-consistency: ask several times, vote

A cheap upgrade: sample several chains of thought at a non-zero temperature and take the most common final answer. This is called self-consistency. If 4 of 5 chains reach ₹3,435 and one slips to ₹3,375, the vote corrects the slip. The cost is several times the tokens.

From prompting trick to trained skill

Recent reasoning models are trained, not only prompted, to produce long chains of thought before answering, often with reinforcement learning that rewards correct final answers. They can check their own work, notice mistakes and backtrack, all within the written reasoning. The next lesson’s DeepSeek case study shows how one such model was trained.

Tree of thought: exploring, not only walking

Chain of thought walks one path. Some problems need you to try a path, hit a dead end and back up.

Meera’s next puzzle: “Build a hamper of exactly 4 different items totalling exactly ₹1,700.” The menu: kaju katli box ₹650, chocolate box ₹540, dry fruit pack ₹480, mini plum cake ₹350, diya set ₹220, namkeen tin ₹180.

A model writing one chain tends to go greedy: “Start with the most expensive: kaju katli ₹650 + chocolate box ₹540 = ₹1,190. Add dry fruit ₹480: ₹1,670. Need ₹30 more with one item…” Dead end. A single chain often then fudges the arithmetic rather than starting over.

Tree of thought (ToT), proposed in 2023, turns reasoning into a search:

1. Propose: from the current partial solution, ask the model for several possible next steps.
2. Evaluate: ask the model (or a simple check) how promising each partial solution is: “sure”, “maybe”, “impossible”.
3. Search: keep the best few, expand them, and abandon dead ends, as in a breadth-first or depth-first search.

flowchart TD
    S["Start: 4 items, total 1700"] --> A["Kaju katli 650"]
    S --> B["Chocolate box 540"]
    A --> A1["+ chocolate box: 1190"]
    A --> A2["+ dry fruit: 1130"]
    A1 --> D["+ dry fruit: 1670, need 30: dead end"]
    A2 --> E["+ plum cake: 1480, need 220"]
    E --> F["+ diya set: 1700, solved"]

The tree explores “kaju katli + dry fruit” alongside the greedy path, discovers that the greedy branch is a dead end, and finds kaju katli + dry fruit + plum cake + diya set = ₹1,700. The paper’s best-known result was on the Game of 24 puzzle (combine four numbers with arithmetic to make 24): GPT-4 with chain of thought solved about 4% of puzzles, while with tree of thought it solved 74%.

The cost is the catch. Every proposal and evaluation is a model call; a tree can take dozens or hundreds of calls for one answer. In practice, for a puzzle like this, you would let the model call a small Python tool (Module 06) that searches exhaustively in milliseconds. Tree of thought shines when the steps are fuzzy and need judgement, such as planning or creative writing, where no simple program can check them.

Where it breaks

Reward hacking is real. Preference-trained models can become sycophantic, agreeing with the user or flattering them, because human raters tend to prefer agreeable answers. Many models also drift towards longer answers, because length often wins comparisons.

Preference data encodes someone’s taste. Whose comparisons trained the model shapes its tone, caution and values. A model tuned on one group’s preferences may not suit Crumb’s customers.

More thinking costs more. Chain of thought multiplies output tokens; self-consistency multiplies them again; tree of thought can multiply calls by 100. Latency and cost grow accordingly. Use them where accuracy matters, not for “What time do you open?”.

Reasoning cannot fix missing knowledge. A model can reason flawlessly from a wrong delivery-fee figure. Retrieval (Module 09) and tools (Module 06) supply facts; reasoning only combines them.

Try it yourself

1. A reward model scores reply A at 0.5 and reply B at 1.5, but the human preferred A. What is $P(A\succ B)$ and the loss?
2. Using dpo_loss, what happens to the loss if the policy raises the chosen reply’s log-probability from -12.0 to -11.0 and lowers the rejected one’s from -15.0 to -17.0?

Show the answer

1. $\sigma (0.5-1.5)=\sigma (-1)=0.27$, so the loss is $-\log 0.27\approx 1.31$. A large loss: the reward model strongly disagrees with the human and will be corrected.
2. The bracket becomes $0.1\times ((-11+13)-(-17+14))=0.1\times (2+3)=0.5$, and the loss falls to $-\log \sigma (0.5)\approx 0.47$. The policy has widened the gap between chosen and rejected replies, relative to the reference.

Common mistakes

• Thinking RLHF adds knowledge. It mostly changes behaviour: tone, format, helpfulness, refusals. Knowledge comes from pretraining and from what you put in the prompt.
• Dropping the KL leash. Without it, the policy exploits the reward model and quality collapses while the reward climbs.
• Using chain of thought everywhere. It adds tokens, cost and latency. Save it for multi-step problems.
• Trusting the reasoning text as an explanation. It can be incomplete or after the fact. Verify answers that matter.
• Building a tree search where a tool would do. If a program can check or search the answer exactly, let the model call it.

Recap

• A pretrained base model continues text; supervised fine-tuning teaches it the assistant format, and preference training teaches it which answers people prefer.
• RLHF trains a reward model on human comparisons, then optimises the assistant for that reward with a KL penalty that keeps it close to its starting point.
• DPO reaches a similar goal with a single supervised loss on preference pairs, no reward model or RL loop required.
• Chain of thought improves reasoning by giving the model more computation and a written scratchpad; self-consistency votes over several chains.
• Tree of thought explores and evaluates many partial reasoning paths, backtracking from dead ends, at a much higher cost in model calls.

Arjun explained it to Meera at the end of the day, using the two replies to the squashed cake.

“The raw model learned from the internet, so it writes whatever comes next on a web page: another complaint. Then people showed it good replies, and that taught it the job. Then people compared pairs of replies, this one’s better than that one, thousands of times, and it learned what ‘better’ means to them. And when a question needs working out, like a hamper total, it does better if it writes the working down, like you would on the back of an order slip.”

“And the ones that ‘think’ for a while?”

“They’ve been trained to do that on their own. The most famous recent example came from a lab in China whose model made headlines for how cheaply it was trained.” He pulled up a news article Meera had forwarded him in January. “Priya says it’s the best case study there is for everything in this module.”

Go deeper

• Training language models to follow instructions with human feedback, the InstructGPT paper.
• Chain-of-thought prompting, the original chain-of-thought paper.
• Self-consistency, on sampling several chains and voting.
• Hugging Face docs, see the TRL library for SFT, reward modelling, PPO and DPO trainers.
• Anthropic docs