Sampling, temperature and top-p

Two screenshots sit side by side on Meera’s phone. Same test account, same question, asked an hour apart: “My cake arrived squashed. What can you do?”

Reply one: “We’re so sorry! We’ll issue a full refund of ₹1,450 right away.” Reply two: “We’re so sorry! We can offer store credit for your next order.”

“Which of these is our policy?” asks Meera. “Because a customer will screenshot whichever one suits them and send it to me. Then I either pay ₹1,450 I did not plan to, or I look like a liar.”

Crumb’s actual policy, written in the handbook, is neither: a squashed cake gets a photo check by the store team, then a replacement or store credit; refunds need a manager’s approval.

Arjun is baffled. “I didn’t change anything between those two runs. Same code, same question, same model.”

Priya draws a six-sided die on the whiteboard. “You didn’t change anything. The model rolled this.”

Arjun reaches for the obvious knob

Arjun has read about this. “There is a setting called temperature. Zero means no randomness. I will set it to zero and the bot will always give the same answer.”

He adds temperature=0 to his Claude call and runs it. The request fails with a BadRequestError: the API says this model does not support the temperature parameter.

“Ah,” says Priya. “Newer reasoning models like Claude Opus 5.5 do not accept temperature at all. We will come back to why.”

Arjun switches the prototype to Gemini, which still accepts the setting, and sets temperature=0. He runs the squashed-cake question ten times. Ten identical answers. He turns the laptop to Meera, triumphant.

Meera reads it. “We’ll issue a full refund of ₹1,450 right away. Every single time.” She looks up. “So now it is reliably wrong.”

Priya nods slowly. “Hold on to that. It is the most important sentence in this lesson. But first, let us see what that knob actually does.”

The model hands you a menu, not an answer

Recall the loop from the first lesson of this module. For every next token, the model computes a probability for every token (chunk of text) in its vocabulary, about 200,000 of them. Then something has to choose one.

Priya writes the reply so far: “We’re so sorry! We can offer a full” and asks Arjun what comes next.

“Refund. Or replacement.”

“And the model thinks the same. Here is its menu, roughly.”

Next token	Probability
" refund"	56.9%
" replacement"	20.9%
" apology"	12.7%
" credit"	9.4%

“Now, if you had to pick, how would you do it?”

“Take the top one. Refund.”

“That is called greedy decoding: always take the most likely token. It is predictable. What is the downside?”

Arjun thinks. “Everything would sound the same. Every email starts ‘We’re so sorry’.”

“Yes, and on long texts greedy picking tends to get stuck in loops, repeating the same phrases. So most chat models do something else by default. They roll a weighted die. ’ refund’ wins 57 times in 100, ’ replacement’ 21 times, ’ credit’ about 9 times.”

That choosing step is called sampling. It is why the same question gives different wordings. And it is exactly what happened to Meera’s screenshots: somewhere early in reply two, the die landed on a less likely token, and every later token was built on top of that choice. Once the model had written “We can offer store”, the rest followed naturally.

“So one roll in ten,” says Meera, “my bot promises store credit, and the other nine it promises a refund.”

“For this exact wording, roughly. And neither one is your policy.”

Following one decision through the temperature dial

Before the probabilities exist, the model produces a raw score for each token. These raw scores are called logits. Higher means more likely. For our four candidates, suppose the scores are:

Token	Raw score (logit)
" refund"	2.0
" replacement"	1.0
" apology"	0.5
" credit"	0.2

To turn scores into probabilities, the model uses a formula called softmax: raise the number e (about 2.718) to the power of each score, then divide each result by the total so they add up to 100%. You do not need to do this by hand. What matters is what temperature does: it divides every score by the temperature before softmax.

Watch the same four scores at four temperatures. These are exact values, which you will reproduce in code shortly.

Temperature	" refund"	" replacement"	" apology"	" credit"
0.2	99.3%	0.7%	0.1%	0.0%
0.5	82.5%	11.2%	4.1%	2.3%
1.0 (as computed)	56.9%	20.9%	12.7%	9.4%
2.0	40.2%	24.4%	19.0%	16.4%

Read down the columns:

• Low temperature sharpens. Dividing by 0.2 multiplies every score by five, which stretches the gaps between them. The top token takes almost everything. At temperature 0, providers skip the die entirely and take the top token: greedy decoding.
• Temperature 1 leaves the model’s probabilities as they are.
• High temperature flattens. Dividing by 2 halves the gaps. Unlikely tokens get a real chance: " credit" goes from 9% to 16%.

Priya’s kitchen version: “Low temperature is the baker who follows the recipe to the gram. High temperature is the baker improvising at midnight. Fun for a new pastry. Not what you want for an allergen label.”

Drag the slider and watch the bars. At the low end, one bar towers over the rest. At the high end they are nearly level, and the long tail of odd tokens wakes up.

Opening the hood: the other knobs

Temperature reshapes the whole menu. Two other settings cut the menu short instead.

Top-p, also called nucleus sampling, keeps the smallest group of top tokens whose probabilities add up to at least p, throws away the rest, and rescales the survivors to add up to 100% again. With top_p = 0.9 at temperature 1:

1. Sort: " refund" 56.9%, " replacement" 20.9%, " apology" 12.7%, " credit" 9.4%.
2. Add up from the top: 56.9%, then 77.8%, then 90.5%. That passes 90%, so stop.
3. Keep three, drop " credit".
4. Rescale: " refund" 62.9%, " replacement" 23.1%, " apology" 14.0%.

Now " credit" can never be chosen at this step. Top-p is useful because it adapts: when the model is confident, the group is tiny; when it is unsure, the group is larger.

Top-k keeps only the k most likely tokens, say the top 40, whatever their probabilities.

Most providers advise changing temperature or top-p, not both, because they interact in ways that are hard to predict. For nearly every task, temperature alone is enough.

Why “temperature 0” still is not perfectly repeatable

Arjun runs the Gemini bot at temperature 0 one hundred times. Ninety-eight replies are identical. Two differ by a word.

This is normal, and it surprises everyone the first time. Providers run your request on GPUs alongside many other customers’ requests, and the order in which the chips add up long lists of numbers can vary slightly between runs. Tiny rounding differences can tip a near-tie between two tokens, and once one token differs, everything after it can differ. Some providers offer a seed setting to make runs more repeatable, and describe even that as “best effort”.

This property, the same input not always giving the same output, is called non-determinism. Design for it: test important prompts many times, not once, and never build a system that breaks if the wording changes slightly.

Why newer models are taking the knobs away

Arjun’s first error was not a bug. Many recent reasoning models (models that think through a problem in hidden reasoning before answering) remove these settings:

• Claude Opus 5.5 rejects temperature, top_p and top_k. You steer it with an effort setting (low, medium or high) that controls how much it thinks, and with clear instructions.
• OpenAI’s GPT-5 reasoning models support only the default temperature.
• Gemini’s Flash models still accept temperature, top_p and top_k, which makes them handy for experiments.

The reasoning behind this: these models write long chains of internal reasoning, and the labs tune sampling for that process themselves. A user cranking temperature to 1.8 mid-reasoning mostly makes the model worse. The lasting skill is not memorising the perfect temperature. It is writing instructions and providing facts so clearly that the most likely answer is the right one.

The real code: build the dial yourself, then use the real one

Step 1: reproduce the temperature table. No API key needed. This is the exact maths from the table above.

# sampling.py
import math
import random
from collections import Counter

scores = {" refund": 2.0, " replacement": 1.0, " apology": 0.5, " credit": 0.2}

def probabilities(scores, temperature):
    scaled = {tok: s / temperature for tok, s in scores.items()}   # temperature divides the scores
    exps = {tok: math.exp(v) for tok, v in scaled.items()}         # softmax, part 1: e to the power
    total = sum(exps.values())
    return {tok: e / total for tok, e in exps.items()}             # softmax, part 2: share of the total

for t in [0.2, 0.5, 1.0, 2.0]:
    probs = probabilities(scores, t)
    print(f"T={t}: " + ", ".join(f"{tok.strip()} {p:.1%}" for tok, p in probs.items()))

T=0.2: refund 99.3%, replacement 0.7%, apology 0.1%, credit 0.0%
T=0.5: refund 82.5%, replacement 11.2%, apology 4.1%, credit 2.3%
T=1.0: refund 56.9%, replacement 20.9%, apology 12.7%, credit 9.4%
T=2.0: refund 40.2%, replacement 24.4%, apology 19.0%, credit 16.4%

Step 2: roll the die a thousand times. random.choices picks items in proportion to their weights, which is exactly what sampling does.

def sample(probs, n=1000):
    tokens, weights = zip(*probs.items())
    return Counter(random.choices(tokens, weights=weights, k=n))

print(sample(probabilities(scores, 1.0)))

Counter({' refund': 553, ' replacement': 203, ' apology': 145, ' credit': 99})

About 99 rolls in 1,000 chose " credit". That is Meera’s second screenshot, happening about one time in ten.

Step 3: add top-p.

def top_p_filter(probs, p):
    kept, running = {}, 0.0
    for tok, prob in sorted(probs.items(), key=lambda item: -item[1]):  # most likely first
        kept[tok] = prob
        running += prob
        if running >= p:          # stop once we have covered p of the probability
            break
    total = sum(kept.values())
    return {tok: prob / total for tok, prob in kept.items()}             # rescale to 100%

print(top_p_filter(probabilities(scores, 1.0), 0.9))

You should see " refund" about 62.9%, " replacement" 23.1%, " apology" 14.0%, and no " credit".

Step 4: the real dial. Now run a creative prompt at three temperatures on Gemini. You will set up the API key in the next lab; come back to this step then.

# temperature_gemini.py
import os
from dotenv import load_dotenv
from google import genai
from google.genai import types

load_dotenv()
client = genai.Client()  # reads GEMINI_API_KEY
MODEL = os.getenv("GEMINI_MODEL", "gemini-2.5-flash")

prompt = "Suggest one name for Crumb's new Diwali sweets hamper. Reply with the name only."

for temp in [0.0, 1.0, 1.8]:
    names = []
    for _ in range(4):
        response = client.models.generate_content(
            model=MODEL,
            contents=prompt,
            config=types.GenerateContentConfig(temperature=temp),
        )
        names.append(response.text.strip())
    print(f"temperature={temp}: {names}")

Library APIs change quickly. If a name has moved, check the docs linked below.

A typical run:

temperature=0.0: ['Diwali Delights', 'Diwali Delights', 'Diwali Delights', 'Diwali Delights']
temperature=1.0: ['Festival of Flavours', 'Diwali Delights', 'Golden Glow Hamper', 'Mithai Magic']
temperature=1.8: ['Lantern Light Treasury', 'Crumbs of Joy', 'Sparkle & Saffron Box', 'Diya Dreams Collection']

Low temperature gives you the single most typical name, over and over. High temperature gives you variety, and the occasional odd one.

Matching the setting to the job

On models that still accept it, here is a sensible starting point:

Task at Crumb	Starting temperature	Why
Sorting emails into “refund”, “delivery”, “allergy”	0 to 0.2	You want the same label every time
Pulling the order number out of an email	0	There is exactly one right answer
Drafting a support reply	0.3 to 0.7	Natural wording, steady content
Naming the Diwali hamper, writing Instagram captions	0.9 to 1.2	You want variety and ideas

On models that do not accept it, you get roughly the same effect from instructions: “Reply with exactly one of these labels” works far better than any temperature setting.

Where it breaks in production

• Using temperature to fix wrong answers. Low temperature makes answers consistent, not correct. Fix wrong answers by giving the model the facts it needs.
• High temperature on factual work. At 1.5, a policy question can produce a creative, made-up policy.
• Trusting a single test. Because output varies, a prompt that looked perfect once may misbehave one time in twenty. Run important prompts many times.
• Copying a temperature into every call. Some models reject the parameter outright, as Arjun found. Check the model’s documentation.
• Expecting identical output at temperature 0. It is close, not guaranteed. Never parse a reply with code that breaks if one word changes.

Try it yourself

1. In sampling.py, change " credit" to a score of 1.8 and print the table again. What happens at temperature 0.2 now that two tokens are close?
2. Run sample at temperature 2.0. Roughly how many of 1,000 rolls choose " credit"?
3. Run the Gemini experiment with a factual prompt: “What is 17 times 23? Reply with the number only.” at temperatures 0, 1.0 and 1.8. Predict the result before you run it.

Show the answers

1. With " refund" at 2.0 and " credit" at 1.8, the two are a near-tie. At temperature 0.2 the gap is stretched, but " credit" still keeps a real share (about 27%), and " refund" about 73%. Low temperature only makes things predictable when the model has a clear favourite.
2. About 164, matching the 16.4% in the table, give or take a dozen because of randomness.
3. Almost always 391 at every temperature. When the model is very confident, the right token has such a large share that even a flattened die almost always lands on it. Temperature matters most when the model is genuinely unsure, which is exactly when you should be giving it more information.

Common mistakes

• “Temperature 0 means correct.” It means consistent. A consistent wrong answer is still wrong.
• “Higher temperature makes the model smarter or more creative.” It makes rarer tokens more likely. Past about 1.3, that mostly means less coherent.
• Tuning temperature and top-p together. Change one at a time, and usually only temperature.
• Testing once. Sampling means you need several runs to know how a prompt behaves.
• Assuming every model has these knobs. Many reasoning models do not.

Recap

• At each step, the model produces raw scores (logits), turns them into probabilities with softmax, and sampling picks one token.
• Greedy decoding always takes the top token; normal sampling rolls a weighted die.
• Temperature divides the scores: low sharpens towards the top token, high flattens towards variety.
• Top-p and top-k cut the menu down to the most likely tokens before the roll.
• Even temperature 0 is not perfectly repeatable, and many reasoning models no longer accept these settings.
• Consistency comes from sampling settings; correctness comes from giving the model the right information.

Arjun explains it to Meera over the counter. “The model writes by rolling a weighted die for each word. Temperature decides how lopsided the die is. But it was never told our squashed-cake policy, so even with a perfectly lopsided die, it landed on the typical answer, not ours. The fix is to give it the policy.”

“Then give it the policy,” says Meera.

“To do that properly, I need to stop testing in chat windows and talk to these models from code,” says Arjun. Priya is already writing three names on a sticky note.

Go deeper

• Gemini API docs, generation config (temperature, top-p, top-k): https://ai.google.dev/gemini-api/docs
• OpenAI docs, text generation and reasoning models: https://platform.openai.com/docs
• Claude docs, effort and model parameters: https://docs.anthropic.com