Gradient descent

Crumb sells cold coffee, and cold coffee has a problem: the cold brew has to be made the night before. Last Tuesday, a 31-degree day, the Koregaon Park store prepped 90 cups and sold 70. Twenty cups went down the sink at ₹45 each. On Saturday, at 37 degrees, they prepped 60 and ran out by 2pm. The shift lead counted 28 customers who left without buying anything.

Meera wanted a rule for the evening shift. “The weather app tells us tomorrow’s temperature. Give me a number of cups.”

Arjun had five days of data and a spreadsheet. He typed a formula, cups = 2 × temperature − 10, and compared it to reality. Too low. He tried 3 × temperature − 20. Closer. He nudged the 3 to 2.9, then the 20 to 22, then back. Twenty minutes later he had a decent line and a headache.

Priya leaned over his shoulder. “You’re doing gradient descent. Slowly, by hand, by feel. Let’s give the computer a compass instead.”

The data, and what “a good line” means

Here are Arjun’s five days at Koregaon Park:

Temperature (°C)	25	28	31	34	37
Cold coffees sold	52	61	70	80	88

The model is a straight line with two parameters (the adjustable numbers a model learns): a slope $w$ and an intercept $b$.

$$\hat{y}=wx+b$$

$x$ is the temperature and $\hat{y}$ (“y-hat”) is the predicted number of cups. Arjun’s guess of $w=3,b=-20$ predicts 55 cups at 25 degrees. Reality was 52, an error of 3.

To judge a line, you need one number for “how wrong is it overall?”. Take each day’s error, square it (so a miss of −3 and a miss of +3 count the same, and big misses count a lot more), then average. That is mean squared error, or MSE, the most common loss function (a formula that boils “how wrong” down to one number) for predicting quantities:

$$L=\frac{1}{n}\sum _{i=1}^n({\hat{y}}_i-y_i{)}^2$$

A perfect line has loss 0. The goal is to find the $w$ and $b$ that make $L$ as small as possible.

Arjun’s brute-force idea

Arjun had a better plan than nudging by hand. “Computers are fast. Let me try every combination.”

import numpy as np

temps = np.array([25, 28, 31, 34, 37], dtype=float)
sales = np.array([52, 61, 70, 80, 88], dtype=float)

best = (None, None, float("inf"))
for w in np.arange(0, 20, 0.01):          # 2,000 slopes
    for b in np.arange(-100, 100, 0.1):   # 2,000 intercepts
        loss = ((w * temps + b - sales) ** 2).mean()
        if loss < best[2]:
            best = (w, b, loss)
print(best)

Four million combinations. It ran for about fifteen seconds on his laptop and found $w\approx 3.03$, $b\approx -23.7$, with a loss of 0.14. He looked pleased.

“Good,” said Priya. “Now do the late-delivery network you’re building this week. It’ll have 241 parameters.”

“Same idea, more loops.”

“Suppose you only try 10 values for each parameter. That’s 10 times 10 times 10, 241 times. $10^{241}$ combinations. There are about $10^{80}$ atoms in the observable universe.”

Arjun did not reply. Brute force works for two knobs and is hopeless for anything you would call a neural network. An LLM has billions of knobs. There has to be a way to improve the knobs without trying them all.

A walk in the fog

“Have you been up Sinhagad at dawn in the monsoon?” Priya asked. “Fog so thick you can’t see your own feet. Say you want to get down to the car park. What do you do?”

“Walk downhill?”

“Which way is downhill? You can’t see.”

Arjun thought. “I’d feel it. Whichever way the ground slopes down under my feet, I’d step that way. Then feel again.”

“That’s the whole algorithm.”

• Where you stand is the current values of the parameters, $w$ and $b$.
• Your height is the loss. Lower is better.
• The slope under your feet is the gradient (the list of slopes of the loss, one per parameter, pointing in the direction where the loss rises fastest).
• Your stride is the learning rate (a small number that sets how far you move each step).

You never see the whole mountain. You only feel the local slope, step, and feel again. With two parameters, the “mountain” is a surface you could draw. With 241, or a billion, nobody can picture it, but the procedure is identical, because computing a slope does not care how many directions there are.

“What if I take giant strides to get down faster?” Arjun asked.

“Try it and see,” Priya said. “On paper first.”

Following one parameter, step by step

To see every number, shrink the problem. One parameter, one data point. The model is $\hat{y}=wx$ (no intercept). The single data point is $x=2$, $y=6$. The right answer is obviously $w=3$, but the algorithm does not know that.

The loss for this one point is:

$$L(w)=(2w-6{)}^2$$

Its derivative (the slope of the loss as you change $w$, from lesson 1) is:

$$\frac{dL}{dw}=2(2w-6)\times 2=8w-24$$

The update rule says: new $w$ = old $w$ − learning rate × slope. Start at $w=0$ with a learning rate of 0.05.

Step	$w$	Slope $8w-24$	Change $-0.05\times \text{slope}$	New $w$	Loss before
1	0.00	−24.00	+1.20	1.20	36.0
2	1.20	−14.40	+0.72	1.92	12.96
3	1.92	−8.64	+0.43	2.35	4.67
4	2.35	−5.18	+0.26	2.61	1.68
5	2.61	−3.11	+0.16	2.77	0.60

The slope is negative, which means “increasing $w$ decreases the loss”, so $w$ goes up. It creeps towards 3. Nobody told it the answer; it felt its way there.

Now take giant strides

Same start, learning rate 0.3:

Step	$w$	Slope	Change	New $w$
1	0.00	−24.0	+7.20	7.20
2	7.20	+33.6	−10.08	−2.88
3	−2.88	−47.0	+14.11	11.23

It leaps right over the answer to 7.2, then overcorrects to −2.88, then flies off to 11.23, further away each time. This is divergence. In real training it looks like a loss that climbs and then turns into nan (not a number, which is what a computer prints when values overflow).

There is a neat way to see exactly where the cliff edge is. For this problem, every step multiplies your distance from the answer by $(1-8\eta )$, where $\eta$ (“eta”) is the learning rate:

Learning rate	Multiplier $1-8\eta$	What happens
0.01	0.92	Shrinks the gap by 8% a step: slow but steady
0.05	0.60	Shrinks by 40% a step: good
0.125	0.00	Lands on 3 in one step: perfect (and lucky)
0.25	−1.00	Bounces between 0 and 6 forever
0.3	−1.40	Each bounce 40% bigger: explodes

The best learning rate depends on how steep the valley is. Steeper valleys need smaller strides. A real network has a different steepness in every direction, which is why choosing the learning rate is part science, part trying a few values.

Opening the hood: the update rule

Back to two parameters and the full dataset. For MSE on a straight line, calculus gives the slope for each parameter:

$$\frac{\partial L}{\partial w}=\frac{2}{n}\sum _i({\hat{y}}_i-y_i)x_i\frac{\partial L}{\partial b}=\frac{2}{n}\sum _i({\hat{y}}_i-y_i)$$

The curly $\partial$ just means “slope with respect to this one parameter, holding the others still”. You do not need to derive these; read them. The slope for $b$ is twice the average error. If the line predicts too high on average, the slope is positive, and stepping against it lowers $b$. The slope for $w$ is the same thing, but each error is weighted by its temperature, so hot days have more say over the slope.

Put both slopes in a list, $\nabla L$ (“nabla L”, the gradient), put both parameters in a list, $\theta$ (“theta”), and the entire algorithm is one line:

$$\theta \leftarrow \theta -\eta \nabla L$$

New parameters equal old parameters minus learning rate times gradient. The minus sign is “go downhill”. Repeat until the loss stops falling.

Gradient descent in code

Here it is on the cold coffee data. One extra step at the top: the temperatures are standardised (rescaled to mean 0 and standard deviation 1, as in lesson 1). You will see why in a moment.

import numpy as np

temps = np.array([25, 28, 31, 34, 37], dtype=float)
sales = np.array([52, 61, 70, 80, 88], dtype=float)

# Standardise the input so one learning rate suits both parameters
mu, sigma = temps.mean(), temps.std()     # 31.0 and about 4.24
x = (temps - mu) / sigma

w, b = 0.0, 0.0      # start anywhere
lr = 0.1             # the learning rate

for step in range(101):
    pred = w * x + b                 # 1. predict
    error = pred - sales
    loss = (error ** 2).mean()       # 2. measure (MSE)
    dw = 2 * (error * x).mean()      # 3. slope for w
    db = 2 * error.mean()            #    slope for b
    w -= lr * dw                     # 4. step downhill
    b -= lr * db
    if step % 20 == 0:
        print(f"step {step:3d}  loss {loss:8.2f}  w {w:6.2f}  b {b:6.2f}")

t = 30
print(f"Forecast at {t}°C: {w * (t - mu) / sigma + b:.1f} cold coffees")

Line by line, the new parts: error is how far off each day’s prediction is. dw and db are the two slope formulas from above, in NumPy. w -= lr * dw is the update rule. The last line converts 30 degrees into the same standardised units before predicting.

step   0  loss  5093.80  w   2.57  b  14.04
step  20  loss     0.82  w  12.75  b  69.55
step  40  loss     0.14  w  12.87  b  70.19
step  60  loss     0.14  w  12.87  b  70.20
step  80  loss     0.14  w  12.87  b  70.20
step 100  loss     0.14  w  12.87  b  70.20
Forecast at 30°C: 67.2 cold coffees

The loss falls from 5,094 to 0.14 in about forty steps and then stops, because it has reached the bottom. That 0.14 is exactly what Arjun’s four-million-combination search found, in 101 cheap steps instead of four million.

(Why is $w$ 12.87 and not 3.03? Because it is the slope per standard deviation of temperature, not per degree. One standard deviation is 4.24 degrees, and $12.87/4.24\approx 3.03$. Same line, different units.)

Why the standardising mattered

Arjun tried removing the standardisation, using raw temperatures with the same learning rate of 0.1. The loss went to infinity within a few dozen steps.

Look at the slope formula for $w$: every error is multiplied by $x$. With raw temperatures around 30, the slope for $w$ is roughly 30 times bigger than the slope for $b$. A step size that is sensible for $b$ is wildly too big for $w$, which overshoots and explodes, exactly like the learning rate 0.3 table. Shrink the learning rate until $w$ behaves, and now $b$ crawls.

In the fog analogy: the valley is a long, narrow ravine, steep across and nearly flat along. You keep bouncing between the walls while barely moving down the length. Standardising reshapes the ravine into a round bowl, where every direction is about equally steep and one stride length fits all. This is the main reason almost every deep learning pipeline standardises its inputs.

Tasting by the tray: mini-batches

The code above looks at all five days before taking each step. That is called batch gradient descent. Fine for five rows. Hopeless for Crumb’s 50,000 product photos, where computing one exact slope would mean processing every photo before moving once.

Priya’s bakery version: you want to know whether today’s croissant batch needs more butter. You could taste all 400 croissants (accurate, and nobody wants to), taste one random croissant (fast, but it might be the odd one), or taste a tray of 32 (a good estimate, quickly).

• Stochastic gradient descent (SGD) uses one example per step. Very cheap, very noisy.
• Mini-batch gradient descent uses a small random group of examples, a mini-batch (usually 32 to 256), per step. This is what everyone actually does. Confusingly, libraries still call it “SGD”.

Each mini-batch gives an estimate of the true slope: right on average, wrong by a bit each time. So the path down the mountain wobbles. That is mostly fine. You get hundreds of slightly wobbly steps in the time one perfect step would take.

Where gradient descent goes wrong

The cold coffee problem is a perfect bowl: one bottom, and every path leads there. Neural networks are not so kind. Here is what goes wrong in practice, in rough order of how often it bites.

The learning rate is wrong. By far the most common problem. Too high and the loss jumps around or becomes nan. Too low and training crawls, and you conclude the model “can’t learn” when it simply has not had time. When anything looks off, try the learning rate 10 times smaller and 3 times larger before changing anything else.

Features are on different scales. As you saw, raw inputs create ravines. Standardise them.

The landscape is bumpy. A neural network’s loss surface is a mountain range of ridges, plateaus and valleys inside valleys. Researchers spent years worrying about local minima (dips that are not the lowest point). It turns out that with millions of parameters, most dips are roughly as good as each other. The bigger practical problems are plateaus (flat regions where the slope is nearly zero and progress stalls) and saddle points (downhill in some directions, uphill in others). The upgrades to gradient descent in two lessons’ time, momentum and Adam, exist largely to push through those.

It finds the best answer for the data you gave it, not for the world. Arjun’s line was trained on days between 25 and 37 degrees. In July, on a 23-degree monsoon day, it predicted 46 cups and the store sold 18, because rainy days have nothing to do with the hot days in the data. Gradient descent did its job perfectly. The data was the problem. You will see this again, and properly, in lesson 5 and in Module 29’s forecasting lesson.

Try it yourself

Run the code above with these learning rates, keeping 100 steps: 0.01, 0.1, 0.5, 1.0, 1.1. Before you run each, write down your prediction: converge, crawl or explode?

Show the answer

• 0.01: crawls. The loss is still about 93 after 100 steps.
• 0.1 and 0.5: both reach 0.14, the best a straight line can do on this data.
• 1.0: stuck. The loss stays at 5,093.8 forever, bouncing between two points, like the 0.25 row in the one-parameter table.
• 1.1: explodes, reaching around $10^{19}$.

Notice the narrow gap between “works well” (0.5) and “broken” (1.0). On real models you find the right learning rate by trying a few values a factor of 3 to 10 apart.

Common mistakes

• Getting the sign wrong. w += lr * dw climbs instead of descending. If the loss rises steadily from the very first step, check the sign.
• Treating the learning rate as a one-time choice. It is the most important setting in training. Revisit it whenever anything changes: new data, new model size, new batch size.
• Skipping input scaling. Unscaled features make one learning rate wrong for every parameter at once.
• Judging progress by a single step. With mini-batches, the loss jumps around. Look at the trend over many steps.
• Thinking zero loss is the goal. Zero loss on the training data usually means the model has memorised it, which is a problem you will meet in lesson 5.

Recap

• A loss function, such as MSE, turns “how wrong is the model?” into one number.
• The gradient is the slope of the loss for every parameter; stepping against it lowers the loss.
• Gradient descent repeats: predict, measure, compute the gradient, update $\theta \leftarrow \theta -\eta \nabla L$.
• Too small a learning rate crawls; too big overshoots and diverges. Standardising inputs makes one learning rate fit all parameters.
• Real training uses mini-batches: noisy but cheap estimates of the slope, hundreds of steps for the cost of one exact step.

That evening Arjun explained the cold coffee rule to Meera. “The computer starts with a random line and checks how wrong it is against the last five days. Then it works out which way to tilt and slide the line to be a bit less wrong, and moves it a little. It does that about forty times and the line stops improving. Thirty degrees tomorrow, prep 67 cups.”

The store sold 66. Meera was impressed for about three seconds. “And the photo sorter? That’s going to be millions of these little numbers, isn’t it?”

“Which is why nobody writes the slope formulas by hand,” Priya said. “Arjun, install PyTorch. Tomorrow the computer does the calculus too.”

Go deeper

• PyTorch tutorials: the “Learn the Basics” series, which you start next lesson.
• NumPy documentation for the array maths used here.