RNNs, LSTMs and GRUs

Croissant dough has to proof: rest in a warm, humid cabinet for about an hour while the yeast works. Crumb’s central kitchen has six proofing cabinets, and each one logs its temperature once a minute. Nobody reads the logs. They are there for the health inspector.

Last Saturday someone propped a cabinet door open for five minutes to wheel in a trolley. The temperature dropped three degrees, the door closed, the cabinet warmed back up, and the rest of the hour looked perfect. Forty minutes later, three trays of croissants came out of the oven flat and dense. About 180 croissants, roughly ₹16,000 of butter, flour and labour, binned before the shops opened.

“The log knew,” Meera said, holding a printout with a little dip near the top. “It was right there. Can a model read these and warn us before we bake?”

Arjun was already nodding. “Sixty readings in, one answer out. That’s the late-delivery model with more inputs.”

Priya raised an eyebrow. “Try it. Then try it on next Saturday.”

Arjun’s first model: sixty inputs, one answer

Arjun generated practice logs: sixty readings around 27°C with a little sensor noise. In half of them, a door-open event knocked the temperature down by 3°C for five minutes, somewhere between minute 5 and minute 15, which is when the kitchen loads trolleys. Those batches were labelled “ruined”.

He fed the 60 readings into a familiar network from Module 27: a flattened input of 60 numbers, a hidden layer of 64 neurons, one output.

class MLP(nn.Module):
    def __init__(self, length: int = 60):
        super().__init__()
        self.net = nn.Sequential(nn.Flatten(), nn.Linear(length, 64), nn.ReLU(), nn.Linear(64, 1))

    def forward(self, x):          # x: (batch, 60, 1)
        return self.net(x)         # (batch, 1)

Validation accuracy: 100%. He was halfway through a message to Meera when Priya asked what would happen if the door were opened at minute 32 instead of minute 8.

He generated 500 logs with the dip between minutes 30 and 40.

validation (dips at minutes 5-15):   100.0%
new logs (dips at minutes 30-40):     53.4%

A coin flip. The network had not learned “a dip anywhere means ruined”. It had learned “low readings at input positions 5 to 19 mean ruined”. Every input position has its own separate weights in a linear layer, so a dip at position 32 arrives through weights that never saw a dip during training. As far as the model was concerned, minute 32 was a completely different feature.

And there was a second problem. “Proofs aren’t always sixty minutes,” Priya said. “Brioche takes ninety. Some bread, forty-five. Your model has exactly sixty input slots.”

Reading a recipe card by card

“How do you read a recipe?” Priya asked.

“Top to bottom, one line at a time.”

“And by step seven, do you still remember step two?”

“The important bits. ‘The butter must be cold.’ I carry that forward. I don’t remember the exact wording.”

“So you have a small notepad in your head. You read a line, update the notepad, read the next line, update it again. And crucially, you use the same reading skill on every line. You don’t have a separate brain for line seven.”

That is a recurrent neural network, or RNN. Instead of looking at all sixty readings at once through sixty separate sets of weights, it reads one reading at a time and keeps a small vector called the hidden state (the notepad: a summary of everything it has read so far). At each step it combines the new reading with the old notepad to write a new notepad:

$$h_t=\tanh (W_x{x}_t+W_h{h}_{t-1}+b)$$

$x_t$ is the reading at minute $t$, $h_{t-1}$ is the notepad from the previous minute, and $h_t$ is the updated notepad. The function tanh squashes values to between −1 and 1 so the notepad cannot grow without limit. The weights $W_x$, $W_h$ and $b$ are the same at every step: the same reading skill applied to every line. After the last reading, a normal linear layer reads the final notepad and makes the prediction.

That shared weight is what fixes Arjun’s bug. A dip at minute 8 and a dip at minute 32 go through exactly the same weights. If the network learns “write ‘cold!’ on the notepad when the temperature drops”, that skill works at any minute. And because it just keeps reading until the sequence ends, a 45-minute proof and a 90-minute proof are equally fine.

flowchart LR
    X1["minute 1"] --> C1["RNN cell"]
    X2["minute 2"] --> C2["RNN cell"]
    X3["minute 3"] --> C3["RNN cell"]
    XN["minute 60"] --> CN["RNN cell"]
    H0["empty notepad"] --> C1
    C1 -->|h1| C2
    C2 -->|h2| C3
    C3 -->|"..."| CN
    CN -->|h60| O["Linear: ruined?"]

The diagram shows the same cell copied out along time. This is called unrolling. It is one cell with one set of weights, used sixty times.

Following one dip through the notepad

To see how the notepad behaves, shrink it to a single number. Give the cell $W_x=0.5$, $W_h=0.8$ and $b=0$. Feed it a “dip” signal of 1 at minute 1, then nothing (0) for the next nine minutes, and watch the notepad:

Minute	Input $x_t$	Calculation	Notepad $h_t$
1	1	$\tanh (0.5\times 1+0.8\times 0)$	0.462
2	0	$\tanh (0.8\times 0.462)$	0.354
3	0	$\tanh (0.8\times 0.354)$	0.276
4	0	$\tanh (0.8\times 0.276)$	0.217
5	0	$\tanh (0.8\times 0.217)$	0.172
10	0	…	0.055

The memory of the dip fades by roughly a quarter every minute. After ten minutes it is down to 0.055, about an eighth of where it started. After fifty minutes it would be indistinguishable from noise. A notepad written in pencil, rubbed a little every time you turn the page.

“So just make $W_h$ bigger,” Arjun said. “Then it fades slower.”

“Bigger than about 1 and it doesn’t fade, it grows, every step, until tanh pins it at 1 and it’s stuck there forever. Then the notepad can’t record anything new. The plain RNN lives on a knife edge between forgetting and exploding.”

The same problem, during training

There is a worse version of this fading, and it happens during learning. To learn “a dip at minute 8 means ruined”, the network has to connect the loss at minute 60 back to the weights used at minute 8. Backpropagation does this by going backwards through every step in between, which is called backpropagation through time (the chain rule applied across the unrolled steps).

At each step back, the gradient gets multiplied by roughly $W_h$ times the slope of tanh at that point. In the tiny example, that is about $0.8\times 0.9\approx 0.72$ per step. Over 52 steps from minute 8 to minute 60:

$$0.72^{52}\approx 0.00000004$$

The learning signal reaching minute 8 is forty billionths of its original size. This is the vanishing gradient problem you met in Module 28, stretched across time instead of across layers. The network simply cannot feel how its minute-8 behaviour affected the result.

The LSTM: a notepad with an eraser, a pen and a reading lamp

In 1997, Sepp Hochreiter and Jürgen Schmidhuber proposed a fix that dominated sequence modelling for twenty years: the LSTM, short for long short-term memory.

Priya’s version: give the baker two things instead of one. A cell state, $c_t$, which is the recipe binder: a long-term record that runs straight along the conveyor belt from step to step. And a hidden state, $h_t$, which is what the baker says out loud at each step. Three gates control the binder. Each gate is a little sigmoid layer producing numbers between 0 (closed) and 1 (open), computed from the current input and the previous hidden state:

• The forget gate $f_t$ is the eraser: how much of the old binder to keep.
• The input gate $i_t$ is the pen: how much new information to write in.
• The output gate $o_t$ is the reading lamp: how much of the binder to say out loud right now.

$$c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t{h}_t=o_t\odot \tanh (c_t)$$

Here ${\tilde{c}}_t$ is the candidate new content (a tanh layer, like the plain RNN’s update), and $\odot$ means “multiply element by element”.

The magic is in the first equation. The binder is updated by multiplying by the forget gate and adding new content. There is no squashing tanh and no weight matrix on the path from $c_{t-1}$ to $c_t$. If the network learns to set the forget gate to 1 and the input gate to 0, the binder is copied forward perfectly, step after step.

Trace it with numbers. At minute 8 the dip arrives. The input gate opens ($i=0.9$), the candidate says “cold!” ($\tilde{c}=-0.99$), and the binder was empty:

$$c_8=f\times 0+0.9\times (-0.99)=-0.89$$

For the next fifty-two normal minutes, the network has learned to keep the pen mostly lifted ($i\approx 0$) and the eraser almost unused ($f=0.99$). After 52 steps:

$$c_{60}\approx 0.99^{52}\times (-0.89)\approx -0.53$$

More than half the memory survives, compared with essentially none for the plain RNN. And on the backward pass, the gradient flowing along the binder is multiplied by $f$ at each step, not by a weight times a tanh slope, so it fades just as slowly. The forget gate is a learned dial for “how long should I remember this?”.

The GRU: fewer gates, nearly as good

In 2014, the GRU (gated recurrent unit) simplified the LSTM. It merges the cell and hidden state into one, and uses two gates: an update gate (how much to replace the old state with new content, combining the LSTM’s eraser and pen into a single dial) and a reset gate (how much of the old state to use when proposing new content). It has about three quarters of the LSTM’s parameters, trains a bit faster, and on most tasks performs about the same. Neither reliably beats the other; try both.

The real code

PyTorch provides all three as drop-in layers with the same interface. Here is a classifier that can use any of them:

import torch
from torch import nn

class SeqClassifier(nn.Module):
    def __init__(self, cell: str = "lstm", hidden: int = 32):
        super().__init__()
        rnn_cls = {"rnn": nn.RNN, "lstm": nn.LSTM, "gru": nn.GRU}[cell]
        self.rnn = rnn_cls(input_size=1, hidden_size=hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):              # x: (batch, minutes, 1)
        out, _ = self.rnn(x)           # out: (batch, minutes, hidden), one notepad per minute
        return self.head(out[:, -1])   # use the LAST minute's notepad -> (batch, 1)

Line by line:

• input_size=1 because each minute has one feature, the temperature. A sales sequence with price and weather as well would have input_size=3.
• hidden_size=32 is the size of the notepad: 32 numbers.
• batch_first=True means inputs are shaped (batch, time, features). Without it, PyTorch expects (time, batch, features), a historical default that catches everyone once.
• self.rnn(x) returns two things. out holds the hidden state at every step. The second item is the final state: h_n for RNN and GRU, or a pair (h_n, c_n) for LSTM, each shaped (layers, batch, hidden). Here the second item is ignored, and the last step of out is used instead (the same values as h_n for a one-layer, one-direction network).

Check the shapes and parameter counts:

x = torch.randn(64, 60, 1)                  # 64 logs, 60 minutes, 1 reading each
for cell in ("rnn", "lstm", "gru"):
    m = SeqClassifier(cell)
    print(cell, m(x).shape, sum(p.numel() for p in m.rnn.parameters()))

rnn torch.Size([64, 1]) 1120
lstm torch.Size([64, 1]) 4480
gru torch.Size([64, 1]) 3360

The LSTM has exactly four times the RNN’s parameters (three gates plus the candidate, each the size of one RNN), and the GRU three times (two gates plus the candidate).

The data generator and training loop are the familiar ones:

def make_logs(n, length=60, dip_range=(5, 15), seed=0):
    """Proofing logs: ~27°C with noise; half have a 3°C, 5-minute dip ('ruined')."""
    g = torch.Generator().manual_seed(seed)
    temps = 27 + 0.3 * torch.randn(n, length, generator=g)
    ruined = torch.rand(n, generator=g) < 0.5
    for i in torch.nonzero(ruined).flatten().tolist():
        start = int(torch.randint(dip_range[0], dip_range[1], (1,), generator=g))
        temps[i, start:start + 5] -= 3.0
    return (temps - 27).unsqueeze(-1), ruined.float().unsqueeze(1)   # centred, (n, length, 1)

X_train, y_train = make_logs(2000, seed=1)
X_val, y_val = make_logs(500, seed=2)

torch.manual_seed(0)
model = SeqClassifier("lstm")
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.BCEWithLogitsLoss()

for epoch in range(1, 61):
    perm = torch.randperm(len(X_train))
    for i in range(0, len(X_train), 64):
        idx = perm[i:i + 64]
        loss = loss_fn(model(X_train[idx]), y_train[idx])
        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # guard against explosions
        optimizer.step()
    with torch.no_grad():
        acc = ((model(X_val) > 0).float() == y_val).float().mean().item()
    if epoch % 5 == 0:
        print(f"epoch {epoch:2d}  val accuracy {acc:.1%}")

What actually happened

Arjun trained all three cells on the 60-minute logs, dips between minutes 5 and 15, and tracked validation accuracy:

Epoch	Plain RNN	LSTM	GRU
10	48%	48%	49%
20	49%	51%	50%
25	50%	100%	50%
30	52%	100%	49%
40	99.6%	100%	100%

Look at the shape of it. For twenty epochs, all three sit at coin-flip level. They are searching for a signal that has to travel back 45 to 55 steps. Then the LSTM finds it, suddenly, around epoch 25. The GRU and the plain RNN get there around epoch 40. That sudden jump is typical of long-range learning: nothing, nothing, nothing, then the gradient finally carries enough signal and the model “gets it”. The plain RNN only made it because of the gradient clipping line; in an earlier run without clipping, it was still stuck at 66% at epoch 40.

Then the real test. The LSTM, trained only on dips at minutes 5 to 15, scored 98% on logs with the door opened at minutes 30 to 40. Arjun’s MLP had scored 53% on the same logs. Same weights at every step means the skill transfers to every minute.

Where recurrent networks break

Arjun tried one more thing: two-hour logs (120 readings) for brioche, with the dip still early. After 80 epochs, all three models, plain RNN, LSTM and GRU, were still at 50%. The signal had to travel back more than 100 steps, and even the LSTM’s binder could not carry the learning signal far enough to get started.

That is the honest summary of recurrent networks:

• They are slow to train on long sequences. Each step depends on the previous one, so the 60 steps run one after another, not in parallel. On a GPU, that wastes most of the hardware.
• Memory is still limited. LSTMs and GRUs stretch useful memory from about ten steps to about a hundred, depending on the task. Beyond that they struggle.
• Everything passes through one notepad. By minute 120 the model has to have squeezed everything that matters into 32 numbers. It cannot go back and re-read minute 8.

That last limitation is what attention removes. Instead of a notepad, a transformer lets every position look directly back at every other position. “Did anything odd happen at any minute?” becomes one direct look, not a message passed along 100 steps. In Module 30 you will build one from scratch, and you will see that the 2017 paper that introduced transformers was aimed squarely at replacing RNNs for exactly these reasons.

So why learn RNNs at all? Three reasons. They are still excellent for small, streaming, numeric problems: sensor logs, sales, anything where data arrives one step at a time and you want a tiny model that runs on a cheap device. They are the clearest way to understand why attention was invented. And newer “state space” models used in some modern LLM research borrow the recurrent idea of a carried state, so it keeps coming back.

Try it yourself

1. Train the plain RNN, LSTM and GRU on 30-minute logs (dips at minutes 5 to 15). How many epochs does each need to pass 95% validation accuracy? Compare with the 60-minute table.
2. Change make_logs so dips happen at a random minute anywhere in the log (dip_range=(0, 55)), retrain the LSTM, and test on late dips (45 to 50) again.
3. Remove clip_grad_norm_ and set the learning rate to 1e-2 for the plain RNN. Watch the loss. What do you see?

Show the answer

1. With only 15 to 25 steps between the dip and the end, all three learn quickly: in Arjun’s run the LSTM and GRU were near 100% by epoch 5, and the plain RNN passed 95% by epoch 10. Shorter gaps, stronger gradients, faster learning.
2. Training on dips everywhere fixes the late-dip weakness: accuracy on late dips should climb into the high 90s. The data, not the architecture, was the limit.
3. Expect occasional sudden spikes in the loss, sometimes nan. These are exploding gradients. With clipping, the same learning rate is usually stable.

Common mistakes

• Forgetting batch_first=True. Without it, PyTorch reads your (batch, time, features) tensor as (time, batch, features), and trains happily on nonsense.
• Using a feedforward network for variable-position patterns. It learns positions, not patterns. Use shared weights across time (an RNN, or the convolutions you met in Module 28).
• Taking out[:, 0] instead of out[:, -1]. The first step’s hidden state has only seen one reading.
• No gradient clipping. Recurrent networks are prone to exploding gradients. Clip to a norm of about 1.
• Training on sequences that do not cover the real cases. A model trained only on early dips is not a dip detector.

Recap

• An RNN reads a sequence step by step, updating a hidden state with the same weights at every step, so it handles any length and recognises patterns at any position.
• Training it uses backpropagation through time, where gradients are multiplied at every step and tend to vanish or explode.
• An LSTM adds a cell state updated by multiply-and-add, with forget, input and output gates, so memory and gradients survive much longer.
• A GRU does the same job with two gates and fewer parameters; it performs about as well.
• Even LSTMs struggle beyond about a hundred steps and are slow, which is why attention and transformers replaced them for language.

Arjun explained it to Meera with the printout in his hand. “My first model memorised where the dip usually was, so a door opened at a different time fooled it. The new one reads the log minute by minute, like a baker reading a recipe, and keeps a note that says ‘it got cold earlier’. It has a little eraser it controls, so the note doesn’t fade before the end. It catches a dip at any time in the first hour.”

“And brioche? Two-hour proofs?”

“There it still loses the thread. That’s a real limit. For brioche I’d add a one-line rule for now: ‘flag any reading below 25 degrees’. Sometimes the simple thing is the right thing.”

Meera nodded and pulled out a different printout: daily croissant sales, Koregaon Park store, the last eighteen months. “Then here’s a sequence with no one-line rule. How many do we bake tomorrow?”

Go deeper

• PyTorch tutorials include sequence models with nn.LSTM and nn.GRU; the API reference for each lists exact input and output shapes.
• Understanding LSTM Networks by Chris Olah is the classic illustrated explanation of the gates.
• Attention Is All You Need, the paper that replaced RNNs for language. Read the introduction now; it will make complete sense after Module 30.