PyTorch tensors, layers and parameters

The single neuron from Monday had a blind spot, and the delivery data showed it. Late orders bunched up between 6pm and 9pm, when office workers ordered dinner and the roads around Baner jammed. Before 6pm and after 9pm, the same distance was fine.

A single neuron can only say “the later the hour, the more likely late” or “the earlier, the more likely late”. It cannot say “late in the middle of the evening, fine either side”. That shape is a bump, and one weighted vote draws only straight slopes.

“You need a hidden layer,” Priya said. “A few neurons that each learn a piece of the shape, and one more neuron that combines them. Five inputs, sixteen hidden neurons, then eight, then one output.”

Arjun counted: 241 parameters. “Fine. I’ve got the NumPy code from Monday. I’ll extend it.”

Arjun extends his NumPy neuron

Arjun started small: two inputs, a hidden layer of two neurons with ReLU (the “keep positives, zero the negatives” function, $\max (0,x)$), then one output neuron. He wrote the forward pass, then the backward pass by hand, applying the chain rule from lesson 1 layer by layer.

import numpy as np

x = np.array([1.0, 1.0])          # one order: distance (standardised) and rain
y = 0.0                           # it arrived on time

W1 = np.array([[0.5, -1.0],       # hidden layer: 2 neurons x 2 inputs
               [0.7,  0.4]])
b1 = np.array([0.2, -0.1])
W2 = np.array([[1.5, 0.9]])       # output layer: 1 neuron x 2 hidden values
b2 = np.array([-0.6])

# Forward
h_pre = W1 @ x + b1               # [-0.3, 1.0]
h = np.maximum(0, h_pre)          # ReLU: [0.0, 1.0]
z = (W2 @ h + b2)[0]              # 0.3
p = 1 / (1 + np.exp(-z))          # 0.574

# Backward, by hand
dz = p - y                        # 0.574, from Monday's formula
dh = dz * W2[0]                   # blame passed back to each hidden value
dW1 = np.outer(dh, x)             # blame for each hidden weight
print(dW1.round(3))

[[0.862 0.862]
 [0.517 0.517]]

It looked plausible. Priya asked him to check one number the slow, honest way: nudge W1[0, 0] by a millionth and see how much the loss actually changes. That is the definition of a derivative, from lesson 1.

Arjun's formula for W1[0,0]:  0.862
Nudge and measure:            0.000

Completely wrong. Not slightly off: the true slope is zero and he had 0.862.

He stared at it for ten minutes before he saw it. The first hidden neuron’s raw value was −0.3, so ReLU had zeroed it. A neuron that outputs zero no matter how you wiggle its weights a little has no effect on the loss, so its weights’ slopes must be zero. He had forgotten to pass the blame through the ReLU. The fix was one line, dh = dh * (h_pre > 0).

“Two neurons, and you’ve already made the classic mistake,” said Priya, not unkindly. “Your real network has three layers and 241 parameters. The photo model next month will have a few million, with layers far stranger than ReLU. Do you want to hand-derive all of that?”

The kitchen with a sous-chef

“NumPy is a good set of kitchen knives,” Priya said. “PyTorch is a commercial kitchen. Same knives. But it adds two things.”

“Bigger ovens?”

“That’s one. It can run everything on a GPU, a chip that does thousands of multiplications at the same moment. The other is the important one. Imagine a sous-chef standing behind you with a notebook, writing down every single thing you do. Two cups of flour, into this bowl, whisked with that, baked at this temperature. When the croissant comes out wrong, the sous-chef reads the notebook backwards and tells you exactly how much each step contributed to the mistake.”

“So I write the forward recipe, and it works out the backward recipe.”

“Including the ReLU you forgot. Every time.”

That notebook is called autograd (automatic differentiation), and it is the single reason modern deep learning is practical. Here is how the pieces fit together.

Tensors: the ingredients

Everything in PyTorch is a tensor: a grid of numbers with any number of dimensions. You already know the small cases.

Dimensions	Common name	Crumb example	Shape
0	scalar	today’s total sales	()
1	vector	one order’s 5 features	(5,)
2	matrix	2,000 orders × 5 features	(2000, 5)
4	batch of images	32 photos, 3 colour channels, 64×64 pixels	(32, 3, 64, 64)

Every tensor carries three properties you should check whenever something breaks:

• shape: how many numbers along each dimension. Most bugs live here.
• dtype: the type of number. Deep learning almost always uses float32 (32-bit decimals).
• device: where it lives: cpu, cuda (an NVIDIA GPU) or mps (an Apple Silicon GPU). Tensors must be on the same device to be combined.

Install PyTorch and try it:

uv add torch

(Without uv, pip install torch works too. On Linux with an NVIDIA GPU, the PyTorch install selector gives the exact command for your setup.)

import torch

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
print(device)                         # e.g. "mps" on an M-series Mac

x = torch.tensor([12.0, 2.0, 19.0, 1.0, 4.2])   # one order's raw features
print(x.shape, x.dtype, x.device)     # torch.Size([5]) torch.float32 cpu

photos = torch.rand(32, 3, 64, 64)    # random numbers standing in for images
print(photos.ndim)                    # 4

X = torch.rand(4, 3)
W = torch.rand(3, 2)
print((X @ W).shape)                  # torch.Size([4, 2]): inner 3s matched

Operations look like NumPy: +, *, @ for matrix multiplication, .mean(), .reshape(). Broadcasting (automatically stretching a smaller tensor to fit a bigger one) works the same too, so X - X.mean(dim=0) subtracts each column’s mean from every row. Moving to the GPU is photos.to(device). Same code, often 10 to 50 times faster for big models.

The device line at the top is something every script in this module starts with. For now, everything runs fine on a laptop CPU; the GPU starts to matter in Module 28’s image lessons.

Autograd: the notebook, in action

Mark a tensor with requires_grad=True and PyTorch starts writing in the notebook. Every operation involving it is recorded in a small graph. Call .backward() on a final single number, usually the loss, and PyTorch walks that graph in reverse, applying the chain rule at every step, and leaves each parameter’s slope in its .grad attribute.

Here is Monday’s single neuron and order A, the one Arjun traced by hand:

from torch import nn

x = torch.tensor([1.0, -4/3, 2.0, 1.0])                   # order A, standardised
w = torch.tensor([0.8, 0.3, 0.1, 1.2], requires_grad=True)
b = torch.tensor(-0.5, requires_grad=True)

z = x @ w + b                                              # score: 1.3
loss = nn.functional.binary_cross_entropy_with_logits(z, torch.tensor(0.0))
loss.backward()                                            # read the notebook backwards

print(loss)     # tensor(1.5410, ...)
print(w.grad)   # tensor([ 0.7858, -1.0478,  1.5717,  0.7858])
print(b.grad)   # tensor(0.7858)

Look back at lesson 1’s gradient table: 0.786, −1.048, 1.572, 0.786. Identical. No formula written. binary_cross_entropy_with_logits is the sigmoid-then-cross-entropy calculation from lesson 1 in one function.

Following one order through a real network

Now the network Arjun got wrong. Same order, same weights, but built from PyTorch’s pieces. First, the parts list.

Layers: blocks of knobs

nn.Linear(in_features, out_features) is a linear layer (also called a dense or fully connected layer): it holds a weight matrix and a bias vector and computes $\mathbf{y}=W\mathbf{x}+\mathbf{b}$ for a whole batch at once. The weights are the multipliers, one per input-to-output connection. The bias is one extra number per output. Together they are the layer’s parameters, created with small random values and requires_grad=True already switched on.

Then nn.ReLU() is an activation function (a simple non-linear function applied to every value, giving the network its bends). And nn.Sequential chains layers in order.

net = nn.Sequential(
    nn.Linear(2, 2),     # hidden layer: 2 inputs -> 2 neurons
    nn.ReLU(),
    nn.Linear(2, 1),     # output layer: 2 -> 1 score
)

# Overwrite the random weights with Arjun's numbers, so we can follow along.
with torch.no_grad():                         # don't record this in the notebook
    net[0].weight.copy_(torch.tensor([[0.5, -1.0], [0.7, 0.4]]))
    net[0].bias.copy_(torch.tensor([0.2, -0.1]))
    net[2].weight.copy_(torch.tensor([[1.5, 0.9]]))
    net[2].bias.copy_(torch.tensor([-0.6]))

The forward pass, one layer at a time

Call each layer on the order and print what comes out. Note the extra pair of brackets on the input: layers expect a batch, so one order is a batch of one, shape (1, 2).

x = torch.tensor([[1.0, 1.0]])     # shape (1, 2): one order, two features

h_pre = net[0](x);  print(h_pre)   # tensor([[-0.3000,  1.0000]], grad_fn=<AddmmBackward0>)
h = net[1](h_pre);  print(h)       # tensor([[0., 1.]], grad_fn=<ReluBackward0>)
z = net[2](h);      print(z)       # tensor([[0.3000]], grad_fn=<AddmmBackward0>)

Trace it in words:

1. Hidden layer. Neuron 1: $0.5(1)-1.0(1)+0.2=-0.3$. Neuron 2: $0.7(1)+0.4(1)-0.1=1.0$.
2. ReLU. Neuron 1’s −0.3 becomes 0. It is silent for this order. Neuron 2’s 1.0 passes through.
3. Output. $1.5(0)+0.9(1.0)-0.6=0.3$. Sigmoid of 0.3 is 0.574, a 57% chance of late.

Each printed tensor has a grad_fn. That is the notebook entry: AddmmBackward0 means “a matrix multiply-and-add happened here”, ReluBackward0 means “a ReLU happened here”. When you call backward, PyTorch walks these entries in reverse.

The backward pass

The order was on time, so:

loss = nn.functional.binary_cross_entropy_with_logits(z, torch.tensor([[0.0]]))
loss.backward()

print(loss)                 # tensor(0.8544, ...)
print(net[0].weight.grad)   # tensor([[0.0000, 0.0000],
                            #         [0.5170, 0.5170]])
print(net[2].weight.grad)   # tensor([[0.0000, 0.5744]])

There it is. The first hidden neuron’s weights get a slope of exactly zero, because the ReLU was “off” for this order, and autograd knew to block the blame there. The ReluBackward0 entry in the notebook did what Arjun’s hand-written code forgot. The second neuron gets 0.517, matching the nudge test. The output weight connected to the silent neuron also gets 0: it multiplied a zero, so it could not have mattered.

Why a network needs bends

Arjun wondered aloud why ReLU was there at all. “If it just zeroes things, wouldn’t the network learn better without it?”

Priya wrote on the board: layer 1 multiplies by a matrix $A$. Layer 2 multiplies by $B$. Layer 3 by $C$. “What does the whole stack do?”

$$C(B(A\mathbf{x}))=(CBA)\mathbf{x}$$

“Multiplying three matrices gives one matrix. Three linear layers with nothing between them are exactly equivalent to one linear layer. All that depth buys you nothing: it can still only draw straight lines.”

The activation function breaks that collapse. Because ReLU treats negatives and positives differently, each neuron becomes a “hinge” that switches on above some threshold. Combine a few hinges and you can build a bump: one neuron that switches on at 6pm, another that switches on at 9pm with a negative weight to switch the effect off again. That is how the network learns the rush-hour shape a single neuron could not.

Building the real network

Now the full late-delivery model. The quick way is nn.Sequential:

model = nn.Sequential(
    nn.Linear(5, 16), nn.ReLU(),   # 5 features -> 16 hidden values
    nn.Linear(16, 8), nn.ReLU(),   # 16 -> 8
    nn.Linear(8, 1),               # 8 -> 1 raw score (a "logit")
)

batch = torch.randn(4, 5)          # 4 orders, 5 features each
print(model(batch).shape)          # torch.Size([4, 1]): one score per order
print(sum(p.numel() for p in model.parameters()))   # 241

Where does 241 come from? Each linear layer has (inputs × outputs) weights plus (outputs) biases: $5\times 16+16=96$, then $16\times 8+8=136$, then $8\times 1+1=9$. ReLU has no parameters. $96+136+9=241$.

Running data through the model from input to output is the forward pass. Right now these outputs are meaningless, because the weights are random. Training fixes that tomorrow.

For anything beyond a straight stack (two inputs, skip connections, multiple outputs, all coming in later modules), you write a class that inherits from nn.Module. This is how real PyTorch code looks:

class LateDeliveryNet(nn.Module):
    def __init__(self, n_features: int = 5):
        super().__init__()                    # always call this first
        self.hidden1 = nn.Linear(n_features, 16)
        self.hidden2 = nn.Linear(16, 8)
        self.out = nn.Linear(8, 1)
        self.act = nn.ReLU()

    def forward(self, x):                     # x: (batch, n_features)
        x = self.act(self.hidden1(x))         # (batch, 16)
        x = self.act(self.hidden2(x))         # (batch, 8)
        return self.out(x)                    # (batch, 1)

model = LateDeliveryNet()
for name, p in model.named_parameters():
    print(name, tuple(p.shape))

hidden1.weight (16, 5)
hidden1.bias (16,)
hidden2.weight (8, 16)
hidden2.bias (8,)
out.weight (1, 8)
out.bias (1,)

Two rules. Create layers in __init__, as attributes on self, so PyTorch can find their parameters (that is how model.parameters() knows about all 241). Describe the flow of data in forward. Then always call model(x), not model.forward(x): the call runs some bookkeeping hooks on the way in that PyTorch and other libraries rely on.

The shape comments on each line are a habit worth stealing. When a shape error appears three layers deep, they tell you instantly which line disagrees with your plan.

Where it breaks

Arjun hit every one of these in his first week. You probably will too.

Mixed number types. NumPy creates float64 by default; PyTorch layers use float32. If you load data with torch.from_numpy(arr), it keeps float64, and the first layer fails with mat1 and mat2 must have the same dtype, but got Double and Float. Convert with torch.tensor(arr, dtype=torch.float32) or .float().

Mixed devices. Model on the GPU, data on the CPU: Expected all tensors to be on the same device. Move both with .to(device), and do it for every batch inside the training loop, not only once.

Missing batch dimension. Layers expect (batch, features). A single order of shape (5,) should become (1, 5) with x.unsqueeze(0). Sometimes a missing batch dimension does not crash but silently computes the wrong thing, which is worse.

Linear collapse. Forgetting activations gives you a deep-looking model that can only draw straight lines, as you saw. No error, just disappointing results.

Memory adds up. Every float32 parameter takes 4 bytes. 241 parameters is about 1 KB. The photo classifier in Module 28 might have 2 million: 8 MB, still nothing. A 7-billion-parameter open LLM needs 28 GB just to hold its weights in float32, more than most laptops have. That is exactly why Module 20’s local LLM lessons store weights in 16, 8 or even 4 bits instead. Training needs several times more again, because the gradients and optimiser state are tensors too.

The notebook keeps writing. Autograd records everything involving requires_grad=True tensors, which costs memory. When you only want predictions, wrap the code in with torch.no_grad(): to switch the notebook off. You saw that when setting Arjun’s weights by hand above.

Try it yourself

1. Without running anything, count the parameters of nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 3)). Then check with sum(p.numel() for p in model.parameters()).
2. Feed it a batch of 7 random examples, torch.randn(7, 10). What shape comes out?
3. Feed it torch.randn(7, 12). Read the error and explain it in one sentence.
4. Rebuild Arjun’s two-neuron example, but change the first hidden bias from 0.2 to 0.6. Run forward and backward. Is the first neuron’s weight gradient still zero? Why?

Show the answer

1. First layer: $10\times 32+32=352$. Second: $32\times 3+3=99$. Total 451.
2. torch.Size([7, 3]): 7 examples, 3 scores each. The batch dimension passes straight through.
3. The error mentions mat1 and mat2 shapes cannot be multiplied (7x12 and 10x32): the first layer expects 10 features per example and received 12.
4. No. Neuron 1’s raw value becomes $0.5-1.0+0.6=0.1$, which is positive, so ReLU lets it through and its weights now affect the loss. Their gradients become non-zero. Whether a neuron gets any blame depends on whether it was “on” for that particular input.

Common mistakes

• Hand-writing gradients “to understand them” in real code. Do it once on paper to learn, as Arjun did. Then let autograd do it. Hand-written backward passes are where subtle bugs hide.
• Forgetting super().__init__() in an nn.Module. You get a confusing error the moment you assign a layer. Make it the first line, always.
• Creating layers inside forward. A layer created in forward gets new random weights on every call and is invisible to the optimiser. Create layers in __init__, use them in forward.
• Ignoring dtype and device until something crashes. Print x.shape, x.dtype, x.device the moment anything looks odd.
• Calling model.forward(x). It works today and breaks hooks tomorrow. Use model(x).

Recap

• A tensor is a multi-dimensional array with a shape, dtype and device; it can run on a GPU.
• Autograd records every operation on tensors that require gradients, and .backward() replays them in reverse to fill in .grad exactly, including the cases humans forget, like ReLU blocking blame.
• nn.Linear(in, out) holds a weight of shape (out, in) and a bias of size out; these are its parameters.
• Without activation functions, stacked linear layers collapse into one; ReLU’s bend lets networks learn bumps and curves.
• Real models are nn.Module classes: layers in __init__, data flow in forward, called as model(x).

On Friday, Meera stopped by Arjun’s desk and asked what he had done all week. He tried the sous-chef version.

“I write down how to get from an order’s details to a ‘chance of being late’. Pass it through a few layers of little weighted votes, with a switch after each one so the network can learn bumps, like rush hour. PyTorch watches every step. When the guess is wrong, it reads its notes backwards and tells every one of the 241 numbers how it should change. I don’t have to do the calculus. I did it by hand once and got it wrong.”

“So it’s learnt rush hour?”

“Not yet. It’s got 241 random numbers and it thinks every order is about 50-50.” He turned the screen towards her. “It knows how to compute its mistakes. It doesn’t know how to learn from them yet. That’s Monday: the training loop.”

Go deeper

• PyTorch tutorials: the “Learn the Basics” pages on tensors, autograd and building models match this lesson closely.
• PyTorch install selector for GPU setups.