Backprop is the chain rule with bookkeeping

Last time we built one unit and watched it fail. Given a dataset shaped like a flower, a single neuron converged cleanly to 46% accuracy, which on two balanced classes is worse than a coin. Nothing was broken. A unit draws one straight line, and there is no straight line through a flower.

The fix is a hidden layer. What's surprising is how little the training procedure changes: the same chain rule from section 04 of that post, walked back through a few more nodes. This post is those extra nodes.

01 · Same unit, four times

Put four units where there was one. Each has its own weights and its own bias, each computes the same $z = w^{T}x + b$ from the same input, and each is squashed. Then a fifth unit reads all four outputs and produces the answer.

Two pieces of notation carry the rest of the series. Square brackets are layers, parentheses are examples: $a^{[1](3)}$ is the activation of layer 1 on training example 3. The input counts as layer 0 ($X = A^{[0]}$), and it isn't counted in the network's depth — the thing we just built is a two-layer network.

Writing out four units separately is tedious, and the tedium is a hint:

$$\begin{aligned} z^{[1]}_1 &= \left(w^{[1]}_1\right)^{T}x + b^{[1]}_1, \quad a^{[1]}_1 = g\!\left(z^{[1]}_1\right) \\ z^{[1]}_2 &= \left(w^{[1]}_2\right)^{T}x + b^{[1]}_2, \quad a^{[1]}_2 = g\!\left(z^{[1]}_2\right) \\ &\;\;\vdots \end{aligned}$$

Stack those weight vectors as the rows of a matrix and all four lines collapse into one. Do the same with the examples — one column per example — and the whole training set goes through in a single call:

$$Z^{[1]} = W^{[1]}X + b^{[1]}, \qquad A^{[1]} = g\!\left(Z^{[1]}\right)$$ $$Z^{[2]} = W^{[2]}A^{[1]} + b^{[2]}, \qquad A^{[2]} = \sigma\!\left(Z^{[2]}\right) = \hat{Y}$$

That's the forward pass, complete. Step through it on a real 2 → 3 → 1 network, and hover any cell to see the arithmetic that produced it:

Switch it to shapes only and read the column of dimensions, because that's the part worth memorising:

$$W^{[l]} : \left(n^{[l]},\, n^{[l-1]}\right) \qquad b^{[l]} : \left(n^{[l]},\, 1\right) \qquad Z^{[l]}, A^{[l]} : \left(n^{[l]},\, m\right)$$

where $n^{[l]}$ is the number of units in layer $l$ and $m$ is the number of examples. Every shape is forced by the ones around it. That's more useful than it sounds — the majority of bugs in this kind of code are transposes, and a wrong transpose usually can't multiply at all.

The bias is $(n^{[l]}, 1)$ and gets added to a matrix that has $m$ columns. That's broadcasting doing the obvious thing: one bias per unit, reused for every example, because a bias belongs to a unit and not to a data point.

Generalising to any depth is a for loop:

$$Z^{[l]} = W^{[l]}A^{[l-1]} + b^{[l]}, \qquad A^{[l]} = g^{[l]}\!\left(Z^{[l]}\right)$$

02 · Six lines

Now the derivatives. Same procedure as the single unit: seed the output, walk backwards, multiply by local derivatives. The only new thing is that "multiply" now means matrix multiply, and getting the shapes right is most of the work.

The output layer is free, because we did it already. With a sigmoid output and log loss, all the ugly terms cancel:

$$dZ^{[2]} = A^{[2]} - Y$$

From there, each layer needs three things: the gradient for its weights, the gradient for its bias, and the blame to pass down to the layer below.

$$dW^{[2]} = \frac{1}{m}dZ^{[2]}A^{[1]T} \qquad db^{[2]} = \frac{1}{m}\sum_{i}dZ^{[2](i)}$$ $$dA^{[1]} = W^{[2]T}dZ^{[2]} \qquad dZ^{[1]} = dA^{[1]} \odot g'\!\left(Z^{[1]}\right)$$ $$dW^{[1]} = \frac{1}{m}dZ^{[1]}X^{T} \qquad db^{[1]} = \frac{1}{m}\sum_{i}dZ^{[1](i)}$$

Six lines. Each one is a sentence:

• $dW$: a weight's gradient is its error times its input. $dZ$ says how wrong the unit was, $A^{T}$ says what it was looking at. Exactly the $x_1(a-y)$ from last post, batched.
• $db$: the bias has no input to multiply by, so its gradient is just the error, summed across the batch.
• $dA$: the same weights, used backwards. Forward, $W$ sends activations up; here $W^{T}$ sends blame back down, and each hidden unit collects blame in proportion to how much the layer above leaned on it.
• $dZ$: the one local factor. Blame arriving at $A^{[1]}$ has to cross the activation to reach $Z^{[1]}$, and $\odot$ — elementwise multiply — is how a per-element derivative gets applied.

The $\frac{1}{m}$ makes it the gradient of the average cost rather than the sum, which is what keeps the learning rate meaningful when the batch size changes.

Step through all six on real matrices:

Two things to check as you go. Every $dW^{[l]}$ has exactly the shape of $W^{[l]}$ — it has to, it's about to be subtracted from it. And $g'(Z^{[1]})$ for $\tanh$ is computed as $1 - (A^{[1]})^2$, using the output rather than the input, which is the identity from the last post finally paying rent.

Then the update, unchanged from one unit:

$$W^{[l]} := W^{[l]} - \alpha\, dW^{[l]}, \qquad b^{[l]} := b^{[l]} - \alpha\, db^{[l]}$$

03 · Proving the six lines

Six hand-derived equations is six chances to be quietly wrong. And "quietly" is the problem: a network with one bad gradient term doesn't crash, it trains slightly worse, and you find out three weeks later.

There's a cheap way to be sure. Backprop computes $\partial J/\partial\theta$ analytically. The definition of a derivative computes the same thing with two extra forward passes:

$$\frac{\partial J}{\partial \theta_i} \approx \frac{J(\theta_i + \varepsilon) - J(\theta_i - \varepsilon)}{2\varepsilon}$$

If the derivation is right, the two agree to about ten digits.

The interesting part is the other three buttons. Each is a real bug:

• Drop $\odot\, g'(Z)$ and notice what still passes — every gradient in layer 2 is perfect, because the missing factor sits below them. Only the first layer is wrong. A network like this still learns. It just learns worse, forever.
• Drop $1/m$ and every gradient is off by the same constant. The direction is right, so it trains fine and merely behaves as though the learning rate were 24× larger than the one you set.
• Flip the sign and you get relative error 1.0, the maximum. Every step climbs. The cost rises smoothly and monotonically, which looks much more like a bad learning rate than like a sign error.

Gradient checking is far too slow to train with — it costs two forward passes per parameter — which is exactly why it's the right tool for verifying. Run it once on a tiny network, then turn it off.

04 · Why the weights start random

The single-unit post said to initialise randomly and moved on. With a hidden layer the reason becomes visible:

Both networks are identical except for how $W^{[1]}$ started. Watch the weight matrix on the left: four rows, all identical, staying identical no matter how long it trains. If every unit starts with the same weights, every unit computes the same $z$, receives the same gradient and takes the same step. Four units doing one unit's job.

Starting at exactly zero is worse than that, and the left map shows it — there's no boundary at all, just flat colour. With $W^{[1]} = 0$ every activation is $\tanh(0) = 0$, which makes $dW^{[2]} = \frac{1}{m}dZ^{[2]}A^{[1]T}$ zero too. The whole network is frozen except for the output bias, so every prediction is the same number and accuracy sits at exactly 50%.

Any asymmetry breaks it. randn * 0.01 is the classic choice: random enough to separate the units, small enough that $z$ starts near the middle of the activation instead of out on a saturated tail. Biases can start at zero — it's the weights that decide whether two units ever see anything different.

A fixed 0.01 turns out to be a blunt instrument, though. A unit reading 2 inputs and a unit reading 500 want very different scales, and every figure here uses the fan-in-aware version instead:

$$W^{[l]} \sim \mathcal{N}(0,1) \cdot \sqrt{\frac{2}{n^{[l-1]}}}$$

That factor is He initialisation, and section 06 shows what it's protecting.

05 · The flower

Everything is in place. Same six equations, in a loop, on the dataset that beat a single neuron:

Press train. Somewhere around a thousand iterations the boundary stops being a line and starts growing petals, and the held-out accuracy climbs from 46% into the low nineties.

That's 17 parameters. Not a subtle model — it's the same neuron from the last post, four of them side by side, feeding a fifth.

Now move the $n_h$ slider, because the shape of that curve is the lesson:

• $n_h = 1$ stalls near 70%. One hidden unit is one line wrapped in a $\sigma$, so a hidden layer of size 1 can't beat logistic regression by much.
• $n_h = 3$ or $4$ is where it clicks — around 95% on data it has never seen.
• $n_h = 20$ hits 98.9% on the training set and drops to 88% on the held-out set. The extra capacity went into memorising individual points; look for the little islands the boundary wraps around single dots.

That gap between the two accuracies is overfitting, and it's the reason the figure holds out 30% of the data. Train accuracy alone would have told you the biggest network was the best one.

06 · Deeper, and what goes wrong

If one hidden layer helps, more should help more, and it does — up to a point that took the field two decades to get past.

The argument for depth is real. Deep networks build features in stages: edges, then shapes, then faces. And it's not only intuition — there are functions a deep network computes with $O(\log n)$ units that a shallow one needs $O(2^n)$ for. Computing the XOR of $n$ inputs is the standard example: pair them up in a tree and you need $\log n$ layers; do it in one hidden layer and you need a unit per input combination.

So why didn't people just stack fifty layers in 1995? Watch what reaches layer 1:

Each bar is $\|dW^{[l]}\|$ from a single backward pass through a freshly initialised network. With sigmoid activations and eight hidden layers, layer 1 receives a gradient of about $2 \times 10^{-5}$ while the last hidden layer gets $8 \times 10^{-2}$ — three and a half orders of magnitude apart. The cause is in the $dZ$ line: every layer the gradient crosses multiplies it by another $g'$, and $\sigma' \le 0.25$. Twenty of those in a row is $0.25^{20}$. The early layers aren't learning slowly, they aren't learning.

Switch the activation to ReLU and the bars flatten out. $g' = 1$ wherever a unit is active, so the product stops decaying geometrically. That change, plus initialising the weights to a scale matched to the fan-in, is most of what unblocked deep networks.

Then turn on skip connections and watch the left side lift by three orders of magnitude. The idea is one addition:

$$a^{[l]} = g\!\left(z^{[l]}\right) + a^{[l-1]}$$

The layer's output includes its own input. Backwards, that gives the gradient a second road home — the $+\,a^{[l-1]}$ passes it through untouched, so it arrives at layer 1 even when the multiplicative path has strangled it. It also changes what each layer has to learn: instead of a full transformation, it only has to learn a correction to what it already received, and a layer with nothing useful to add can settle on $g(z) \approx 0$ and get out of the way.

That one addition is why residual networks go hundreds of layers deep. It's also sitting inside every transformer block, which is where this is going next.

07 · What's next

Everything up to here is one object repeated. A unit is a weighted sum and a squash. A layer is units stacked into a matrix. Training is: guess, measure with a loss whose derivative cancels the activation's, walk the chain rule backwards, take a small step against the gradient. Six equations, and the only thing that changes as networks get larger is how many times you run them.

Both posts so far have trained on data I generated, which is convenient and a little too clean. The next one points the single unit from the first post at 209 actual photographs: what happens to an image on the way in, why nobody loops over examples, and what a weight looks like when you draw it back into a picture. It also covers softmax, which is how any of this answers a question with more than two possible answers.

Beyond that, the pieces you'd add, in roughly the order they'd help:

• Regularisation — the machinery for keeping the $n_h = 20$ run from memorising its training set
• Better optimisers — momentum, RMSProp, Adam. Plain gradient descent takes the same size step regardless of the terrain
• Mini-batches — these figures use all 280 examples per step; real training doesn't, and can't

Then the actual destination: attention, and how a transformer is assembled out of exactly the parts on this page. That's the next series.

The library behind every figure here is about a thousand lines of dependency-free TypeScript — matrices, activations, forward, backward, gradient checking. No framework, nothing hidden. The most useful hour I spent on it was the one where the gradient check failed and I found out my skip connections had been quietly wrong.