A neuron is a line and a squash

I took Andrew Ng's deep learning specialization and did what everyone does with course notes: filed them away and forgot most of it. So I'm rebuilding it here, from the bottom, with the one addition that would have saved me a month — the equations on this page are wired to code you can move.

Nothing below is a simulation. There's a small neural network library behind these figures, about a thousand lines of TypeScript with no dependencies, and it does real forward passes, real backward passes and real gradient descent. When a figure claims a derivative equals something, the number came out of running it. In the next post you can break its backward pass on purpose and watch a gradient check catch you.

This post covers exactly one neuron. That sounds like a small target, and it is, but everything after it — layers, backprop, and eventually a transformer — is this same object repeated. Get the single unit fully in hand and the rest is bookkeeping.

01 · Three numbers

A unit takes a vector of numbers and produces one number. It does it in two moves.

First, a weighted sum:

$$z = w^{T}x + b = w_1x_1 + w_2x_2 + \dots + w_nx_n + b$$

The weights $w$ say how much each input matters. The bias $b$ is the part people skip past, and it's the only thing standing between you and a model that can't say anything the origin doesn't already say — with $b = 0$, $x = 0$ must map to $z = 0$, forever.

Then a squash, to turn that number into something between 0 and 1:

$$\hat{y} = a = \sigma(z) = \frac{1}{1 + e^{-z}}$$

$y$ is the true label, $\hat{y}$ is what we said, and $\sigma$ is the activation function. Three numbers — $w_1$, $w_2$, $b$ — and the whole unit is on screen. Drag them:

The dashed arrow is $w$ itself, and the boundary is always perpendicular to it, because $z$ grows fastest in the direction $w$ points. That's not a coincidence to memorise, it's just what a dot product is: $w^{T}x$ measures how much of $x$ lies along $w$.

Two things worth doing before moving on. Set $b$ to zero and try to separate the blobs — the line is pinned to the centre of the plot and you can only spin it. And crank both weights up: the colours get harder, the uncertain band collapses. $\|w\|$ is the unit's confidence, and confidence is not the same thing as being right.

02 · The squash

Why squash at all? Because without it, depth is a lie. Stack two linear layers and you get

$$W^{[2]}\left(W^{[1]}x\right) = \left(W^{[2]}W^{[1]}\right)x$$

which is one linear layer wearing two matrices. A hidden layer without a non-linearity is not a hidden layer. It's a wasted matrix multiply.

So each unit needs a function that bends. There are several, and the honest way to compare them is next to the thing backprop will actually ask them for, which is not $g$ but $g'$:

Click through them. The shapes matter less than the derivatives:

• Sigmoid maps to $(0,1)$, which is what you want from an output that means "probability". Its derivative has the nice property $\sigma'(z) = \sigma(z)(1-\sigma(z))$ — the derivative is computable from the output, so a network that cached $a$ never has to look at $z$ again. But $\sigma' \le 0.25$ always, and it's essentially zero across most of the range. A saturated sigmoid unit doesn't learn.
• tanh is the same S shifted to be centred on zero, with $\tanh'(z) = 1 - \tanh(z)^2$. Centred outputs are easier for the next layer to work with, which is why it beats sigmoid everywhere except the output.
• ReLU is $\max(0, z)$, and its derivative is 1 or 0. Nothing saturates on the positive side. It's undefined at exactly $z = 0$, which sounds like a problem and never is — the odds of a float landing exactly there aren't worth an if.
• GELU is ReLU with the corner smoothed off, and it lets slightly-negative inputs through instead of zeroing them. GPT-2 shipped with the $\tanh$ approximation of it. It's what you'll meet later in this series.

The pattern to hold onto: an activation whose derivative is small most of the time is an activation that throttles learning. That sentence looks like trivia now. In the next post it becomes the reason deep networks didn't work for twenty years.

03 · How wrong is wrong

The unit makes a prediction. We need a number saying how bad it was, so we can make it less bad. The obvious choice, the one everyone reaches for first, is squared error:

$$\mathcal{L}(\hat{y}, y) = \tfrac{1}{2}(\hat{y} - y)^2$$

It's wrong here, and the reason is worth seeing rather than being told:

Drag $\hat{y}$ toward the wrong end. The left chart is the loss, and both curves rise — squared error looks fine. The right chart is the gradient that reaches $z$, and that's where they split. Squared error's is

$$\frac{\partial \mathcal{L}}{\partial z} = (\hat{y} - y)\,\hat{y}(1-\hat{y})$$

That trailing $\hat{y}(1-\hat{y})$ is $\sigma'$, dragged along by the chain rule. It's near zero exactly when the model is most confident — including when it's confidently, catastrophically wrong. Squared error hands its worst mistakes the smallest correction.

The fix is to choose a loss whose derivative cancels that factor. Log loss:

$$\mathcal{L}(\hat{y}, y) = -\left[y\log\hat{y} + (1-y)\log(1-\hat{y})\right]$$

Read it as two cases, because it is two cases:

$$\begin{cases} y = 1: & \mathcal{L} = -\log\hat{y} \quad \Rightarrow \text{ we want } \hat{y} \text{ large} \\ y = 0: & \mathcal{L} = -\log(1-\hat{y}) \quad \Rightarrow \text{ we want } \hat{y} \text{ small} \end{cases}$$

It isn't a formula someone picked because it worked. It falls out of asking for the most likely parameters. For a binary label, the model's claim about a single example is

$$P(y \mid x) = \hat{y}^{\,y}(1-\hat{y})^{1-y}$$

which is a compact way of writing "if $y = 1$ the probability is $\hat{y}$, and if $y = 0$ it's $1 - \hat{y}$" — substitute and check. We want to maximise that. Logs are monotonically increasing, so maximising the log maximises the thing:

$$\log P(y \mid x) = y\log\hat{y} + (1-y)\log(1-\hat{y}) = -\mathcal{L}(\hat{y}, y)$$

Maximising the likelihood is minimising its negative. The minus sign in front of log loss is the entire difference between the two.

One example isn't a model. The cost is the loss averaged over the whole training set:

$$J(w, b) = \frac{1}{m}\sum_{i=1}^{m}\mathcal{L}\!\left(\hat{y}^{(i)}, y^{(i)}\right) = -\frac{1}{m}\sum_{i=1}^{m}\left[y^{(i)}\log\hat{y}^{(i)} + \left(1-y^{(i)}\right)\log\left(1-\hat{y}^{(i)}\right)\right]$$

Superscripts in parentheses index examples, and they will keep meaning that for the rest of the series. Square brackets, when they show up, mean layers.

04 · Derivatives, one edge at a time

We have a number to minimise. To minimise it we need to know which way each parameter should move, which means partial derivatives, which sounds like the point where you're supposed to open a calculus textbook.

You don't need one. Any expression can be drawn as a graph of small operations, and the derivative of the whole thing with respect to any input is a walk backwards through that graph, multiplying local derivatives as you go. Start with something with no machine learning in it at all:

$$J(a,b,c) = 3(a + bc) \qquad u = bc, \quad v = a + u, \quad J = 3v$$

Step it backwards. Seed the output with $\frac{\partial J}{\partial J} = 1$, then at every node multiply by the local derivative of that node with respect to its input:

$$\frac{\partial J}{\partial v} = 3, \qquad \frac{\partial J}{\partial u} = \frac{\partial J}{\partial v}\frac{\partial v}{\partial u} = 3 \cdot 1, \qquad \frac{\partial J}{\partial b} = \frac{\partial J}{\partial u}\frac{\partial u}{\partial b} = 3 \cdot c$$

One walk, every derivative. Not one walk per parameter — one walk, total. That property is the only reason training a model with billions of parameters is affordable, and it's the whole of backpropagation. Everything in the next post is this, with matrices.

Now switch the figure to the logistic example and step through it. The two ugly derivatives are

$$\frac{\partial \mathcal{L}}{\partial a} = \frac{a - y}{a(1-a)}, \qquad \frac{\partial a}{\partial z} = a(1-a)$$

and their product is

$$\frac{\partial \mathcal{L}}{\partial z} = \frac{a-y}{a(1-a)} \cdot a(1-a) = a - y$$

The figure computes both numbers independently and they land on the same value, to every digit it prints. This is the cancellation section 03 was engineered for, and it's why log loss stuck: with a sigmoid output, the gradient at $z$ is the error. Everything below follows in one line each:

$$\frac{\partial \mathcal{L}}{\partial w_1} = x_1(a - y), \qquad \frac{\partial \mathcal{L}}{\partial w_2} = x_2(a - y), \qquad \frac{\partial \mathcal{L}}{\partial b} = a - y$$

A weight's gradient is its input times the error. The bias has no input to multiply by, so it just gets the error. Worth remembering — the same two sentences describe every layer of every network in this series.

05 · Rolling downhill

Gradients tell us which way is up. So we go the other way:

$$w := w - \alpha\frac{\partial J}{\partial w}, \qquad b := b - \alpha\frac{\partial J}{\partial b}$$

The minus is because we want a minimum. $\alpha$ is the learning rate, and it's the one number here with no principled value — you pick it, and picking it badly is the most common way to waste a day.

The right-hand panel is the cost surface with the bias pinned at zero, so it's the whole parameter space rather than a slice of it. The white trail is the path. Turn $\alpha$ down to 0.001 and it crawls; nothing is broken, each step is just a rounding error. Turn it past 10 and the path starts bouncing across the valley instead of walking down it — the gradient points the right way, but the step taken along it is longer than that direction stays valid for.

Two details from the notes that matter later:

Initialise $w$ randomly, not at zero. With two units starting identical, they compute the same $z$, get the same gradient, and take the same step — forever. Two units doing one unit's job. Biases can start at zero; it's the weights that decide whether units ever diverge.

Keep the initial weights small. Large weights mean large $z$, and large $z$ on a sigmoid or tanh means you start out on the flat part of the curve where $g' \approx 0$. The classic move is randn * 0.01; the modern one scales by the number of inputs a unit reads, which the next post has a figure for.

Now switch the figure to the flower.

Watch the cost drop, flatten out around 0.66, and stop. Accuracy parks around 46% — on a balanced two-class dataset, worse than guessing. And gradient descent is doing everything right: the path walks straight down to the bottom of the bowl. The bottom of the bowl is a straight line drawn through a flower.

06 · What's next

A single unit draws a single straight boundary. That's not a bug in the training, it's the model's entire vocabulary. The fix is more units, arranged in layers, and the interesting part is that the training procedure barely changes — the same chain rule, walked back through more nodes.

That's the next post: the forward pass as matrices, backprop as six equations, why the boundary suddenly curves, and what breaks when you stack ten layers instead of two.

After that, the same machinery goes toward how language models are built. It's the same neuron the whole way up.