A photograph is 12,288 numbers in a row

The first post built one unit and the second stacked a few into layers. Both trained on datasets I generated — two blobs, a flower — because generated data is honest about what it is and small enough to watch.

Real data is not like that, and the week-2 assignment of the course is where it stops pretending. It takes 209 photographs, cat or not cat, and points the single unit from the first post straight at them. No hidden layer, no convolutions, nothing from the second post. Just $z = w^{T}x + b$, a sigmoid, and the gradient descent loop already derived.

It gets 70% of held-out photographs right.

That number is the whole reason this post exists. It is much better than it has any right to be, and it is also a ceiling that no amount of training gets past — and both halves of that come from one decision made before the model sees anything: how a picture becomes an $x$.

01 · The reshape

A colour image is a block of numbers: height, then width, then three channels. The course images are 64 by 64, so that is $64 \times 64 \times 3 = 12{,}288$ numbers per photograph. A unit takes a vector, not a block, so the block gets unrolled:

$$(64, 64, 3) \longrightarrow (12{,}288, 1)$$

In the notebook that is one line, train_set_x_orig.reshape(m_train, -1).T, and it is easy to read straight past. Hover a pixel and watch where its three numbers land:

Two things are happening here, and only one of them is obvious.

The obvious one is the ordering. Walk along the column and you pass through R, G, B of one pixel, then the pixel to its right, and at the end of a row you jump to the opposite edge of the image — x[191] is the last pixel of row 0, x[192] is the first pixel of row 1. Adjacent in the column, opposite sides of the picture.

The one worth stopping on: the model is never told any of this. It receives 12,288 numbers and assigns a weight to each. It has no idea that two of them came from touching pixels, or that the image had two dimensions, or that there were channels. Permute all 12,288 features — the same permutation for every example — and logistic regression is unmoved. Not approximately unmoved: each weight's gradient depends only on its own feature, so with the weights starting at zero the fitted $w$ comes out permuted the same way and the accuracy matches to the last digit.

That is a strange thing to have thrown away on line one, and everything convolutions do is a way of putting it back.

The other half of the preprocessing is the /255, which the notebook also does in one throwaway line. Raw bytes go to 255, and $z = w^{T}x + b$ summed over 12,288 of those is an enormous number before training has moved anything. Enormous $z$ is the flat tail of the sigmoid, where $\sigma' \approx 0$, which the first post covered as a reason to keep weights small — the same argument applies to inputs, and dividing by 255 is the cheapest possible fix.

02 · Whose loop runs

The assignment is emphatic that you must not loop over the 209 examples. Instead of one $z$ at a time, all of them at once:

$$Z = w^{T}X + b, \qquad A = \sigma(Z), \qquad dw = \frac{1}{m}(A - Y)X^{T}, \qquad db = \frac{1}{m}\sum_i (a^{(i)} - y^{(i)})$$

$X$ is $(12{,}288 \times 209)$ — one column per photograph — and the four lines above are the entire gradient computation for the entire dataset. The reason given is speed, and it is worth checking that claim rather than repeating it. Both versions are in the figure, computing identical numbers:

They agree to about $10^{-16}$, which is float noise. And at the assignment's own $m = 209$, on my laptop, they finish within a few percent of each other. Drag $m$ out to 2000 and the matrix version pulls ahead to roughly 1.5×.

That is nothing like the speedup the assignment is selling, and the reason why is the useful part.

The assignment is written in Python, and vectorising is not about matrices being fast. It is about whose loop runs. In Python the example loop runs in the interpreter, one bytecode dispatch per multiply, while np.dot hands the identical arithmetic to compiled BLAS — so deleting the loop is worth orders of magnitude. In a browser both versions are JavaScript that the JIT compiles to machine code. The loop was never the bottleneck, so the win shrinks to memory order and allocation: real, and small.

Which means the real argument for the matrix form was never the benchmark. It is that $A = \sigma(wX + b)$ is the same expression for one example or two thousand, with $m$ appearing exactly once, in the $\frac{1}{m}$. Every figure in these three posts is built that way, and the second post's six equations are only six because of it.

03 · 209 photographs

Here is the run, at $\alpha = 0.005$ for 2000 iterations:

That figure is a recording, and it is the only one in these three posts that is. The 209 photographs are a course asset I no longer have a copy of, so I cannot rerun this in your browser, and inventing numbers for it would be worse than saying so. The costs and accuracies above are the notebook's own printed output, transcribed.

One detail in that notebook is worth flagging, because it looks like it contradicts the last post. The assignment initialises $w$ to zeros — and the second post spent a section on why you must never do that. Both are right. Zero weights are fatal when there are several units in a layer, because identical weights make identical gradients and the units never diverge. There is exactly one unit here, so there is no symmetry to break, and zero is a perfectly good starting point. The rule was never "random is magic", it was "two units must not start life indistinguishable".

The numbers:

• 99.04% on the training photographs, 70.0% on the held-out 50. A 29-point gap.
• Turning $\alpha$ up to 0.01 makes the memorising worse and the generalising worse: 99.52% train, 68% test.
• Turning it down to 0.0001 gives 68.42% train and 36% test — worse than a coin, which manages 50% without looking at the photographs at all.

The 29-point gap is not a surprise once you count parameters. There are 12,288 weights and 209 examples: 58 dials per photograph. A model with that much room does not need to find a pattern that generalises when it can simply accommodate each example individually, and the second post showed the same thing happening at $n_h = 20$ on the flower.

The 70% is the surprising half. A single straight boundary, in a space with no geometry, gets 70% of unseen photographs right.

04 · The model is a picture

So why does it stop at 70%, and what is it actually doing to get that far?

$w$ has one entry per pixel. Which means $w$ can be folded back into the shape of the images it was learned from and looked at directly — the model is not an abstraction here, it is a 64×64 picture. I cannot do that with the cat weights, but I can rebuild the situation from scratch, which turns out to be more informative anyway:

Start on blob up / down, where the class is which half of the frame a blob sits in. Train it, and the weight image resolves into something you can say out loud: blue up top, red along the bottom. $w^{T}x$ has become "how much ink is up high, minus how much is down low", which is exactly the question, so one template answers it and held-out accuracy goes to 100%.

Now switch to ring / disc. Same 784 weights, same code, same loop. The two classes differ only in where the ink sits — an annulus versus the same shape filled in — and the shape appears anywhere in the frame. Total brightness is equalised across every image, so counting lit pixels is not an available shortcut.

Let it run. Training accuracy climbs past 99% while held-out accuracy parks around 70%.

Those are the cat numbers, from a dataset with no cats in it. And the weight image shows why, because it never resolves into a ring detector: a template is nailed to fixed coordinates, and the thing it is looking for keeps moving. A weight at pixel (12, 20) can only ever mean "bright here is evidence", and there is no arrangement of 784 such statements that expresses "a ring, wherever it is". The model does the only thing left — averages over every position the ring appeared in during training — and that average is a blur that happens to be right about 70% of the time.

That is the ceiling. Not a training problem, not a data-volume problem. A single template cannot be in two places at once, and photographs put things in different places. The fix is a unit that slides the same small template across the image and reports where it fired, which is a convolution, and which is a different post.

05 · More than two answers

One thing from week 1 that the rest of the course quietly depends on, and this series will lean on entirely.

Sigmoid answers one question: how likely is $y = 1$. Most interesting questions have more than two answers, and for those the last layer is softmax:

$$\text{softmax}(z)_i = \frac{e^{z_i}}{\sum_{k} e^{z_k}}$$

$k$ scores in, $k$ probabilities out, guaranteed to sum to 1.

Nothing from the first post is lost. With $k = 2$, softmax is the sigmoid, applied to the difference of the two logits: $\text{softmax}([z_0, z_1])_0 = \sigma(z_0 - z_1)$. The same unit, given more than one output.

Then switch the logits to ≈1000 and watch the naive column — the formula transcribed exactly as written above — return NaN. $e^{1000}$ overflows a float64 somewhere past $z \approx 709$, and $\infty / \infty$ is undefined. The fix is one line: subtract the largest logit before exponentiating. It is free because adding a constant to every logit cannot change the result — the shift cancels between numerator and denominator — so you may as well choose the constant that guarantees the biggest exponent is $e^0 = 1$. Every library does this, and the definition as printed in every textbook does not.

The temperature slider divides the logits before the exponent. Nothing about the model changes — same weights, same logits — but at $T = 0.1$ the distribution collapses onto its winner and at $T = 3$ it flattens towards uniform. That is the knob on every language model's sampling, and it is one division.

06 · What's next

Three posts, and the object has not changed once. A unit is a weighted sum and a squash. A layer is units in a matrix. Training is guess, measure, walk the chain rule backwards, step against the gradient. What changed in this one is only what goes in the front and what comes out the back: pixels in, and a distribution over $k$ answers out instead of one number.

The two ceilings this post ran into are both about structure the reshape destroyed:

• Position. One template, fixed coordinates, and a pattern that moves. Convolutions.
• Order. Nothing in a column of 12,288 numbers says which came first, and for language that is the entire content. Attention.

Softmax over a vocabulary, a loss that is cross-entropy against the token that actually came next, and the same six equations from the second post — that is a language model, once the layer in the middle knows how to look at the rest of the sentence. That layer is the next series.

The library behind every live figure here is still about a thousand lines of dependency-free TypeScript, now with images in it. The generated datasets are seeded, so the boundary described above is the one you see. The one recording is labelled as one.