The loss reads six numbers out of 301,542

Last time finished the model. 163,009,536 parameters with the output head counted separately, a forward pass that turns the four ids of "Hello, I am" into a (1, 4, 50257) tensor of logits, and six greedy steps that answered Featureiman Byeswickattribute argue. That output is not a bug. It is what 163 million numbers drawn from a random initialiser have to say, and the only useful response to it is to stop reading the text and start measuring it.

This post is that measurement. Chapter 5 runs the same untrained model on a two-row toy batch and gets 10.7940, and every part of that number is worth taking apart: where the 10 comes from, why it is a logarithm at all, and what it becomes when you exponentiate it back.

Three of the seven figures below compute everything they show. The other four put a live computation beside numbers transcribed from the book, and each says so in its own footer: the toy batch's token ids and its one decoded output in section 01, the six target probabilities in section 02 and the six per-token perplexities section 05 derives from them, and the two untrained corpus losses in section 04. Those came out of a real 124-million-parameter GPT-2 with random weights, which is 250 MB at half precision, and this blog does not ship that. Everything downstream of them is recomputed here, which is why one figure prints 48,725.29 where the book prints 48725.8203. Section 06's figure is live all the way down: it tokenizes the real 20,479 characters of "The Verdict" with the real GPT-2 byte-level BPE, and every count it reports is computed when you move the slider.

01 · Two functions that look like nothing

Chapter 5 opens with two helpers, four lines each, and the only interesting thing in either of them is a dimension appearing and disappearing.

def text_to_token_ids(text, tokenizer):
    encoded = tokenizer.encode(text, allowed_special={'<|endoftext|>'})
    encoded_tensor = torch.tensor(encoded).unsqueeze(0)
    return encoded_tensor

def token_ids_to_text(token_ids, tokenizer):
    flat = token_ids.squeeze(0)
    return tokenizer.decode(flat.tolist())

unsqueeze(0) adds the batch dimension the model insists on; squeeze(0) takes it away again. Run the untrained model through them and it continues "Every effort moves you" with rentingetic wasnم refres RexMeCHicular stren, which is the previous post's failure wearing different clothes. The chapter changes one thing in the config to make that cheap: context_length drops from 1,024 to 256, so the whole chapter runs on a laptop. Everything else stays, including qkv_bias: False, which is why the book's parameter counts sit 27,648 below the real checkpoint's.

Then the batch the rest of the section lives on. inputs and targets are both (2, 3), targets being inputs shifted one position forward, and the forward pass returns (2, 3, 50257). That is 301,542 numbers. The loss is about to read six of them.

Before it does, the chapter takes the argmax over that last axis, which gives (2, 3, 1), and decodes the first row: Armed heNetflix against a target of effort moves you. That comparison is the honest starting point and also a dead end, because a decoded string tells you the model is wrong without telling you how wrong, or in which direction, or whether yesterday's version was worse. Flattening for the loss gives (6, 50257) and (6,), and from there the answer is one number.

Drag the sequence length and watch the two counters in the footer, because the ratio between them is the idea:

At every setting the right-hand counter is exactly the vocabulary size smaller than the left-hand one. One more thing worth unlearning here: the book describes the 50,257 in torch.Size([2, 3, 50257]) as the embedding dimensionality determined by the vocabulary size. It is the vocabulary size. 768 is the embedding dimension, and the output head is the thing that turns one into the other. A reader who takes that sentence literally will conflate the two for the rest of the book.

02 · Six steps, then one call

Figure 5.7 breaks the loss into six steps: logits, softmax, pick the probability at the target index, log, mean, negate. Steps 1 to 3 have already run by this point in the chapter, which is what makes the rest so small. The picking is one line of advanced indexing, probas[text_idx, [0, 1, 2], targets[text_idx]], and it pulls one scalar per position rather than a slice:

$\pi_{b,t} = P\big(\text{token} = y_{b,t} \mid \text{context}\big) = \mathrm{probas}[b, t, y_{b,t}]$

Six of those come out: 7.4541e-05, 3.1061e-05, 1.1563e-05 for the first row and 1.0337e-05, 5.6776e-05, 4.7559e-06 for the second. Steps 4, 5 and 6 are three lines on top of them, and then three more lines produce the same number a completely different way.

log_probas = torch.log(torch.cat((target_probas_1, target_probas_2)))
avg_log_probas = torch.mean(log_probas)
neg_avg_log_probas = avg_log_probas * -1        # tensor(10.7940)

logits_flat = logits.flatten(0, 1)              # torch.Size([6, 50257])
targets_flat = targets.flatten()                # torch.Size([6])
loss = torch.nn.functional.cross_entropy(logits_flat, targets_flat)
                                                # tensor(10.7940)

Two of that second three are only reshaping. So the whole of cross entropy, for a language model, is:

$L = -\frac{1}{N}\sum_{n=1}^{N} \log \pi_n, \qquad N = B \times T = 2 \times 3 = 6$

Step through the six, then switch to the live column and edit the logits, because only one cell per row is ever read:

The other 50,256 probabilities in each row affect the loss only through the denominator that made them sum to one. Push one target logit up and every other probability in that row falls without anyone touching it, which is why you only ever have to measure one. Two footnotes. Raschka averages the logs and then negates; Giles Thomas negates and then averages; the two agree because negation is linear. And F.cross_entropy takes logits, not probabilities, because it runs the softmax itself, so handing it softmaxed output is a silent bug rather than an error.

03 · Why the logarithm

The general form of cross entropy is a sum over the whole vocabulary,

$H(p, q) = -\sum_{i=1}^{V} p_i \log q_i$

and it is fair to ask where the logarithm comes from before accepting that the sum collapses to one term. Giles Thomas builds it up in his part 20 rather than inheriting it, and the argument is the clearest thing I read while working through this chapter.

Start with what a surprise function has to do: be large for unlikely events and small for likely ones. The obvious candidate is $1 - p$. It fails three ways. Spacing: moving from $p = 0.999$ to $p = 0.5$ takes it from 0.001 to 0.5, and moving from 0.5 to 0.001 takes it from 0.5 to 0.999, two equal steps for two wildly unequal changes in how surprised you should be. Additivity: one die showing a three has probability 1/6 and surprise 0.8333; two dice showing threes has probability 1/36 and surprise 0.9722, and no arithmetic relates those two numbers. Slope: the derivative is $-1$ everywhere, so it claims the gap between 0.99 and 0.98 is the same size as the gap between 0.02 and 0.01.

Drag the probability and read the three tabs below the curves, because each one rules out a different candidate:

Under $-\ln p$ the dice give 1.7918 nats and 3.5835 nats, exactly double, and that is the test that matters. Additivity over independent events is what makes a sum over six positions mean anything at all.

The rest builds quickly. Scale each surprise by how often the event actually happens and you get entropy, $H(p) = -\sum p \log p$, which Giles Thomas is honest about disliking, because both halves of the product are the same number. Cross entropy fixes exactly that: the surprise comes from the model's $q$ and the frequency comes from reality's $p$, so the two halves are finally different things. And when $p$ is one-hot, every term where $x$ is not the target contributes $0 \cdot \log q(x) = 0$, so the whole sum is $-\log q(\text{target})$, which is where section 02 started.

$p \text{ one-hot} \implies H(p,q) = -\log q(x_{\text{target}})$

One note on units, because everything after this depends on it. Base 2 gives bits, the natural log gives nats, and PyTorch uses nats. It does not matter which you pick as long as you say which: 10.7940 nats is 15.5725 bits.

04 · 10.79 was predictable

A model that knows nothing spreads its probability roughly evenly, so it assigns about $1/V$ to whatever comes next, and its loss has to land near

$L_{\text{random}} \approx -\log\!\left(\frac{1}{V}\right) = \log V, \qquad \ln 50257 = 10.8249$

That makes the first loss you ever print the cheapest bug check in the chapter. If it comes out at 3, your targets are leaking. If it comes out at 40, your logits are unscaled or your indices are wrong. The book's three measurements are 10.98758347829183 on the training split, 10.98110580444336 on validation and 10.7940 on the toy batch, all within 0.17 nats of the prediction.

Set the vocabulary size and read the line the loss has to start on:

The part the book does not point at is the direction of the miss. Both corpus numbers sit above $\ln V$, not below it. Exponentiate the training one and you get 59,135 against a vocabulary of 50,257, a factor of 1.177. An untrained model is not uniform. It is randomly peaked, and a random peak in the wrong place costs more than flatness, so it does slightly worse than a flat guess over the whole vocabulary.

05 · Perplexity says 4 twice

Perplexity is the loss on a scale with units. Raise the base of your logarithm to the power of the loss: nats, so torch.exp. For one-hot targets it collapses in four lines to the reciprocal of one probability,

$\mathrm{PPL} = e^{L} = e^{-\log \pi} = \frac{1}{\pi} \quad \text{(one-hot, one position)}$

and the sanity checks all work. Certain and correct gives 1. Uniform over $V$ gives $V$. An even split between four tokens with the rest at zero gives 4, which is what makes Raschka's phrase, that perplexity signifies the effective vocabulary size the model is uncertain about, feel right.

Then Giles Thomas finds the wrinkle, in his part 21. Four-token vocabulary, target at index 1. The distribution [0.25, 0.25, 0.25, 0.25] gives $p_{\text{correct}} = 0.25$ and perplexity 4. So does [0.0, 0.25, 0.0, 0.75]. The first model was spread across four options. The second was down to two, and it picked the wrong one.

Drag the four bars and watch the two readouts disagree:

Their Shannon entropies are 2.0000 bits and 0.8113 bits, and $2^H$ is 4.000 against 1.755. That is the correction, and it is worth stating in his terms: perplexity is not a measure of how many vocabulary items the model was choosing between, because that quantity is the entropy of the model's own output and needs no reference to the target at all. Perplexity is computed against the right answer. It charges a model for being confidently wrong at exactly the same rate as for being uniformly confused.

The second half of the sidebar is that the average hiding inside perplexity is geometric, not arithmetic:

$\mathrm{PPL} = \left(\prod_{n=1}^{N} \frac{1}{\pi_n}\right)^{1/N} = \prod_x \left(\frac{1}{q(x)}\right)^{p(x)}$

The book's own six positions make that concrete. Their per-token perplexities are 13,415 / 32,195 / 86,483 / 96,740 / 17,613 / 210,265. The arithmetic mean is 76,118.5. The geometric mean is 48,725.29, which is what torch.exp computes on the loss those six produce. The book prints 48725.8203 because it exponentiated the unrounded loss rather than the five-significant-figure probabilities it had printed a page earlier; the gap is rounding, and it is worth saying which number came from where rather than quietly picking one.

Now drag any one of the six per-token perplexities the book's own example produced:

The two averages fail in opposite directions, which is the reason to care which one you have. Raise the largest of the six to 1,000,000 and the arithmetic mean goes from 76,118.5 to 207,741, a factor of 2.73, while the geometric mean moves from 48,725.29 to 63,187, a factor of 1.30. Drop the smallest to 1.0 instead and the arithmetic mean barely notices, 76,118.5 down to 73,883, while the geometric mean falls to 9,996. The arithmetic mean is dominated by its largest term and the geometric mean by its smallest, so a single catastrophic token cannot run away with perplexity, and a single perfect one pulls it down hard.

The closing move is not in the book at all. Ten sequences completed by mat six times, lap three times and dog once give exponents 6/10, 3/10 and 1/10, which are exactly the true $p(x)$. One-hot targets over a corpus recover the distribution nobody wrote down, so nothing is thrown away by pretending each individual token was certain.

06 · A real corpus, split in the middle of a word

Scaling from six tokens to a corpus is bookkeeping, and the bookkeeping has one sharp edge. The file is "The Verdict" again, 20,479 characters and 5,145 GPT-2 tokens, and train_ratio = 0.90 is applied with int(0.90 * 20479) = 18431, which slices the string, not the token list. Tokenize the two halves independently:

$ npx tsx scripts/llm/part07-split.mts
chars       20479
tokens       5145
split_idx   18431                        <- int(0.90 * 20479), on characters
train       18431 chars     4612 tokens
val          2048 chars      534 tokens
                                         <- 4612 + 534 = 5146, one more than the file has
seam        ...ed my 'techniq | ue' collapsed ...
windows        18 train        2 val     <- max_length = stride = 256
batches         9 train        1 val     <- batch_size 2, drop_last True / False
per epoch    4608 train      512 val
unread         24 tokens

The extra token exists because the cut falls between the q and the u of technique. Neither fragment is a piece the whole word would have produced, so the two halves together carry one token the file does not have.

Drag the ratio and watch the seam, because the split is on characters and the tokenizer does not care where you cut:

The window arithmetic underneath is the same closed form post three shipped, called rather than rewritten, and the two agree on all 3,328 window configurations its check covers. At max_length = stride = 256 the windows do not overlap, so 4,612 tokens give 18 rows and 534 give 2. At batch_size = 2 with drop_last=True on training and False on validation, that is 9 training batches and 1 validation batch, which is exactly what the book prints. Nine batches of (2, 256) is 4,608 input tokens per epoch, one batch is 512, and 24 tokens are never read in either role.

Two more details from listing 5.2 that matter next post. calc_loss_loader takes an optional num_batches cap, and the training loop calls it with eval_iter=5, so its printed training loss is over 5 of the 9 batches while its validation loss is over 1 of 1. Those are not the same quantity as the baselines. And it opens with an explicit length check that returns nan for an empty loader, which is why pushing the ratio to 0.99 in the figure above gives you zero validation batches rather than a crash. The baselines themselves, 10.98758347829183 and 10.98110580444336, sit right on the $\ln 50257$ line from section 04.

07 · What's next

What is missing from what was built here, in rough order of usefulness. A loop that changes the weights. Everything above measures and nothing learns. zero_grad, forward, loss, backward, step, with AdamW at lr = 0.0004 and weight_decay = 0.1, ten epochs, about five minutes on a laptop. The book's training loss falls from 9.781 to 0.391 while validation falls from 9.933 to 6.452 and stops, and 6.452 nats is a perplexity of 634. A second curve, because one loss number is a reading and two, one on data the model trains on and one on data it does not, is a diagnosis; the gap between them opens after epoch 2. A reason to care about the tail, since the loss reads one probability per position but decoding reads the whole row, which is where temperature and top-k live. And a larger corpus: 5,145 tokens against Llama 2's 2 trillion is a factor of $3.9 \times 10^8$, and the book's sidebar puts about $690,000 at the other end of that ratio.

It is worth saying why a loss function from 2019 is worth a post of its own. Nothing in sections 02 through 05 depends on which normalisation, activation or positional scheme the block uses. The loss reads one probability per position and the entire architecture is upstream of it, which is a claim this post demonstrates rather than borrows. What has changed inside the block since then is a later post in this series.

The figures run on src/lib/minigpt, dependency-free TypeScript, seeded, sitting on the same stable softmax, cross entropy and entropy helpers the neural-network posts used. The tokenizer in the corpus figure is the real GPT-2 byte-level BPE from part two, checked against four published tiktoken vectors. The thing that took longest was not any of the code: it was believing the arithmetic. A 90/10 split turns a 5,145-token file into 5,146 tokens, and I re-ran the encoder three times before I accepted that the file was fine and the seam was real.