Eight steps to train it, fourteen runs to tune it

Last time we built a number. A model with 50,257 tokens in its vocabulary and no idea what comes next spreads its probability roughly evenly, so its cross entropy has to land near $\ln(50{,}257) = 10.8249$, and the book's untrained GPT-2 measures 10.98758 on the training split and 10.98111 on validation. Exponentiate the first: 59,135, effectively choosing at random among fifty-nine thousand tokens. That number is what this post drives down.

Driving it down is the least interesting part of what follows, and the book agrees. Its own text calls the training loop a standard procedure, and figure 5.11 draws it as eight steps that would look identical for an image classifier. Everything worth arguing about sits either side: what the two loss curves mean while it runs, what you do with the frozen weights afterwards, and where the numbers you handed the optimiser came from. The short answer to the last one is that somebody carried them out of the book's five-minute laptop run into a 3.26-billion-token one and spent fourteen training runs finding out whether they were right.

Four of the seven figures below compute everything they show: the transformer that trains in this tab, the alignment counter, the top-k table and the parameter counter. The other three put a live computation beside transcribed numbers and say so in their own footers: the book's printed training log, the book's own thousand draws, and Giles Thomas's published test-set losses. Real GPT-2 weights are not here and will not be. GPT-2 small is a quarter of a gigabyte at half precision.

01 · Five minutes on a laptop

Figure 5.11 lays out the loop in eight steps and listing 5.3 implements them one for one. The inner half is seven lines:

for input_batch, target_batch in train_loader:
    optimizer.zero_grad()
    loss = calc_loss_batch(input_batch, target_batch, model, device)
    loss.backward()
    optimizer.step()
    tokens_seen += input_batch.numel()
    global_step += 1

Reset the gradients, measure, assign blame, step. The only line that knows it is a language model is calc_loss_batch, and all that does is flatten (B, T, 50257) into (B·T, 50257) and hand it to cross_entropy.

The run around it: GPT_CONFIG_124M with context_length cut from 1,024 to 256, which makes it a laptop job; "The Verdict" at 20,479 characters and 5,145 tokens; batch_size=2 with max_length = stride = 256, giving 9 training batches and 1 validation batch; AdamW(lr=0.0004, weight_decay=0.1); ten epochs, so 90 optimiser steps and 4,608 tokens an epoch.

One detail the book leaves implicit will trip you. global_step starts at −1 and is incremented before the modulo check, so the first evaluation fires at step 0, after the first weight update, which is why the log opens at 9.781 rather than 10.98758.

The book puts the whole thing at about five minutes on a MacBook Air. Giles Thomas ran the same code on an RTX 3090 in just under eleven seconds, roughly 27 times faster: the argument for GPUs in one line.

Here is that loop running, on a model small enough to fit in this page. Press train and watch the two lines:

That is a character-level GPT with 28,064 parameters over a 55-symbol alphabet, forward and backward written by hand, on the first 2,592 characters of the same short story. Both lines start near $\ln 55 = 4.0073$, the loss of a model that has not yet noticed that e is more common than q. They fall together, and then one stops: training drops below 1.6 while validation climbs past 2.4 inside half a minute on both machines I measured it on. That parting is the next section.

02 · The gap between the two curves

The book's run: training loss 9.781 down to 0.391, validation 9.933 down to 6.452, separating past the second epoch. Convert and the second acquires units. Validation perplexity goes from 20,599 to 634, and 634 is not "good but a bit overfit", it is a model choosing at random among 634 tokens for every word it writes. Training ends at $e^{0.391} = 1.5$. That gap is the whole diagnosis, and the conversions are mine.

The evidence is better than the curve. Drag the playhead across the epochs and read the sample printed at each stop, with the copied part highlighted:

It prints four samples, one per epoch, because generate_and_print_sample sits outside the batch loop: a row of commas, then the word "and" twenty-four times, then the phrase it tells you to search the file for, and then this:

Every effort moves you know," was one of the axioms he laid down across the Sevres and silver of an exquisitely appointed luncheon-table, when, on a later day, I had again run over from Monte Carlo; and Mrs. Gis

Searched against the real file, 184 consecutive characters of that are Wharton's, word for word. The epoch-9 sample copies 114. Both are measured by the figure, in your browser, and they are why the perplexity conversion is not the interesting part.

The arithmetic that makes it inevitable is one the book does not do. The model chapter 5 trains holds 163,009,536 parameters, not the 124 million on the label, and section 06 explains the gap. It sees 4,608 tokens an epoch: 35,375 parameters per token of training data.

Which brings up a correction of Giles Thomas's, and it is the hinge of the rest. The book presents sampling and top-k as ways to reduce training data memorisation. His objection: memorisation is what the model does during training, and at this ratio it is unavoidable. What temperature and top-k address is parroting, repeating what was memorised because greedy decoding always takes the same path. Memorisation lives in the weights, parroting in the decoder, and the next two decoding sections touch only the decoder.

03 · One alignment out of thirty-two

Before that, a detour the first figure forced on me, and it is where the sliding window from post three sends its bill. Chapter 2's loader sets stride = max_length, so the training windows are ids[0:32], ids[32:64], ids[64:96] and so on, none overlapping. Under learned absolute positional embeddings that has a consequence nobody states. Drag the offset and watch how many of the thirty-two positions light up:

At stride 32 the answer is one. The character at file offset 37 is presented at position 5 and nowhere else, ever, and the positional table has one row per position. Free-running generation slides its window one character per step, so after the prompt it asks about alignments that were never trained.

Two numbers say what that costs and they disagree. Teacher-forced argmax accuracy asks whether the model names the true next character given the true prefix: memorisation as a property of the weights. Longest verbatim span asks what free-running greedy decoding produces. At stride 32 my run passes the first and fails the second:

$ PART08_SECONDS=60 npx tsx scripts/llm/part08-verify.mts
stride  1: 2300 windows, a mid-file character is seen at 32.00 of 32 positions
stride 32: 72 windows, a mid-file character is seen at 1.00 of 32 positions
← the shipping run's own table, elided

== the same model at stride 32, the book's own loader setting ==
final train 0.0440 / val 7.9260, teacher-forced 0.978, best span 10
sample: "I HAD always thought Jack Gisburn vioreck id ite nouge iioug s. iiseptthepptpp macntis. ntag \", \"aloornrenred s. \", ithedet vit iionoug s thepas pat n cate te teMat t \"aghed e at a"
← 97.8% right, and it cannot recite ten characters

The weights memorised. The decoder cannot reach it. Setting stride = 1 with the book's own seeded shuffle fixes the alignment and costs memorisation capacity: 2,300 windows instead of 72, so on the same budget the accuracy is lower and the recital far better. Neither knob alone produces a long verbatim span; both together do, which is why the first figure ships at stride 1 and why its stride control is worth flipping. The book gets away with stride = max_length because 163 million parameters memorise every alignment implicitly. 28,064 cannot.

04 · Temperature is a reshape, not a dice roll

Greedy decoding is argmax, which is why a memorising model always emits the same passage. The first change is torch.multinomial: treat the softmax output as what it is, a distribution, and draw from it. The second is temperature,

$p_i(T) = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}$

applied to the logits, before the softmax. The common mental model, flatten the bar chart, describes the effect but not the operation, and the difference matters: dividing the probabilities by $T$ and renormalising does nothing at all, because the $T$ cancels top and bottom. The division has to land before the exponential or it has no effect.

The book's nine-word toy vocabulary is the right object to hold this on, because the whole distribution fits on screen: closer, every, effort, forward, inches, moves, pizza, toward, you, with logits [4.51, 0.89, -1.90, 6.75, 1.63, -1.62, -1.89, 6.28, 1.79]. Take the temperature down to 0.1 and back up to 5, then turn the draws up:

At $T = 1$, forward has probability 0.5721 and the book's thousand draws returned it 582 times. One of those is the model, the other is a sample of it. At $T = 0.1$ the distribution collapses onto forward at 0.9910, and at $T = 5$ it spreads until pizza reaches 0.0430, which is where the sentence every effort moves you pizza comes from.

The book's printed histogram carries a lesson about histograms in general: it has eight lines for a nine-word vocabulary. torch.bincount sizes its output to max(sample) + 1, and you at id 8 was never drawn, so it has no row. pizza reads 0 x pizza even though its probability at $T = 1$ is about $10^{-4}$. A zero in a histogram is a statement about your sample size, not about the model.

05 · Top-k, and a minus infinity you have met before

Temperature buys diversity and pays for it in nonsense. Top-k removes the nonsense and keeps the diversity: take the $k$ largest logits, set everything below the $k$-th to $-\infty$, then softmax. Since $e^{-\infty} = 0$, those tokens get probability exactly zero and the rest renormalise over a smaller denominator.

One number makes the mechanism legible. At $k = 3$ the survivors are [0.0615, 0.5775, 0.3610] on closer, forward, toward. forward was 0.5721 before the mask and is 0.5775 after it: throwing six words away made the winner more likely. Top-k renormalises rather than deletes. And this is chapter 3's causal-mask trick, for the same reason: masking with zero fails, because softmax turns a zero input into a positive output.

Listing 5.4's decoding tail carries three of this section's claims:

if top_k is not None:
    top_logits, _ = torch.topk(logits, top_k)
    min_val = top_logits[:, -1]
    logits = torch.where(logits < min_val,
                         torch.tensor(float('-inf')).to(logits.device),
                         logits)
if temperature > 0.0:
    logits = logits / temperature
    probs = torch.softmax(logits, dim=-1)
    idx_next = torch.multinomial(probs, num_samples=1)
else:
    idx_next = torch.argmax(logits, dim=-1, keepdim=True)

Pull $k$ down to 1 and watch the row sum stay at 1.0000 the whole way:

The mask goes on first and the divide second, and it happens not to matter, for two reasons rather than one. Dividing by a positive $T$ preserves the ordering of the finite logits, so top-k selects the same three either way; and $-\infty / T$ is still $-\infty$, so the sentinels survive. Get the order backwards in your own code and you will be fine. Get it backwards with a large finite sentinel instead of -inf and you will not.

Then the default nobody expects. generate has temperature=0.0, so calling it with no decoding arguments takes the else branch and is exactly generate_text_simple. Appendix C gives two ways to force that determinism, and top_k=1 is the other one.

06 · Someone else's weights

Section 5.4 is state_dict and torch.save, plus one sentence worth keeping: AdamW carries per-parameter historical moments, and resuming without them means "the optimizer resets, and the model may learn suboptimally or even fail to converge properly". Section 5.5 is the payoff. Three config edits ready the book's GPTModel for OpenAI's checkpoint: context_length back to 1024, qkv_bias to True, and the three size fields. That is the entire surface. The rest is a hand-written name mapping, which its author says "took a lot of guesswork since OpenAI used a slightly different naming convention from ours."

Pair the OpenAI names with the attributes they land on, and watch the parameter counter:

Sixteen distinct OpenAI parameter paths fill twenty-one GPTModel attributes through twenty-one assign calls, and the gap takes two sentences. params["wte"] is read twice, into the token embedding and into the output head, which is OpenAI's weight tying. And c_attn is one fused (768, 2304) tensor that np.split divides three ways along the last axis into $W_q$, $W_k$ and $W_v$, its bias twin doing the same. Every weight gets .T and no bias does, because TensorFlow stores (in, out) and PyTorch's nn.Linear stores (out, in).

That double assignment is where the parameter count comes from. OpenAI's checkpoint holds 124,439,808 distinct numbers. The book puts those values in two separate tensors, so chapter 4 builds 163,009,536 and section 5.5 loads 163,037,184, the difference being the 27,648 query, key and value biases qkv_bias: True turns on. Four numbers, one architecture, and any figure printing one owes you the convention.

The strongest sentence in the chapter is its last: if the mapping is wrong you will know, because the model will not produce coherent text. Coherence is the unit test for four chapters of work.

07 · Fourteen runs and a noise floor

weight_decay=0.1 runs unchanged through chapters 5, 6 and 7. The learning rate does not: 0.0004 in chapter 5, 5e-4 in section 5.4's own checkpoint reload, 5e-5 in chapter 6, 0.00005 in chapter 7. None of the four is justified anywhere.

Giles Thomas carried chapter 5's pair out of the toy run, in his words they "were just copied from the tiny training run that we do in section 5.2 of the book", into a 3,260,252,160-token single-epoch run on 8 by A100 40 GiB at global batch 96, about three and a half hours and US$35 a go in February rising past US$50 by April. Then he changed one thing at a time, between 4 February and 21 April 2026. Baseline test-set loss 3.691526; OpenAI's GPT-2 small, 3.499677.

Set the noise band to the training-run measurement and watch how many stop being results:

The ranking is not what the folklore predicts. Scheduling the learning rate was worth 0.089609, the biggest single win, and it is the schedule rather than the number: raising the peak to 0.0014 without warmup and cosine decay diverged, average training loss NaN by global step 1,851 and everything NaN by 2,468. Removing dropout was worth 0.050244 and 7.6% more throughput as well, 266,284 to 286,589 tokens a second, the rare intervention with no trade-off. Weight decay 0.1 down to 0.01 was worth 0.048586, and the 0.1 is GPT-3's number rather than a law.

Weight tying cost 0.182779, the worst result of the series, with a caveat. The tied run started at a loss of about 460 instead of about 11, because nn.Embedding initialises from $\mathcal{N}(0,1)$ while nn.Linear initialises from $\mathcal{U}(\pm\sqrt{1/768})$, roughly 28 times narrower. He chose not to fix it, so the honest reading is that naive one-line weight tying hurt, not that weight tying hurt.

Two findings matter more than the table. Several of these sit near the noise floor, and he measured the floor rather than guessing, re-seeding after model construction so that weight-initialisation randomness separated from training-loop randomness. Which effects survive depends on that choice, on which standard deviation you read, and on where the band is centred; the figure exposes all three rather than picking for you. And the sting: batch size, never on the intervention list, was worth 0.252474 on its own, more than double his best deliberate stack of 0.113765, from moving to a cloud box that happened to fit a batch of 96.

Gradient clipping is the one you can watch here. Turn clip grad norm off in the first figure and the curve barely moves: at twenty seconds my clipped and unclipped runs finished 0.006 nats apart. Turn it on and the readout says it fires on most steps at a threshold of 1.0, which is what he found when he measured instead of taking the tutorial default, and why he raised his to 3.5 and ran a no-clipping control to prove it was inert inside the limit. Clipping is not there to improve the loss. It is there so that one cliff does not end the run.

08 · What's next

What is missing, in rough order of usefulness. A learning-rate schedule, the largest measured win above, which the book leaves in appendix D. Gradient clipping in the book's own code, at a hundredth of the cost. More data, because 4,608 tokens an epoch causes everything in section 02. And top-p sampling, the sibling of top-k that appendix B names and does not cover.

The next post keeps the same weights and stops asking them for the next token. Remove the 768 to 50,257 output head, bolt on a 768 to 2 one, and a next-token predictor becomes a spam classifier: the objective changes, the architecture barely does.

The figures run on src/lib/minigpt, dependency-free TypeScript, seeded, on the same stable softmax and cross entropy the neural-network posts used. The transformer in the first one is 28,064 parameters, which its own counter prints, forward and backward and AdamW written by hand. Checking it took longer than writing it. Central differences against every parameter tensor appeared to fail at 8e-5 on the query and key projections, and that was not a derivation error but the finite-difference noise floor at entries whose true gradient is around $10^{-8}$: widening the stencil dropped the residual to 3.6e-7 at the same point, and a wrong backward pass does not improve when only the numeric side gets more accurate.