Fine-tuning is one new layer and 5.7% of the weights

Last time we watched a model memorise. Chapter 5 pushes its training loss down to 0.391 while the validation loss sits at 6.452, and the character-level transformer I trained in the browser alongside it did the same thing faster: 2,592 characters of "The Verdict", train 0.461 against validation 3.430 in forty seconds, and then it recites 41 characters of the story back at you verbatim. A model that has only been pretrained is a next-token predictor, and that is all it is.

Chapter 6 opens by checking that OpenAI's weights loaded correctly, feeding the model Every effort moves you and getting back forward. The first step is to understand the importance of your work. Then it asks the same model a question: is this text message spam. The model does not answer. It repeats the question back, word for word, and starts the message again. That failure is the reason the chapter exists. Following an instruction is a separate skill that has to be trained in separately, and chapter 6 takes the other route, which is to stop asking and rewire the model so the question cannot come up.

What is live here and what is not. Six figures follow, and not one of them computes everything it shows, so each carries a line saying which of its numbers came from where. All the arithmetic is real and runs in your browser: the split and batch counts, the padding, the parameter budgets, the softmax, the cross entropy, the confusion matrix. The tokenizer is the real GPT-2 byte-level BPE, 1.5 MB of tables fetched from the post that built it. Three numbers the chapter never prints I produced myself, by downloading the dataset and replaying the chapter's own two listings against it. But nothing here runs GPT-2, because 124 million parameters is 250 MB at half precision, so every accuracy, loss and timing you read is transcribed, either from the chapter's printed output or from the ablation tables in Raschka's repository, and no figure claims otherwise.

01 · Ask it nicely and it repeats the question

There are two ways to fine-tune, and the chapter's sidebar states the trade plainly. Instruction fine-tuning is more versatile and demands larger datasets and more compute. Classification fine-tuning needs less of both, and its use is confined to the classes it was trained on. Chapter 6 is the second one. Chapter 7 is the first.

The argument for doing it the narrow way is not theoretical. Here is what the base model does when you ask it in English, transcribed from the chapter's own printed output at max_new_tokens=23:

Is the following text 'spam'? Answer with 'yes' or 'no': 'You are a winner
you have been specially selected to receive $1000 cash
or a $2000 award.'
The following text 'spam'? Answer with 'yes' or 'no': 'You are a winner

Giles Thomas ran the same prompt against the same model and got the same echo, then tried the harder thing: a transcript-style preamble, the trick that used to rescue pre-chat-template completion models. The 124M model answered with empty turns, Bot: then User: then Bot:. The base model is not failing because the instruction was phrased badly.

A classifier sidesteps all of it. The chapter's figure 6.3 makes the contrast in one picture: a classification-fine-tuned model needs no instruction alongside its input at all. You hand it the message. That is why it is cheap, and it is also why it can never tell you anything except which of two labels it picked.

02 · 1,494 messages

The dataset is the UCI SMS Spam Collection, 5,572 rows, and it is 4,825 ham against 747 spam. The chapter throws away almost all of the ham: it samples 747 ham messages at random_state=123, concatenates them with the spam, and works with 1,494 messages from then on. Its stated reason is "for simplicity, and because we prefer a small dataset".

Slide the dataset back to its natural shape and watch what a classifier that reads nothing is worth:

That is what the balancing buys. On the raw distribution, answering ham every time scores 4,825 of 5,572, which is 86.59%, within nine points of the number the whole chapter is built to produce and requiring no model at all. Balanced, the same do-nothing classifier scores exactly 50%, so 95.67% is 45.67 points of work rather than nine. It cost 4,078 real messages to make the metric legible, and Giles Thomas makes the same point from the other end, with a two-line def is_spam(text): return False.

The splits fall out of int(), not rounding. With $N = 1494$:

$\text{train\_end} = \lfloor 1494 \times 0.7 \rfloor = 1045, \qquad \text{validation\_end} = 1045 + \lfloor 1494 \times 0.1 \rfloor = 1194$

which is 1,045 rows for training, 149 for validation, 300 for test. The chapter never prints those three. It prints the batch counts they produce, 130 and 19 and 38, and $\lfloor 1045/8 \rfloor = 130$ only comes out right because the training loader sets drop_last=True. So five training messages are discarded permanently, and the model is trained on 1,040 rows. The reason for drop_last is one the book does not give: in a ragged final batch of five, each message contributes a fifth of that gradient update instead of an eighth, and the last five rows of the file get an outsized vote.

03 · Everything is 120 tokens long

Text messages are not all the same length, which every chunk of pretraining data was. You can truncate to the shortest or pad to the longest, and the chapter pads, using <|endoftext|> (id 50256) as the pad token. max_length is set by the longest training message, and it comes out at 120 tokens.

I checked that with this repo's own byte-level BPE rather than taking it on trust, and it reproduces exactly: 120. What is more interesting is what sets it. Exactly one training message is 120 tokens long, a garbled chain letter about a girl named Margaret Hello, and every other message in the corpus is padded to fit it. The median is 32 tokens. Across a training batch, 74.66% of the positions hold a pad token.

Type a message, drag the length, and watch where the read-out marker lands:

The marker sits on a pad token for every message shorter than 120 tokens, which is almost all of them. This is the chapter's largest silence: figures 6.11 and 6.12 both use unpadded four-token examples, so a reader never sees that the slice at -1 normally reads a hidden state whose input token is <|endoftext|>. It works anyway, and the reason is the only genuinely interesting idea in the chapter.

Exercise 6.1 invites you to pad to the model's full 1,024 instead, and the count it is pointing at is real: $T(T+1)/2$ unmasked cells goes from 7,260 to 524,800, which is 72.3 times as many. It does not mean 72.3 times the work. At $d = 768$ the per-token linear cost of a block is $12d^2 = 7{,}077{,}888$ multiply-accumulates against $2Td$ for the scores, so attention is 2.5% of the forward pass at 120 tokens and 18.2% at 1,024. Total compute goes up 10.2 times, and Raschka's measured wall clock goes up 8.8. The rest of the model is linear in length, and that is where the time is. Accuracy goes the wrong way too: rows 1 and 13 of the ablation table read 95.00% and 78.33%.

One small thing while we are here. The chapter warns that validation and test messages longer than max_length are silently cut. On this split none are: the longest validation message is 71 tokens and the longest test message is 92.

04 · The row that has seen everything

The model still emits one 2-vector per input token, so a four-token input produces a [1, 4, 2] tensor and the code slices [:, -1, :] to get [1, 2]. The question is why that slice, when the position it lands on is usually a pad token.

The causal mask is the answer, written once and 0-indexed:

$\text{Attn}(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}} + M\right)V, \qquad M_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases}$

Position $t$ has a receptive field of exactly $t+1$ tokens, so position $T-1$ is the only one whose receptive field is the whole sequence for every $T$. Not the only one that has read the message, note. Under right-padding, every position from the last real token onwards has all of it. The last position is unique in being the index you can write down without knowing the length.

Click a row to choose which position the classifier is trained on and read from, and count what that position has seen:

Position 0 attends to itself and nothing else. Its hidden state is a function of one token, so asking it to classify a message is asking it to classify the first word. Raschka trained and read there instead, and row 2 of his ablation table scores 75.00% against row 1's 95.00%. One index, twenty points. The honest deflation is that 75.00% is not chance either, which says the first word of a text message carries real signal, and says nothing good about accuracy as a metric. And turning the causal mask off entirely, which is row 17 of the same ablation table, scores 95.33% against 95.00%, a difference of one test message out of three hundred: the mask is not what makes the model good, it is what makes the last position addressable. Bidirectional encoders do not have the problem at all: with no mask, every position sees everything, which is why BERT can put a classification token at the front and read that one.

05 · One line of surgery

The architectural change is one assignment, and the code around it is six lines of bookkeeping:

for param in model.parameters():
    param.requires_grad = False

model.out_head = torch.nn.Linear(in_features=768, out_features=2)

for param in model.trf_blocks[-1].parameters():
    param.requires_grad = True
for param in model.final_norm.parameters():
    param.requires_grad = True

The order matters. Freezing happens before the head is replaced, and the optimizer is constructed after both, so the new head is included. requires_grad=False also freezes updates rather than stochastic forward-pass behaviour, which is Giles Thomas's catch: leaving dropout on in a frozen block would inject noise that nothing can learn around, and it is why the chapter loads the model with drop_rate at 0.0.

Pick what to unfreeze and watch the budget, then check it against what Raschka measured:

The head goes from Linear(768, 50257) at 38,597,376 parameters to Linear(768, 2) at 1,538, a shrink of 25,095.8 times. What is left trainable is one transformer block at 7,087,872, the final LayerNorm at 1,536 and that head, so 7,090,946 of 124,441,346, which is 5.70%. The model here carries 163,037,184 parameters before the swap where the architecture post built 163,009,536, and the 27,648 difference is the query, key and value biases: chapter 4's config turns them off and OpenAI's checkpoint has them on.

Then the ablation, which is where the chapter's own exercise 6.2 lands. Unfreezing the last two blocks reaches 98.33% in half the time that unfreezing all twelve takes to reach 96.67%. On 300 messages those are 295 and 290 correct, and the binomial standard deviation of each is about 1.1 points, so I am not going to call a five-message gap a finding. What is not noise is the clock: 0.33 minutes against 0.69. Giles Thomas ran both configurations on an RTX 3090 and got the opposite ordering, 95.67% frozen in 15 seconds against 97.67% unfrozen in 42. Two runs, two orderings, one conclusion: at this sample size the freeze budget buys time, not accuracy. The printed appendix E takes the idea further and trains 2,666,528 LoRA parameters at the same lr=5e-5, for 98.00%.

06 · The number you optimize

Accuracy cannot be a training objective. It counts matches, so its derivative is zero almost everywhere and there is nothing for gradient descent to descend. Cross entropy is the differentiable stand-in, and on a batch of logits with integer targets it is

$\mathcal{L}(z, y) = -\log \frac{e^{z_y}}{\sum_j e^{z_j}} = -z_y + \log \sum_j e^{z_j}, \qquad \arg\max_i \operatorname{softmax}(z)_i = \arg\max_i z_i$

The second identity is why the chapter says softmax is optional here. It is strictly increasing, so it cannot reorder anything.

Drag the two logits the chapter actually printed, then switch to a whole freshly initialized head:

The check next to argmax z = argmax softmax(z) never goes out, wherever you drag. The second mode answers the number that confuses everyone in this chapter: before the first optimizer step the model sits at 46.25% accuracy, which is near chance, and 2.453 loss, which is three and a half times chance. Those are not in conflict. Argmax reads the sign of the logit gap and cross entropy reads its magnitude, so a confidently wrong classifier is cheap by one measure and expensive by the other. Scale a fixed set of logit gaps and accuracy does not move at all, because scaling never flips a sign, while the loss climbs without a ceiling. At a spread of 4.16 it passes 2.453. That is a dial and not a model of what PyTorch's initializer does, but it is the right shape.

07 · Five epochs, and thirteen mistakes

The training loop is the pretraining loop with one index changed. calc_loss_batch differs in exactly one line:

logits = model(input_batch)          # chapter 5: every position
logits = model(input_batch)[:, -1, :]  # chapter 6: the last one

and train_classifier_simple differs from train_model_simple in exactly two ways: it counts examples instead of tokens, and it measures accuracy instead of printing sample text. evaluate_model and calc_loss_loader are untouched. AdamW at lr=5e-5 and weight_decay=0.1, five epochs over 130 batches, is 650 optimizer steps and 5.65 minutes on an M3 MacBook Air. Loss falls from 2.153 at the first logged step to 0.083 at step 600. The thirteen log lines are spaced 3, 3, 2, 3, 2 across the five epochs, which looks like a bug and is arithmetic: eval_freq is 50 and 130 is not a multiple of it.

Two places the log misleads. Its Training accuracy: 100.00% is measured on five batches, forty examples, and the real training-set number is 97.21%, which is 1,011 of 1,040. And validation at 97.32% coming out above test at 95.67% needs no mechanism: at 149 and 300 examples the standard deviation of the difference is 1.97 points, so a 1.65-point gap is 0.84 of one.

Which leaves the number the chapter closes on. 95.67% of 300 is 287 correct, and the chapter never says which thirteen it got wrong, which for a spam filter is the entire product question. The test set's class balance is not printed either, so I ran listings 6.2 and 6.3 against the real archive: it is 151 spam and 149 real messages, not the round 150 you would assume.

Choose how those thirteen errors fall, then say what a lost message is worth to you:

Every arrangement on that slider is consistent with the published 95.67%, and they describe two products you would make opposite decisions about. Once you put a price on the two error types the choice stops being a matter of taste and becomes arithmetic, and the bar chart tilts the moment the price does. Accuracy sits still through all of it, which is the argument for never shipping it alone.

08 · What's next

What I would add, roughly by usefulness. Read the last non-padding position rather than the last position: it is row 16 of Raschka's own table, 98.33% at the same speed, and the chapter's silence about pad tokens is exactly the gap it closes. Report precision and recall rather than accuracy, with the cost ratio written down somewhere a reviewer can argue with. And check the honest baseline before deciding any of this was worth it, because on the IMDb sentiment set a scikit-learn logistic regression scores 88.85% against a fully fine-tuned GPT-2's 91.88%, and a 395M ModernBERT beats both at 95.07%.

The other branch of stage 3 is instruction fine-tuning, where the model has to answer in words again. That removes the labelled output space this whole post depends on, and with it the ability to write down a number like 95.67% at all. The evaluation problem gets genuinely hard, and it is the next post.

The figures run on src/lib/minigpt/, dependency-free TypeScript, seeded, sitting on the same matrix and softmax code the neural posts used. The hardest part was not the arithmetic. It was getting the padding figure to disagree with itself honestly: a read-out marker sitting on an <|endoftext|> token looks like a bug in the figure until you understand it is a fact about the model.