Two appendices, and the seven things nobody does any more

Last time the model stopped predicting text and started answering. Eleven hundred hand-written instruction pairs split 935 / 55 / 110, two epochs on GPT-2 medium, and a printed average of 50.32 from a Llama 3 8B judge over the 110 held-out instructions. Two reruns of the same recipe scored 49.45 and 51.75. Two points of spread on an unchanged recipe is most of what a judge score is worth, and that was the last honest thing this series had to say about evaluation.

Two appendices are left. Appendix D inserts one line into the five-line training loop, and in the only measured comparison here it is the largest single training win of all. Appendix E is one ten-line class making 124,441,346 frozen weights adaptable through 2,666,528 new ones, and the line doing the work initialises a matrix to zero. Then the part a book printed in 2024 cannot do: what a 2026 model does differently, one slot at a time.

Everything here computes in your browser. No recordings, no model weights, here or anywhere in the series. Four of the six figures compute their results rather than quoting them, and two of those reproduce counts the book prints: 124,441,346 trainable before freezing and 2,666,528 of adapter at rank 16. A third, 406,286,336, is the companion repository's rather than the book's, and the figure says which. The fifth puts a live computation beside six accuracies transcribed from appendix E. The sixth is a table of fourteen transcribed test losses with two bands computed on top, and its footer says so.

01 · One line, inserted before the step

The loop the training post built is five lines: forward, loss, zero_grad(), backward(), step(). Appendix D does not restructure it. It assigns the optimiser a new learning rate immediately before step(), on every iteration, and that assignment is the whole of the schedule.

Warmup comes first. The book's reasoning: starting with smaller weight updates decreases the risk of large destabilising ones. The mechanism is a straight line from initial_lr up to peak_lr, and the rule of thumb is 0.1% to 20% of total steps. It computes int(0.2 * total_steps), prints 27, and then describes the result in prose as 20 warmup steps, which is the hardcoded value it overwrote one line earlier. Both figure captions say 20. Run the code and you get a 27-step ramp.

After the ramp, a half cosine down toward min_lr:

$\text{lr}(t) = \text{initial\_lr} + t\cdot\frac{\text{peak\_lr} - \text{initial\_lr}}{\text{warmup\_steps}}, \qquad t < \text{warmup\_steps}$

$p(t) = \frac{t - \text{warmup\_steps}}{\text{total\_steps} - \text{warmup\_steps}}, \qquad \text{lr}(t) = \text{min\_lr} + (\text{peak\_lr} - \text{min\_lr})\cdot\tfrac{1}{2}\big(1 + \cos(\pi\, p(t))\big)$

Read the second one back in English. At progress 0 the cosine is 1, the bracket is 1, and the rate is exactly peak_lr. At progress 1 the cosine is minus 1, the bracket is 0, and the rate is exactly min_lr. Both identities hold, and the second one never happens. global_step starts at minus 1, is incremented at the top of the body, and stops at total_steps - 1, so on the book's 135-step run the largest progress the cosine sees is 107/108. The last executed rate sits about 2.1 × 10⁻⁷ above min_lr, a number the book does not print and I computed here.

Set warmup to zero and watch the first step, then put it past a fifth of the run and watch the other end:

The third preset carries a bug worth seeing. The listing assigns peak_lr = 5e-4 at the call site, then builds AdamW with no lr argument, so train_model reads the rate back out of optimizer.param_groups and gets AdamW's default of 1e-3. The printed run peaks at twice the rate the code appears to ask for. And it ends at train loss 0.041 against validation loss 6.915, which is not a schedule working. It is a 20,479-character short story being memorised.

02 · What it was worth

Giles Thomas spent February of this year on gradient clipping and March on learning-rate schedules, both as side quests, then read the appendices in April and found both sitting in appendix D. His response is the only honest answer to "should I have just read it first" I have seen anybody give: grinding through it from first principles meant he internalised it better, and reading explanations is faster but shallower.

Then he measured. Fourteen test losses from GPT-2-small-shaped models at 3,260,252,160 tokens each, and not fourteen interventions: eight are single changes against a fixed baseline of 3.691526, two are that recipe on two machines, and four are stacked combinations. Of the eight singles, learning-rate scheduling is the largest win at 0.089609 and gradient clipping is 0.013209, about a seventh of it. Two came out negative: weight tying at −0.182779 and Cerebras-style weight decay at −0.122330. Removing mixed precision was a small positive at 0.012558 and he kept mixed precision anyway, because it was twice as fast and 66% cheaper in the cloud. That is the one row where the loss number is not the decision.

Turn the noise band on and find the two rows that point the wrong way:

One correction, gently. He measured run-to-run spread by pinning the weights and varying only the training seed, getting 3.691526, 3.681356 and 3.680505, and reported a standard deviation of 0.008672. Those three have a mean of 3.684462 and a sum of squared deviations of 7.5205 × 10⁻⁵; the square root of that sum is 0.008672 and the population standard deviation is 0.005007. His figure is the square root of the undivided sum. Every conclusion he drew survives either way, and the smaller band makes the interventions look more real rather than less.

And the number that is not on the list at all: raising the batch size from 6 to 96 was worth 0.252474, more than double every intervention stacked together. The appendix everybody skips holds the biggest single training intervention anyone here measured and one of the smallest, and the biggest lever of all is not an intervention.

03 · A rank you choose

Regular fine-tuning is $W_{\text{updated}} = W + \Delta W$. LoRA writes the update as a product of two skinny matrices, $A$ of shape $(d_{in}, r)$ and $B$ of shape $(r, d_{out})$, and then never forms $\Delta W$ at all:

$W_{\text{updated}} = W + AB, \qquad \operatorname{rank}(AB) \le r \le \min(d_{in}, d_{out})$

$x(W + AB) = xW + xAB$

The distributive law is what makes it practical: the adapter stays a separate object at runtime, so you ship one frozen base model and many small adapters instead of a full copy per customer. A 7B checkpoint is 23 GB; an adapter for it at rank 8 can be 8 MB.

Now the phrase "low-rank", precisely, because this is where every explainer gets loose. Every column of $xAB$ lies in the column space of $A$, which has dimension at most $r$, and every row lies in the row space of $B$, same bound. So the update's rank is capped at $r$ against a possible $\min(d_{in}, d_{out})$. LoRA does not restrict the update to a subspace of the space of matrices, because the rank-$r$-or-less matrices are not a subspace at all. It restricts the rank. And Raschka's own correction, worth having in his words: LoRA does not decompose the matrix directly, it learns the decomposed matrices by backpropagation.

The price is arithmetic you can do in your head. A layer costs $d_{in} \cdot d_{out}$ to fine-tune fully and $r(d_{in} + d_{out})$ through an adapter, so at 768 by 768 with $r = 16$ that is 24,576 against 589,824, or 4.17%.

Put the rank at 16 and check the total against the number the book prints, then drag it and watch what a rank buys:

The second panel closes off a wrong idea, and it does so in the awkward direction. Run plain gradient descent on $\|AB - \Delta W\|_F^2$ to convergence and it lands on the truncated-SVD optimum at every rank, to six decimal places, because every local minimum of that objective is global. There is no gap to admire. Which is exactly why the interesting claim is that LoRA is not solving that objective. It never sees $\Delta W$. It sees a loss on a task, and $A$ and $B$ are whatever backpropagation makes them.

One more thing the ledger shows. The book adapts every nn.Linear, output head included, for 2,666,528 trainable parameters against 124,441,346 frozen ones, a factor of 46.67. The LoRA paper's headline configuration adapts the query and value projections only. Two different methods, one name.

04 · Why B starts at zero

The whole of appendix E is one class, and its two load-bearing lines are invisible in every diagram of LoRA ever drawn:

class LoRALayer(torch.nn.Module):
    def __init__(self, in_dim, out_dim, rank, alpha):
        super().__init__()
        self.A = torch.nn.Parameter(torch.empty(in_dim, rank))
        torch.nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.B = torch.nn.Parameter(torch.zeros(rank, out_dim))
        self.alpha = alpha

    def forward(self, x):
        x = self.alpha * (x @ self.A @ self.B)
        return x

A gets the same initialisation PyTorch gives any nn.Linear. B gets torch.zeros. So $AB$ is the zero matrix, the adapter returns zero for every input, and linear(x) + lora(x) equals linear(x). Not approximately: the floating-point result is bit-identical, because adding zero is adding zero.

The book prints the proof without labelling it as one. It measures the classifier on 80 examples per split before inserting anything and gets 46.25% / 45.00% / 48.75%. Then it freezes every parameter, inserts 2,666,528 new ones, and measures again: 46.25% / 45.00% / 48.75%, unchanged to the second decimal.

There is a consequence neither source states. Whatever the downstream loss is, the chain rule puts a factor of $B^\top$ in the gradient with respect to $A$, so at step 0 that gradient is the zero matrix while $B$'s is not. The first step moves only $B$. $A$ starts moving on the second.

Switch B away from zeros and watch the verdict line go red before a single gradient has been computed:

With B random the figure will not tell you what the three accuracies would have been, because the book does not run that configuration and inventing numbers to fill a gap would be worse than a blank. What it does report is the outcome: 5 epochs of AdamW(lr=5e-5, weight_decay=0.1), a measured 12.10 minutes, ending at 100.00% train and 96.64% validation. The printed code output says 98.00% test and the prose two paragraphs later says 97.33%.

Then the trap, which is about two conventions and not three. The printed appendix multiplies by $\alpha$. The companion repository's current notebook multiplies by $\alpha/r$, and the comment says why:

    def forward(self, x):
        # Note: The original chapter didn't include the scaling by self.rank
        # This scaling is not necessary, but it's more canonical and convenient
        # as this lets us compare runs across different ranks without retuning learning rates
        x = (self.alpha / self.rank) * (x @ self.A @ self.B)
        return x

That comment is in the notebook, not the book: the author correcting his own printed text. The printed appendix runs at lr = 5e-5, exactly the rate chapter 6 uses for a full fine-tune, and it works. The repository runs the identical code at 8e-4, sixteen times higher, and lands elsewhere: 99.81 / 97.99 / 96.67. A hyperparameter copied across the two means nothing.

05 · Seven swaps, and where each one goes

Raschka counts seven differences between the book's GPT-2 and gpt-oss, then spends a section arguing you should build the GPT-2 one first anyway, because you will understand each newer piece better once you know which limitation it solves. He built his own from-scratch Qwen3 starting from his GPT-2 code. Seven swaps, then, as of mid-2026, each a local substitution into a slot this series already built.

The normalisation slot. LayerNorm subtracts the mean and divides by the standard deviation, giving zero mean and unit variance. RMSNorm divides by the root mean square, which scales activations to a comparable magnitude without enforcing either; in Raschka's own worked figure the output has mean 0.77 and variance 0.41, and that is not a bug. It won on cost: no shift term, and one cross-feature reduction instead of two, which lowers communication overhead on a GPU.

The activation slot. GELU is $0.5x\left(1 + \operatorname{erf}(x/\sqrt{2})\right)$, and erf is usually a polynomial fit. Swish is $x \cdot \sigma(x)$, shipped as silu. His verdict on quality: depending on which paper you read either is slightly better, the difference is probably within a standard error, and Gemma still uses GELU. A sigmoid is cheaper than an error function. That is the reason.

The feed-forward slot. Gating is a separate axis from the activation, and it is where the expressivity argument lives. Two matrices become three and the hidden width shrinks so the count matches. Do that arithmetic properly, because the version in circulation does not: at the book's 768 and 3072 the matched width is exactly 2048, and $3 \times 768 \times 2048 = 4{,}718{,}592$ is exactly $2 \times 768 \times 3072$. The claim is expressivity at equal parameters, not fewer parameters.

Edit the vector and read the two output rows, because the second one is not trying to give you what the first one gives you:

LayerNorm reads 0.000 and 1.000 for any input with spread in it, and stops doing so when you flatten the six values, which is its own edge and not RMSNorm's failure. On the second tab, GELU and SiLU differ by at most 0.1930, at x = ±1.9655: a visible gap between two curves that produce models nobody can tell apart.

Three swaps left. The position slot: a learned 1024 by 768 table, 786,432 parameters, replaced by a rotation of the query and key vectors that depends on position and carries none at all. The attention slot: grouped-query attention has several query heads share one key and value projection, so at twelve heads and four groups K and V shrink from 768 by 768 to 768 by 256, which costs fewer parameters and, far more importantly, caches fewer bytes. Around the block: a mixture of experts replaces the one feed-forward with many and routes each token to the top few, sliding-window attention caps how far back some layers look, and dropout is deleted, because a model seeing each token once is not at risk of overfitting it.

Turn all seven on and watch the cache line, then put the context slider at 262,144:

At a 1,024-token context in fp16 the book's model needs 36 MiB of key-value cache. Four groups take that to 12 MiB, and a 128-token window on every second layer takes the original to 20.25 MiB. With 32 experts and 4 active, the expert weights are 94.95% of the parameters while 16.92% of them run on any token, which is the sparse argument in two readouts. One gap in that ledger, stated rather than hidden: real models of this era usually drop the attention biases too, and this one keeps them everywhere, because a switch that prices two changes prices neither.

What did not change is the more interesting list. The residual stream, the direction of the mask, the row softmax, cross-entropy against the next token. The window changes the mask's span, not its causality.

06 · What the swaps do not explain

Gemma 4 shipped in April of this year and is, in Raschka's words, architecturally pretty much unchanged from Gemma 3. It leapt on benchmarks anyway. His conclusion from watching Mistral 3 Large adopt DeepSeek V3's architecture wholesale is the same: why change what is not broken, and a lot of the secret sauce now lives in the training pipeline and the inference-scaling strategies. Read section 02's leaderboard the other way and it agrees: its largest lever was a batch size, which is not an architecture.

And then the one that should stick. Giles Thomas's best from-scratch model reached a test loss of 3.418784, beating OpenAI's own GPT-2 small weights at 3.499677 by 0.080893. He then instruction-tuned sixteen models and had them judged together with the response order shuffled per query. His model scored 19.25 and ranked fourth. GPT-2 small scored 26.73 and ranked second. Two models a twelfth of a nat apart on loss, and one is meaningfully better at the thing anybody actually wants. His summary: loss number goes down is an interesting technical game to play, and it does not cleanly map to real-world performance.

None of which makes the book late. Random forests were published in 2001 and went mainstream around 2013, a year after scikit-learn shipped them: adoption is gated by tooling rather than by papers. A book teaching a 2019 architecture in 2024 while the frontier sits in 2026 is the normal state of this field.

07 · What's next

Five things I would add, in rough order of how much they would repay the effort.

A KV cache, left out of the book by the author's own statement because it increases memory requirements, and also the thing that makes grouped-query attention pay. RoPE, properly, which Raschka calls a tricky topic to explain and defers; that gap is still open. Preference tuning, because the book stops at supervised instruction fine-tuning and a preference stage is standard now. Reasoning, and reinforcement learning with verifiable rewards, where the arrow after chapter 7 points. And one modern model built from this code: the companion repository has GPT-2 to Llama 2, Llama 2 to Llama 3, Qwen3 and Olmo 3, each starting from the GPT-2 implementation. That is section 05, executed.

If you arrived at the end first, the first post is where this starts.

src/lib/minigpt/ is now 7,429 lines of dependency-free, seeded TypeScript sitting on top of the Mat, softmaxStable and gelu the neural posts left behind. The thing that took longest was reproducing 124,441,346 exactly. Three details no diagram shows have to be right at once: appendix E's config turns the QKV biases on where the from-scratch chapter turns them off, the replacement two-class head is an nn.Linear and so carries a bias worth two parameters, and the count is taken after the head swap. Get any one wrong and you land within a few thousand of the answer, which is the worst possible place to be.