Next-word prediction is the whole objective

There are five figures on this page and none of them is a recording. One runs on nothing but the text you type into it. One replays real GPT-2 output measured offline, and says so. The other three put a live computation beside a number or a table I copied out of the book by hand, and each of those three says in its own footer which half is which. Every figure in this series carries that footer and it always begins with one of four words: live., precomputed., recording. or static.

What this page does not have is a model running in it. GPT-2's weights are 250 MB at half precision, which is two orders of magnitude past anything this blog ships to a reader.

01 · The objective is one guess

Chapter 1 defines a parameter before it defines a transformer, and the definition smuggles in the entire book. Parameters are "the adjustable weights in the network that are optimized during training to predict the next word in a sequence." Not to translate. Not to classify. To predict the next word.

The last series ended on softmax over two classes and said the next thing was softmax over a vocabulary. This is that vocabulary. A language model reads some text, emits one score per token it knows, softmaxes those scores into a distribution, and is graded on how much probability it put on the token that actually came next.

Play it against a real one before reading further. Pick what you think came next, then the bars are revealed:

The grading unit is bits, and bits are unforgiving. Over the 48 shipped positions GPT-2 124M ranks the true token first 16 times, exactly a third, and averages 4.9073 bits of surprise per token. Ten of those 48 positions have the true token outside the model's own top sixteen, which is why the grid has a seventeenth cell rather than a tidy sixteen that always contains the answer.

Averaging that surprise over a corpus is the loss, and it is one line:

$$\mathcal{L} = -\frac{1}{n}\sum_{i=1}^{n} \log P_\theta(t_i \mid t_{<i})$$

That is cross-entropy against a one-hot target, which is the function the neural series already wrote, and stable softmax is the layer that produces the distribution, which it also already wrote. Neither changes here. The only difference is that $k$ is now the vocabulary.

One attribution, and it holds for every formula on this page. Chapter 1 has no equations in it at all. These are the standard way of writing down what it says in English, and a reader who goes looking for them in the book will not find them.

02 · The label was already in the file

Readers arriving with a machine-learning background expect labels, because conventional supervised learning cannot start without them. The chapter's answer is one sentence in a NOTE box: "LLMs use self-supervised learning, where the model generates its own labels from the input data."

The mechanism is a slice, offset by one. Given raw text cut into a sequence of tokens, the input and the target for a context of length $c$ are:

$$x = (t_1, \dots, t_c), \qquad y = (t_2, \dots, t_{c+1})$$

Type a paragraph of your own into the box, then drag the context slider and watch the two counters:

The left counter moves with everything you touch. The right one is nailed to zero and stays there, because the label maker is the array index, not a person. Switch the target to every position and it gets worse for the intuition: a window of $c$ tokens is not one training example, it is $c$ of them, scored at once.

Here is the whole label pipeline. It is my code, not the book's, since chapter 1 has none:

export function makeWindows<T>(seq: readonly T[], contextLength: number, stride = 1): TextWindow<T>[] {
  const out: TextWindow<T>[] = [];
  for (let i = 0; i + contextLength < seq.length; i += stride) {
    out.push({
      start: i,
      input: seq.slice(i, i + contextLength),
      nextTarget: seq[i + contextLength]!,
      shiftedTargets: seq.slice(i + 1, i + contextLength + 1),
    });
  }
  return out;
}

The only interesting character is the + 1, and the only interesting absence is a labels argument. The bound is < rather than <= so the last target always exists; a window whose target would run off the end is dropped rather than padded, because padding invents a token the text never contained.

The words and characters toggle is the hole in all of this. How many examples a paragraph yields depends entirely on what counts as one piece, and that question is the next post's. Chapter 1 says "word" everywhere it means "token", which is harmless until chapter 2 arrives and a word becomes one to four predictions.

03 · Two removals, one chapter apart

Chapter 1's first figure nests four circles: AI contains machine learning contains deep learning contains large language models. A nesting diagram carries exactly one bit of information, which is containment order, and the part worth having is the criterion at each boundary. Machine learning learns from data instead of from rules. Deep learning is machine learning with three or more layers, which the book states as its own convention rather than as a law. An LLM is a deep model whose data is text.

The real axis under that figure is manual feature extraction, and the chapter is specific about what it costs. To classify spam the old way, a person decides in advance what is worth measuring: the frequency of trigger words such as "prize", "win" and "free", the number of exclamation marks, the use of all-uppercase words, and the presence of suspicious links. Every one of those is a human hour, and the ceiling of the model is the imagination of whoever wrote the list.

Deep learning removed that list. It did not remove anything else, and the chapter says so in a parenthesis most readers skim: both traditional machine learning and deep learning for spam classification still require the collection of labels, gathered by an expert or by users. That is the same column section 02 deleted, and it is a different column from the features.

So there are two removals, three pages apart, and merging them is the most common way to leave this chapter with the wrong model of it. Deep learning stopped you hand-picking features. Self-supervision stopped you hand-writing labels. Only the second one is what made training on a large portion of the public internet possible, and the spam example returns in chapter 6, where the book fine-tunes its own GPT-2 into a classifier on labels somebody did write.

04 · What 300 billion tokens were

With labels free, the limits are data and money. Chapter 1's only table is GPT-3's pretraining mixture.

Dataset	Description	Tokens	Proportion in training
CommonCrawl (filtered)	web crawl data	410 billion	60%
WebText2	web crawl data	19 billion	22%
Books1	internet-based book corpus	12 billion	8%
Books2	internet-based book corpus	55 billion	8%
Wikipedia	high-quality text	3 billion	3%

The trap is that those two numeric columns look like the same quantity twice. They are not. The token column adds to 499 billion, the model was trained on 300 billion, and the book's sidebar notes that the GPT-3 authors did not say why. The naive reading is that GPT-3 read 300 billion of the 499 billion once each. It did not read any of them once each.

Flip the figure between the two columns and watch the order change:

The proportion column is a sampling weight. Small high-quality sets get read several times over and the giant crawl gets read less than half a time. Doing the division the book prints both operands for and never performs:

$ node -e '
> const rows = [["CommonCrawl",410,60],["WebText2",19,22],["Books1",12,8],["Books2",55,8],["Wikipedia",3,3]];
> const B = 300, pad = (s,n) => String(s).padStart(n);
> let corpus = 0, used = 0, share = 0;
> console.log("dataset         corpus  share  consumed  epochs");
> for (const [name, tok, pct] of rows) {
>   const c = pct / 100 * B; corpus += tok; used += c; share += pct;
>   console.log(name.padEnd(14) + pad(tok+"B",7) + pad(pct+"%",7) + pad(c.toFixed(1)+"B",10) + pad((c/tok).toFixed(2),8));
> }
> console.log("total".padEnd(14) + pad(corpus+"B",7) + pad(share+"%",7) + pad(used.toFixed(1)+"B",10));
> '
dataset         corpus  share  consumed  epochs
CommonCrawl      410B    60%    180.0B    0.44← the biggest set, read less than half a time
WebText2          19B    22%     66.0B    3.47
Books1            12B     8%     24.0B    2.00
Books2            55B     8%     24.0B    0.44
Wikipedia          3B     3%      9.0B    3.00← 0.6% of the corpus, read three times over
total            499B   101%    303.0B← the sidebar says 100%, and the model saw 300B

The bottom row has two problems and only one of them is mine. The share column sums to 101, not 100, so a budget of 300 billion tokens draws 303 billion. That excess is inherited from the GPT-3 paper's own table, and the book's sidebar states that the column "sums up to 100% of the sampled data, adjusted for rounding errors." It does not. Flagging it beats fixing it silently, because a careful reader who adds the column will otherwise assume they misread.

The money is the other reason this table matters. Pretraining GPT-3 is estimated at $4.6 million in cloud computing credits, and that single number decides the shape of the whole book: implement everything, pretrain on a corpus small enough to run on a laptop, then load OpenAI's published weights into the architecture you built. Nobody is reproducing this table at home.

05 · Delete the encoder

The 2017 paper the whole field runs on was written for machine translation, English into German and French. It has two halves. The encoder turns the source sentence into vectors, the decoder turns those vectors into target-language text, and the chapter's own example has the source "This is an example", the words "Das ist ein" produced so far, and one word left to emit.

GPT is that architecture with a deletion. BERT keeps the encoder and is trained by masked word prediction; GPT keeps the decoder and is trained by next-token prediction. The famous one is the simpler one. Drawing them side by side invites the reading that BERT hands something to GPT, so it is worth saying outright: they are two model families derived from two submodules of a third architecture, and no BERT has ever passed anything to a GPT.

Switch between the three, and watch the grids underneath rather than the boxes:

The difference is not the code. It is which cells of an $n \times n$ grid are filled. "Bidirectional" and "unidirectional" stop being adjectives there and become a countable constraint: position four of seven sees seven cells in the encoder and four in the decoder, which is the entire reason BERT cannot write and GPT cannot look ahead.

The figure makes one more thing unavoidable. The 2017 decoder was already causally masked. Masking did not arrive with GPT. What arrived with GPT is that the masked stack is the only stack left.

Each block in that diagram has a dashed empty box in it, and the box is faithful. Chapter 1 draws the transformer three times and never once draws the mechanism it calls a key component: §1.4 names self-attention, parenthesises it away as "(not shown)", and defers it to chapter 3 on grounds of complexity. Two posts from now it is the only thing on the page.

The chapter also states its own scope, which is worth repeating rather than dropping. Not all transformers are LLMs, because transformers also do vision. Not all LLMs are transformers, because recurrent and convolutional ones exist. Raschka then says plainly that he uses "LLM" to mean a transformer-based one similar to GPT, which is a convention honestly declared rather than a definition quietly assumed.

06 · Ninety-six layers is the smaller half

Two numbers sit next to each other in §1.6: the original transformer repeated its blocks six times, and GPT-3 has 96 transformer layers and 175 billion parameters. Sixteen-fold, and it reads like the story.

It is the smaller half of the story, and the chapter gives you no way to see that, because it prints no width for either model. Parameters per block go with the square of the width:

$$N \approx 12\, n_{\text{layers}}\, d_{\text{model}}^2 \;+\; |V|\, d_{\text{model}} \;+\; L_{\text{ctx}}\, d_{\text{model}}$$

The $12d^2$ is $4d^2$ for the four attention projections plus $8d^2$ for a feed-forward with a $4d$ hidden layer. Start on the GPT-2 preset, then drag the two sliders one at a time:

Raise the depth from 12 layers to 96, eight-fold, and the block terms move eight-fold while the total moves 5.78, because the embedding tables do not care how deep the stack is. Raise the width from 768 to 12,288, sixteen-fold, and the block terms move 256-fold and the total 180-fold. That asymmetry is the section. 96 is the number the chapter gives you and 12,288 is the number it does not, and the second one is where the model actually is.

The formula is worth trusting only because it can be checked. On the GPT-2 124M shape it estimates 124,318,464 against the 124,412,160 the book prints in chapter 4, which is short by 93,696. That shortfall is not noise and not a bug: it is twelve blocks at 7,680 each, being two LayerNorms, an output-projection bias and two feed-forward biases, plus 1,536 for the final LayerNorm. The shorthand drops every norm and every bias, by exactly that much. On GPT-3, where nothing can be checked, it lands at 174,588,899,328 against a chapter that says "175 billion" and means it as a round number.

07 · The task moved into the prompt

There are three ways to make a model do a task, and only one of them touches the weights.

	what changes	how much of it
fine-tuning	the weights $\theta$	all 175 billion of them, by gradient descent
few-shot	the input $x$	a few hundred characters
zero-shot	nothing	no weights, no examples, and it still works

The word "learning" in "few-shot learning" is attached to the wrong row. Nothing is learned: the examples go into the input, no number in the model changes, and the effect vanishes the moment the prompt does. The chapter's own figure caption says this correctly, that these tasks are solved "without needing retraining, fine-tuning, or task-specific model architecture changes", and the terminology fights the caption.

The reason any of it works is the chapter's actual climax. The 2017 architecture was built to translate. A decoder-only model trained on nothing but next-token prediction translates anyway, and that surprised the researchers who found it. Raschka is careful about the mechanism, crediting "exposure to vast quantities of multilingual data in diverse contexts" rather than anything mystical, and he names the effect emergent behaviour: performing tasks the model was never explicitly trained to perform.

One precision the chapter has and casual writing loses: ChatGPT is not GPT-3. The first one was GPT-3 fine-tuned on a large instruction dataset, using the method from OpenAI's InstructGPT paper. That distinction is the subject of part 10.

08 · What's next

Everything above is one objective and a shift by one. Nothing in the remaining ten posts is a new idea about learning. What they add is the layer in the middle that lets one position look at the others, and then the machinery to train it.

Three things chapter 1 names and does not show, in the order the book reaches them:

• What a token actually is. Chapter 2. Every count in the labels figure above changes depending on the answer.
• How a position looks at the others. Chapter 3, and the dashed box in the family figure.
• How the block is wired. Chapter 4, where the parameter estimate above stops being an estimate.

The next post is the first of those: byte-pair encoding, the real GPT-2 tokenizer running in your browser rather than described, and the reason 20,479 characters of Edith Wharton come to exactly 5,145 tokens.

Appendix A is a PyTorch primer and this series skips it deliberately. The neural posts already taught tensors, autograd and the training loop in TypeScript, and repeating them would be a whole post before the book starts.

The library behind the live figures on this page is src/lib/minigpt, dependency-free TypeScript, seeded, sitting next to the mininn from the neural series and reusing it rather than replacing it. Where a figure needed a real model it got one offline, and the footer says which.