The training labels were already in the file

Last time ended with the same 20,479 characters counted four ways. A regex splitter over a 1,130-entry vocabulary got 4,690 tokens and could not encode the word "Hello". Byte pair encoding trained on the story itself ran out of pairs after 1,527 merges. Real GPT-2, 50,257 entries, turned the whole story into 5,145 integers at 3.980 characters each, and took the nonsense string Akwirw ier without complaint, because it has no unknown token and no need for one.

Those 5,145 integers are where this post starts. It ends holding one tensor of shape [8, 4, 256]: eight sequences, four tokens each, 256 numbers per token. That is exactly what the first transformer block is handed, and between here and there are three ideas, two of which surprised me.

There are five figures below. Four compute every number they show, in your browser, either from the token stream bundled with this post or from their own arithmetic. The fifth puts a live computation beside eighteen floats transcribed from the book's printout, so your copy and this page agree, and says so on the panel. Two of the four draw seeded Gaussians where a trained model would supply real values, and say that too. None of the five is a recording. And nothing here is trained, which is the thing chapter 2 says least loudly.

01 · The labels were in the text

Supervised learning wants labelled pairs, and nobody labelled Edith Wharton. What chapter 2 does instead is notice that the labels are already in the file. For a next-token predictor the answer to any prefix is the token that follows it, so take a window of $L$ tokens as the input and the same window slid one place right as the target, and one short story becomes a labelled dataset with no annotator, no scraper and no budget.

x^{(i)} = (t_i,\, \dots,\, t_{i+L-1}), \qquad y^{(i)} = (t_{i+1},\, \dots,\, t_{i+L})

Read back in English: the $k$ -th target is the token after the $k$ -th input, which is y_k = t_{i+k+1}. The book demonstrates it at context_size = 4 starting from token 50, and its stated reason for skipping the first fifty is that the passage after them reads more interestingly. Readers hunt for meaning in that 50. There is none.

Step through the four rows the book prints, then switch to text and look at where the spaces went:

x is [290, 4920, 2241, 287] and y is [4920, 2241, 287, 257], the same five ids read through a window that moved by one. Decoded, those are " and", " established", " himself", " in" and " a", and every one of them carries its leading space, which is the whitespace promise the previous post spent a section keeping. The growing-prefix rows make the shift concrete: each row's context is the row above it plus the answer the row above it was graded on.

Turn the mask toggle on and everything past the current target dims. That is figure 2.12, slipping the causal-mask idea into a data-loading section two chapters before anything formalises it: during training the model may not look past the token it is being asked about, which is the only thing keeping this from being a lookup of the answer.

02 · Six predictions, not one

There is a shape mismatch in the book's data loader that chapter 2 never accounts for. GPTDatasetV1 returns a target tensor, not a target token: four inputs go in and four targets come back, when the obvious formulation needs one. Giles Thomas, writing his way through the same book in public, stopped on exactly that sentence and invoked his own no-side-quests rule. His series does not answer it until part 15. He gave the short answer in a comment thread first, and it is the clearest statement of it anywhere.

His answer, in my words: feed the model the fat cat sat on the and it works out a next token for every input token, so ideally it hands back fat cat sat on the mat. At inference you keep the last one and discard the rest, and those discarded five are, in his phrase, side effects of the work needed to compute the last one in the context of the others. At training the loss grades all six.

Flip between the two modes and count what survives:

The probabilities on that panel are seeded noise, because nothing here has been trained and nothing will be for several posts, and the caption says so. The count is the payload. Six input positions produce six graded predictions from one forward pass, and no position waits on the one before it. A 1,024-token context yields 1,024 loss terms from a single pass over the sequence.

The comparison people reach for next is usually wrong, so here it is, carefully. A recurrent language model trained with teacher forcing also emits one prediction and one loss term per timestep, so it gets the same 1,024 signals from the same 1,024 tokens. The difference is when, not how many. The transformer computes all of them at once with no sequential dependency between positions; the recurrent model reaches the same total after 1,024 sequential steps. That is a claim about parallelism, not about supervision.

03 · Two dials

Three lines inside the book's GPTDatasetV1 carry the entire data pipeline, and they earn a fence because the argument is in the indices:

for i in range(0, len(token_ids) - max_length, stride):
    input_chunk  = token_ids[i:i + max_length]
    target_chunk = token_ids[i + 1: i + max_length + 1]

max_length and stride are two independent dials, and running them together is the commonest way to misread this loop. max_length sets how wide each row is. stride sets how far apart consecutive rows start, and therefore how many rows there are and how much they overlap. A naming trap first: the chapter uses three names for the same 4, context_size above, max_length in the loader, context_length once embeddings arrive. They come apart in a real model, where context_length is a fixed architectural limit, 1,024 for GPT-2, and max_length is a data-loading choice that must be no larger.

One coincidence to defuse before it bites. Right after the first demo the book notes that an input size of 4 is chosen for simplicity and that real training uses at least 256. That 256 is how many tokens a row holds. The 256 in the next section is how many floats one token becomes. They are unrelated.

Drag both sliders and watch the three numbers at the bottom move against each other:

N = \left\lceil \frac{n - L}{s} \right\rceil, \quad \text{batches} = \left\lfloor \frac{N}{B} \right\rfloor, \quad \text{overlap} = \frac{\max(L - s,\, 0)}{L}

create_dataloader_v1 defaults to batch_size=4, max_length=256, stride=128, shuffle=True, drop_last=True, num_workers=0, but the two demos in the text run at (1, 4, 1) and (8, 4, 4), both of which the widget reproduces from the real stream. Derived from the book's numbers: 5,145 tokens at $L = 4$ , $s = 4$ give 1,286 rows, which at eight rows a batch is 160 batches with 6 left over. The book picks $s = L$ so that we do not skip a single word, and warns that overlap could increase overfitting. At $s = 1$ the same story gives 5,141 rows, 75% overlap, and every interior token trained on four times.

drop_last=True throws those 6 leftover rows away, and it is not the answer to ragged data. Every row this loader emits is exactly max_length wide, so nothing here is ragged. It exists so that no batch is short, which the book's annotation says prevents loss spikes during training. Padding and attention masks solve a different problem, one that fixed-width windows sidestep.

The two giles presets are his own experiments, run because he could not reason out the edge case from the source. Seven tokens at max_length=3 and stride=2 yield two pairs, and the token 7 never enters any window: a window that cannot produce a full-width target is dropped, not padded and not truncated. The exclusive upper bound in that range looks like an off-by-one and is not. It excludes one input window, never a token. At $s = 1$ the last window starts at 5,140 and its target ends on token 5,144, the last of the stream, which is why tokens never used reads 0 there.

The book leaves one thing alone here, and it is the reason to care about the stride dial beyond row counts. Under learned absolute position embeddings, at stride $L$ a token is only ever presented at one alignment: the character at file offset 37 is always position 5. I trained a small character-level transformer both ways to check. At stride = contextLength it reached 95.1% teacher-forced accuracy on its own training text and still could not recite twelve consecutive characters of it, because free-running generation slides the window by one and asks it about alignments it has never seen. At stride = 1 it memorised less and recited more. The dial that looks like a data-volume setting is also an alignment setting, and the post in this series that trains something comes back to it.

04 · A lookup is a matmul

Token ids cannot go into a network as numbers. Giles Thomas puts the objection better than I would: if once is token 123 and more is token 124, the model has no use for 123.5, because there is nothing halfway between them. Ids are positions in a sorted list, as the previous post spelled out, and arithmetic on them is meaningless.

The honest encoding is one-hot: a vector of 50,257 zeros with a single 1 in the position of the token. That carries no false ordering, and a first layer of weights can learn about each entry independently. Nobody materialises such a vector, and the reason nobody has to is the whole of section 2.7. torch.nn.Embedding(vocab_size, output_dim) is a vocab_size by output_dim matrix of random numbers, and calling it with an id returns that row.

Click a token id, then switch to matmul view and watch every term but one go to zero:

\mathbf{o}_i^{\top} \mathbf{E} = \mathbf{E}_{i,:}

A one-hot row vector times a matrix selects a row of that matrix. The book makes this point itself, calling the embedding layer "a more efficient way of implementing one-hot encoding" followed by a matrix multiply in a fully connected layer, and it is the licence for everything that follows. The lookup is not a shortcut around the layer. It is the layer. Gradients flow back into exactly the rows that were read, the table is trained by the same loss as every other weight, and the only thing the index buys is that you never build the wide vector. At GPT-2's scale that saving is not small: 50,257 x 768 is 38,597,376 multiply-adds per token, of which all but 768 are multiplications by zero, against 768 memory reads. A ratio of 50,257 to one, for an identical answer.

Which raises what that table costs, and the answer moves with the model. The denominators below are the book's own, from chapter 4 and appendix C, and they count the table once because GPT-2 ties the input embedding to the output projection. The shares are my arithmetic on those numbers.

configuration	vocabulary x dimension	table	model total	share
the book's demo	50,257 x 256	12,865,792	n/a	49.08 MiB in float32
GPT-2 small	50,257 x 768	38,597,376	124,412,160	31.0%
GPT-2 XL	50,257 x 1,600	80,411,200	1,557,380,800	5.2%
GPT-3	50,257 x 12,288	617,558,016	175,000,000,000	0.35%

The interesting column is the last one, falling by a factor of nearly ninety while the table itself grows sixteenfold. The vocabulary is fixed at 50,257 however large the model gets; the depth is what grows. The table is a third of GPT-2 small and a third of one percent of GPT-3, and at GPT-3's width it is still, on its own, five times the whole of GPT-2 small.

05 · The same token in two places

The table returns the same row for a token wherever it sits. Figure 2.17 draws exactly that: id 5 in slot 0 and id 5 in slot 3 come back identical. Self-attention will not repair it either, because a weighted sum over positions is indifferent to their order. So order gets injected, with a second table indexed by position instead of by token.

Turn positions off, then compare slot 0 with slot 2:

With positions off the two slots are pixel-identical and their cosine is exactly 1.000, which is the only red on this page. Switch to add and a fresh nn.Embedding for the positions is enough to pull them apart: at the seed this panel ships, the cosine falls to 0.766 and the vectors visibly differ. The three lines that do it are the destination of the whole post:

token_embeddings = token_embedding_layer(inputs)                       # [8, 4] -> [8, 4, 256]
pos_embeddings   = pos_embedding_layer(torch.arange(context_length))   # [4, 256]
input_embeddings = token_embeddings + pos_embeddings                   # [8, 4, 256]

One operator on line three, and the dimension does not grow. The concat mode shows the alternative, drawn in grey rather than red because it is a real design and not a defect: lay the two vectors end to end and the width doubles. At GPT-2's $d = 768$ that turns the first weight matrix consuming it from 768 x 768, or 589,824 parameters, into 1,536 x 768, or 1,179,648. The extra 589,824 buys separation the addition already got for free, and every matrix downstream pays the same tax. Addition instead forces the model to keep "which token" and "where" apart inside the same 256 numbers, and it manages.

The last control is the most useful thing on the page, and it comes from a correction rather than from the book. Giles Thomas read figure 2.18 as licensing position vectors of all-ones for slot 0, all-twos for slot 1 and so on, and wrote that into his part 3. A reader named Simon objected in the comments: under that initialisation the token identity becomes rounding error against the position index. Giles agreed, corrected the post, and concluded that Raschka had been drawing a simplified example.

Simon was right, and it is worth being exact about who said what. The book's own caption on figure 2.18 simplifies the token embeddings, not the position vectors: "The token embeddings are shown with value 1 for simplicity." Its code is a fresh nn.Embedding, whose weights PyTorch documents as drawn from a standard normal, the same scale as the token table. Switch P init to all-k and drag the position index: by position 3 the position vector already outweighs the token vector twice over at this seed, and by position 1,023 it does so 603 times, at which point the token bars are decoration.

Both tables, at this point, are random numbers. Every impressive-looking tensor in chapter 2 is randn with the right shape, and it stays that way through the next three posts. The book says so once, in a sentence, against eight pages of printed floats.

06 · What's next

Three type changes, and not one of them is clever. str became int[] in the last post; here int[] became [B, L] by slicing, and [B, L] became [B, L, d] by reading rows off a table. The only arithmetic in the whole pipeline is one addition.

Three things this skipped, in rough order of how much they would change the numbers above.

Document boundaries. Real pretraining corpora concatenate unrelated documents with <|endoftext|> between them, or the model learns the transition from one text to the next as if it were language. The previous post owns that token; here there is one document.
Padding and attention masks. Not the same thing as drop_last, which drops a short batch to keep the loss stable. Ragged documents need real padding and a mask that tells attention to ignore it. Fixed-width windows over one long stream sidestep the problem.
Other position schemes. The book builds learned absolute positions because that is what GPT-2 did, and describes relative ones in a paragraph without building them. Rotary position embedding, arXiv:2104.09864, encodes offsets rather than absolute indices, and the field has since had to bolt interpolation schemes onto it to run past the context it was trained on. The last post in this series takes the modern architecture apart; this is a pointer, not a treatment.

The [8, 4, 256] tensor is the input to the next post, which is where each token finally gets to look at the rest of the sentence. There is still nothing trainable in the middle: the first pass at attention has no parameters at all, and the three learned matrices arrive the post after that.

The figures run on src/lib/minigpt, dependency-free TypeScript. The window arithmetic is integers throughout; the figures holding vectors draw them from seeded generators, so the panel you see is the panel I measured. The token stream was produced offline by the same byte-level BPE encoder the previous post runs live, checked against the book's printed 8x4 block before it shipped. The one thing on this page I did not compute is the eighteen floats in section 04, which are the book's own printout so your copy and this page agree.