A model has never seen a word

Last time ended on the three ways to make a model do a task. Fine-tuning rewrites all 175 billion of GPT-3's numbers. A few-shot prompt rewrites none of them, because the examples go into the input instead. Zero-shot rewrites none of them and supplies no examples either, and still works.

That input is not text. It is a list of integers drawn from a fixed vocabulary of 50,257 entries, and the vocabulary was assembled by a program that has no idea what a word is.

This post is that program. It runs before the model exists, and it is the only part of a language model with no floating point in it at all. Chapter 2 builds it twice and then reaches for someone else's. Each version it builds fails in a way that forces the next, and the third it declines to build at all.

There are six figures on this page and none of them is a recording. Four compute everything they show from the 20,479 characters of Edith Wharton's "The Verdict", the public-domain short story the book uses, bundled with this post. The fifth runs the real GPT-2 tokenizer: 50,257 entries and 50,000 merge rules, 1.5 MB of OpenAI's own encoder.json and vocab.bpe, fetched when the panel scrolls into view and run by about 240 lines of TypeScript, checked against five tokenizations these sources print. The sixth computes two of its four bars and takes the other two from the figures above it, which its footer records.

01 · What a network can actually read

A network multiplies floating point numbers. Text is categorical: the word "cat" is not 3.7 more of anything than the word "bat". So before any of the machinery in the last post can run, the text has to become vectors, and the book spends its first section on the object that does that.

An embedding is a map from discrete things to points in a continuous space. Word2Vec is the historical reference point, a network trained to predict a word's context from the word or the reverse. The book names it and then makes the correction that matters: an LLM does not use downloaded vectors. It learns its own embedding table as part of its input layer, updated by the same loss as every other weight, because vectors optimised for the task beat generic ones.

GPT-2's smallest models, 117M and 125M parameters, use an embedding size of 768; the largest GPT-3, at 175 billion parameters, uses 12,288. Giles Thomas adds a correction worth carrying: 768 is the 124M model, and GPT-2 XL is 1,600.

Here is the whole reason this post exists. That table has one row per vocabulary entry, and a row is found by an integer. So before a single vector can be looked up, some program has to decide what the integers are.

02 · Splitting is a choice

The book's tokenizer is two lines of Python, and the first one is where every decision lives:

preprocessed = re.split(r'([,.:;?_!"()\']|--|\s)', raw_text)
preprocessed = [item.strip() for item in preprocessed if item.strip()]

The delimiter set is a choice someone made. Split on whitespace alone and "Hello, world. This, is a test." comes back as 11 items with the punctuation still glued on. Add the comma and the period and it becomes 19 items, now with empty strings in it. Filter those out and you get 10 tokens. Those are the book's own printed outputs, and the class above is the version in the book's repository; listing 2.3 as printed drops : and ;, and the counts below depend on keeping them.

Two decisions the book makes out loud. It does not lowercase, because capitalisation separates proper nouns from common ones and marks where sentences start. And it throws whitespace away, which saves memory, would be wrong for anything indentation-sensitive like Python source, and is described as being for now. That "for now" is a promise, and section 06 is where it gets kept.

Turn the delimiters off one at a time and watch what happens to the comma:

Two things the book prints and never explains. The empty strings come from the capturing group: a split that keeps its delimiters leaves a zero-length span wherever two of them sit side by side, and on the whole story there are 830 of those. And that innocent second line is doing two jobs at once, dropping the empties and dropping the whitespace, only one of which is obviously a good idea.

On the whole story the split produces 9,235 pieces and keeps 4,690: 20,479 characters in, 4.367 per token. Section 07 has the other half of that number.

03 · The vocabulary is a closed world

Sort the unique tokens, number them from zero, and that dictionary is the tokenizer. 4,690 tokens collapse to 1,130 unique ones. The first three entries are ('!', 0), ('"', 1) and ("'", 2); entries 49 and 50 are ('Her', 49) and ('Hermia', 50); the last three are ('younger', 1127), ('your', 1128) and ('yourself', 1129).

$V = \operatorname{sorted}\big(\{t_1, t_2, \dots, t_N\}\big), \qquad \operatorname{id}(t) = \text{the index of } t \text{ in } V$

Read that back in English and the correction falls out. The id of a token is its position in a list sorted by code point, which is not alphabetical order and is why ! is 0 and every capital letter sorts before every lowercase one. The book calls the sort alphabetical. Add one new document to the corpus and everything after the insertion point renumbers. Ids are not names. They are positions in a list that belongs to one particular pile of text.

Drag the slider across the story and watch the vocabulary stop growing:

The curve bends, which is the good news, and it bends for a reason with a name: the empirical observation usually called Heaps' law, that vocabulary grows sublinearly in corpus size. The plot is measured rather than fitted, and I am quoting no exponent for it, because nothing in my sources gives one for this text.

The bad news is inside the same 1,130 entries. 697 of them appear exactly once, which is 61.7% of the vocabulary spent on words the story never uses again. And it still does not cover the story it came from: build the vocabulary on the first 90% and the last 469 tokens already contain 41 it has never seen, 8.7%, across 38 distinct words. Among them are sure, coming, greatest and grace. Not exotic words. Ordinary ones that happen to arrive late.

04 · The crash, and the patch that loses

SimpleTokenizerV1 encodes and decodes against those 1,130 entries. Then the book hands it "Hello, do you like tea?" and it raises KeyError: 'Hello'. Not on some rare word buried in the middle. On the first token, index zero, because Edith Wharton never wrote "Hello".

Feed both versions the same sentence, starting with the one the book crashes on:

The important thing is not that V1 crashed. It is what V2 did instead. Run the book's joined pair through it and you get 16 ids with one 1130 and two 1131s, for "Hello" and for "palace". Two different English words are now the same integer. Nothing downstream can undo that. The information is gone at the tokenizer, before the model exists, and a tokenizer that can lose information puts a ceiling on what the model can ever learn.

There is a quieter failure in the same panel, which the book prints and says nothing about. Feed it a sentence where every token is in the vocabulary and the round trip still fails: "It's" comes back as "It' s". A join on spaces plus a regex that removes the space before punctuation cannot rebuild an apostrophe inside a word. The whitespace thrown away in section 02 was information too.

05 · Starting at the bottom

Byte pair encoding gets four sentences and a disclaimer: "a detailed discussion and implementation of BPE is out of the scope of this book." It imports tiktoken instead. Raschka opened the box himself later, in a bonus article, and this is the box.

Start with every byte value, all 256 of them. Count adjacent pairs. Merge the commonest into a new id. Repeat. Nothing neural, nothing linguistic, and the loop is six lines:

for new_id in range(len(self.vocab), vocab_size):
    pair_id = self.find_freq_pair(token_ids, mode="most")
    if pair_id is None:
        break
    token_ids = self.replace_pair(token_ids, pair_id, new_id)
    self.bpe_merges[pair_id] = new_id

$(a, b) \;=\; \arg\max_{(x,y)} \#\{\,i \;:\; s_i = x,\; s_{i+1} = y\,\}, \qquad |V_m| = 256 + m$

Pick the adjacent pair that occurs most often, give it the next unused id, and after m merges the vocabulary is 256 plus m. One footnote, because the figure below shows both columns: that count includes overlapping occurrences, which is what picks the winner, while the greedy left-to-right replacement pass fires fewer times when a pair overlaps itself. On this story that happens once in 1,527 merges.

Press merge three times on Raschka's own example, then once more, then let it run to the end of the story:

The first two lines of that Python are two different things and the figure makes it obvious. range(len(self.vocab), vocab_size) is a cap the caller chose. if pair_id is None: break is the algorithm's own stop rule. Raschka's printed table for the cat in the hat shows three merges ending at 12 tokens, because his trainer takes a vocabulary size; let the same example run until no pair occurs twice and a fourth merge fires, at, ending at 10.

One honest note. Raschka's own implementation is character-level, starting from the first 256 Unicode code points rather than raw bytes. Real GPT-2 is byte-level. On this pure-ASCII corpus the two agree, but only the byte version is true in general, and the byte version is what runs above.

The first four merges on the whole story are "e ", " t", "d " and "t ". Not one is a morpheme; three of them are half a word plus a space. Run to exhaustion, 1,527 merges and a vocabulary of 1,783, and the longest thing it has invented is a 29-character token spelling : "Be dissatisfied with your , a clause that appears in this story and nowhere else in English. BPE does not find units of meaning. It finds repeated bytes, and on a small enough corpus it will happily memorise sentences. That is what a frequency cutoff protects against, and it is why 50,000 merges need an internet rather than a short story.

06 · The real one

The tokenizer GPT-2 shipped has 50,257 entries: 50,000 merges, the 256 byte values, and one <|endoftext|> at id 50256. The book installs tiktoken and calls get_encoding("gpt2"). The figure below runs the same algorithm against OpenAI's own published tables, in your browser.

Type something the story has never contained, and watch where the spaces went:

The round-trip line stays green for anything you can type, emoji included. There is no failure mode because there is no closed world: all 256 byte values have an id before any merge is applied. That is the entire content of "BPE handles unknown words." It does not handle them. It never meets one.

The space in front of a word belongs to the word. do do is [4598, 466], which decodes per token as do and do, and Giles Thomas noticed the detail that makes it stick: the space-prefixed variant has the lower id, so GPT-2 saw it more often. That is the whitespace promise from section 02, kept. It also means the decode is exact rather than approximate, which the <|unk|> tokenizer never was.

<|endoftext|> is worth one paragraph on its own, because it is not one behaviour but three. Pass allowed_special={"<|endoftext|>"}, as the book does, and the string parses as the special token, id 50256. Pass nothing, which is the default, and tiktoken raises rather than guess. Force it to be ordinary text and it becomes seven boring tokens. Giles Thomas made the observation this deserves: a control token that is also a string in the user's input channel is in-band signalling, and the part of your brain that flags SQL injection should be flagging this too. The default raise is the parameterised-query answer. The figure above has no special-token handling at all, which puts it in the third state, and it says so on screen.

The claim that this is really GPT-2's tokenizer is checkable, so here is the check:

$ npx tsx scripts/llm/part02-verify-bpe.mts
PASS "Akwirw ier" -> [33901,86,343,86,220,959]                      <- book, exercise 2.1
PASS "Hello, do you like tea? " -> [15496,11,466,345,588,8887,30,220]  <- book, section 2.5
PASS " In the sunlit terraces of someunknownPlace." -> [554,262,4252,18250,8812,2114,286,617,34680,27271,13]  <- book, section 2.5
PASS "This is some text" -> [1212,318,617,2420]                     <- Raschka, bpe-from-scratch
PASS "do do" -> [4598,466]                                          <- Giles Thomas, part 2

the-verdict.txt: 5145 tokens from 20479 chars in 8ms, round trip exact
vocabulary 50257, <|endoftext|> 50256, distinct ids used 1416
5/5 published-vector cases passed

That first line is exercise 2.1. Eight milliseconds for the whole story is also why the panel above needs no worker: the cost is not encoding but getting the tables usable, 1.5 MB of download and about 24 ms of parsing and map building, once, behind the spinner.

07 · What a vocabulary costs

The book prints both of the next two numbers, several pages apart, and never puts them side by side.

what	tokens	vocabulary	chars/token over the file	chars/token over what its tokens contain
characters, which are bytes here	20,479	256	1.000	1.000
regex word tokenizer	4,690	1,130	4.367	3.574
BPE trained on this story	5,102	1,783	4.014	4.014
GPT-2 BPE	5,145	50,257	3.980	3.980

GPT-2 uses 5,145 tokens where the word splitter used 4,690. It is 9.7% worse, on the book's own corpus, with a vocabulary 44 times larger. The two right-hand columns differ for exactly one row, and that row is the one making the comparison look good: the word tokenizer's 4,690 tokens contain 16,764 characters, not 20,479. It threw the other 3,715 away. Measured against the text it actually represents, it gets 3.574 characters per token, which is worse than GPT-2's 3.980.

Drag the vocabulary size and watch the two costs trade against each other:

The 455-token gap has a shape. Of GPT-2's 5,145 tokens, exactly 164 decode to pure whitespace, and every one of those 164 contains a newline: they are the line breaks the word tokenizer discarded. The other 291 are words GPT-2 had to break into pieces. The inter-word spaces cost nothing extra at all, because they ride inside the token. And comparing the two on token count alone compares a lossy encoding with a lossless one. You cannot get the file back from the 4,690. You can from the 5,145, byte for byte.

The curve for a tokenizer trained here flattens and then stops, because after 1,527 merges no pair in the whole story occurs twice. It lands within 0.8% of GPT-2's token count with a vocabulary 28 times smaller, which looks like a terrible deal for GPT-2 until you remember what the 50,257 is for. Not this file. The next one. On this file GPT-2 spends 1,416 distinct ids, 2.8% of what it carries, and the other 97.2% is there for text it has not been shown.

08 · What comes next

Two type changes, and everything between them was a regex, a sorted list, or a counter in a loop. str became string[], then number[]. The vocabulary that made the second arrow possible is not a property of English; it is a design artifact derived from one pile of text, and every scheme that treats words as the unit inherits a closed world and therefore a crash.

Three things this skipped, in rough order of how much they would change the numbers above.

GPT-2's own pre-tokenizer regex, which runs before any merge and is why punctuation and digits split the way they do.
Vocabulary size as a hyperparameter, traded against sequence length and against an embedding table with one row per entry.
Multilingual token cost. The same 50,257 entries do much worse work on scripts they saw little of, which is a tax paid by some readers and not others.

The next post takes these 5,145 integers and turns each one into a row of a matrix, which turns out to be a one-hot vector times a weight matrix and therefore a trainable layer rather than a preprocessing step. Then it adds a second table indexed by position, and produces the exact tensor chapter 3 consumes.

The tokenizers behind these figures are dependency-free TypeScript in src/lib/minigpt, about 740 lines in one file, with no floating point in any of it. The word tokenizer and the BPE trainer are mine. The GPT-2 encoder is a reimplementation of OpenAI's encoder.py, checked against the five published tokenizations above, and the tables it runs on are OpenAI's own files, unmodified, downloaded once and served from this post. The most useful thing I did while writing it was run the trainer to exhaustion and read the longest token it had invented.