Attention with no parameters at all

Last time chapter 2 finished its job. The 20,479 characters of a public-domain Edith Wharton story became 5,145 token ids, a sliding window cut those into four-token rows, each id pulled a row out of a 50,257 by 256 embedding table, a position vector was added on top, and one batch of eight rows came out as a tensor of shape [8, 4, 256]. That is 12,865,792 parameters in the token table alone, and not one of them has looked at another token yet.

That is the hole. The row for the at position 1 and the row for the at position 4 differ by their two position vectors and by nothing else, and nothing in the pipeline so far knows that the second one belongs to a different noun. A token's vector still means what the token means on its own. Chapter 3 is the machinery that fixes that, and it builds the fix four times, each version adding exactly one idea to the last: no trainable weights, then trainable weights, then a mask, then several heads. This post is the first of the four, the one with no parameters in it at all.

Five of the six figures on this page recompute everything they show from the same 6 by 3 tensor the book prints on its first page of chapter 3, in your browser, at full float64 precision. The sixth, in section 01, counts two exact integers and raises one number you set to a power, and says inside the panel which is which. Nothing here is fetched and nothing is a recording.

01 · One vector, and everything after it

Before transformers, the machine that translated a sentence was an encoder-decoder pair of recurrent networks. The encoder walked the source sentence one word at a time, updating a hidden state, trying to squeeze the whole meaning into the state it held at the end. The decoder started from that vector and emitted the translation. Raschka is explicit that you do not need to understand RNNs to follow the chapter, only that they had a bottleneck, and he states it plainly: the RNN cannot directly access earlier hidden states from the encoder during decoding. Everything the source sentence said has to fit in one vector, because that is all the decoder is given.

There is a second failure underneath the first, and it is arithmetic rather than architecture. Giles Thomas, working through the same book, writes out the part the chapter skips: training a recurrent network means unrolling it in time. Feed a two-layer network a sequence of ten inputs and the backward pass is ordinary backpropagation through 20 layers. Take a five-layer network on a hundred-token sequence and the backward pass goes through 500. That is the same multiplicative decay the backprop post measured on a plain feedforward stack, with the same cause: every layer the gradient crosses multiplies it by another derivative, and for a sigmoid that factor is at most 0.25.

Set the sequence length to 100 and count how far back the gradient has to walk:

The recurrent depth is a straight line in T, because it is the product of two numbers the modeller chose. The attention depth is flat: attention reaches any position from any other in a single hop, so the walk from the end of a sequence to its start is one hop per layer whatever the length. At five layers and a hundred steps the comparison is 500 against 5, and both are exact integers. The decay curve beside them is not a measurement, it is a number you set raised to the power of the sequence length. The feedforward layers shrink a gradient in both architectures, so they are left out of that curve entirely: the contrast is about the sequence axis and nothing else.

02 · Permission to look back

The fix arrived in 2014, and it was a patch on translation rather than a new architecture. Bahdanau, Cho and Bengio modified the encoder-decoder so the decoder could selectively read different parts of the source at each output step, with learned weights deciding what to read. Their own framing of the problem is the one worth keeping: the fixed-length vector is the bottleneck, and the fix is to let the model soft-search the source for the parts relevant to the word it is about to emit.

Giles Thomas has the cleanest way to hold this. You can read an English sentence, keep the whole meaning in your head, and type out the German; or you can read it, keep the gist, and glance back at the original as you go to check you have included everything. The second is obviously easier, and it is the entire idea. Attention is not a clever trick. It is being allowed to look back at the page.

Three years later the transformer keeps the glancing and throws the recurrence away. The mechanism it uses instead is self-attention: every position considers the relevance of all the other positions in the same sequence when computing its own representation. The self is doing real work there. Traditional attention relates elements of two sequences, a source and a target. Self-attention relates positions inside one.

03 · Three steps, one word

The chapter starts with six tokens of the sentence "Your journey starts with one step.", embedded in three dimensions. Raschka says why three: he chose a small embedding dimension so a row fits on the page without a line break. Every number in the rest of this post is a function of these eighteen.

inputs = torch.tensor(
  [[0.43, 0.15, 0.89], # Your     (x^1)
   [0.55, 0.87, 0.66], # journey  (x^2)
   [0.57, 0.85, 0.64], # starts   (x^3)
   [0.22, 0.58, 0.33], # with     (x^4)
   [0.77, 0.25, 0.10], # one      (x^5)
   [0.05, 0.80, 0.55]] # step     (x^6)
)

Notation, declared once. A parenthesised superscript indexes a token, so $x^{(2)}$ is the second one. $T$ is the number of tokens and $d$ the embedding dimension, so here $T = 6$ and $d = 3$. An $\omega$ is a score, an $\alpha$ is a weight, and a $z$ is a context vector: an enriched embedding that carries information about its own token and about all the others.

$\omega_{ij} = x^{(i)} \cdot x^{(j)} = \sum_{k=1}^{d} x^{(i)}_k x^{(j)}_k$

$\alpha_{ij} = \frac{\exp(\omega_{ij})}{\sum_{k=1}^{T}\exp(\omega_{ik})}$

$z^{(i)} = \sum_{j=1}^{T} \alpha_{ij}\, x^{(j)}$

Pick a query token, drag any of the eighteen numbers, and watch the three steps recompute:

Three lines, and each one is a sentence. Dot the query embedding with every embedding, including itself, to get $T$ scores. Squash those scores into $T$ weights that are positive and sum to 1. Take that weighted average of the embeddings. On the book's own numbers with journey as the query the scores are [0.9544, 1.4950, 1.4754, 0.8434, 0.7070, 1.0865], the weights are [0.1385, 0.2379, 0.2333, 0.1240, 0.1082, 0.1581], and the context vector is [0.4419, 0.6515, 0.5683], which is what the panel shows until you touch a slider.

Two things follow from the third step being an average. Because the weights are positive and sum to 1, the result is a convex combination, so a context vector is always a point strictly inside the hull of the six inputs. And because nothing is stored, dragging one coordinate of one token moves every row of the matrix: that token appears in every other token's scores. There is no table of learned affinities anywhere. The weights are recomputed from the embeddings on every forward pass, which is worth saying twice, because "attention weights" sounds exactly like the kind of thing that would be trained.

04 · Why not divide by the sum

The book reaches softmax by way of the obvious alternative. Divide each score by the sum of the scores and you get [0.1455, 0.2278, 0.2249, 0.1285, 0.1077, 0.1656], which sums to 1 and looks like a perfectly good set of weights. It then discards that in favour of softmax for three stated reasons: softmax handles extreme values better, it has more favourable gradient properties, and its outputs are always positive.

The third reason is the one you can break, and the book never breaks it, because it cannot: all eighteen coordinates in the fixture are positive, so all 36 dot products are too. Change one number and the floor gives way.

Press the negative preset and read the middle column:

Flip the first coordinate of one from 0.77 to -0.77 and the score of Your against it becomes -0.2046. Divide by the row sum of 3.7978 and the weight for one is -0.0539, which asks the weighted sum for a negative amount of a token. Softmax on the same six scores returns 0.0669, because exp has no way to return a negative. All three of those numbers are derived here rather than printed anywhere. The column containing the bad weight still sums to 1.000, which is exactly why this bug is easy to ship: summing to 1 is necessary and it is nowhere near sufficient. The near-zero preset is the same failure louder: a denominator of 0.0100 producing a weight of 90.

Raschka's softmax_naive carries a warning that it can overflow and underflow, and his advice is to use torch.softmax instead. The subtract-the-largest-score trick that makes it safe is not in the book; it is what the pixels post showed, where $e^{1000}$ overflows to infinity and the quotient of two infinities comes back as NaN. This post applies softmax straight to the raw scores. The next one divides them by something first, and that division is about gradients rather than tidiness.

05 · One query is one row

One query is one row of a matrix, and the chapter's second half is the refactor that says so. The double loop fills a 6 by 6 grid one dot product at a time, and the grid it fills is exactly inputs @ inputs.T.

attn_scores = torch.empty(6, 6)

for i, x_i in enumerate(inputs):
    for j, x_j in enumerate(inputs):    # 36 iterations in total
        attn_scores[i, j] = torch.dot(x_i, x_j)

attn_scores = inputs @ inputs.T         # the same 36 numbers

The book invites you to check that the two agree and does not say why they must. Giles Thomas fills that gap in, and the rule is one sentence: the element at row $i$, column $j$ of a matrix product is the dot product of row $i$ of the first matrix with column $j$ of the second. Row $i$ of inputs is token $i$'s embedding. Column $j$ of the transpose is token $j$'s embedding, because transposing turns rows into columns. So cell $[i, j]$ is $x^{(i)} \cdot x^{(j)}$, which is the definition of the score.

Step the loop one cell at a time, then press the matmul button and watch the same 36 numbers arrive at once:

The two grids agree to the last digit because they are the same 108 multiplications in a different order. Matrix multiplication is not a faster algorithm here. It is one expression instead of two loops, and the reason to want it is that the expression does not change when $T$ goes from 6 to 1,024.

$\Omega = X X^{T}, \qquad A = \operatorname{softmax}_{\text{row}}(\Omega), \qquad Z = A X$

$X : (T, d), \qquad \Omega, A : (T, T), \qquad Z : (T, d)$

attn_scores      = inputs @ inputs.T
attn_weights     = torch.softmax(attn_scores, dim=-1)
all_context_vecs = attn_weights @ inputs

Three lines is the whole of section 3.3, and row 2 of that last result is [0.4419, 0.6515, 0.5683], identical to the hand-computed context vector from section 03. One property of those matrices is not obvious. $\Omega$ is exactly symmetric by necessity, because $a \cdot b$ and $b \cdot a$ are the same sum, so the largest difference between $\omega_{ij}$ and $\omega_{ji}$ anywhere is 0. $A$ is not symmetric: the largest gap between $\alpha_{ij}$ and $\alpha_{ji}$ is 0.087657, because each row is divided by its own denominator. Both are derived here.

Then the cost. Six tokens is 36 dot products and 108 multiplications, and the count is $T^2 d$, quadratic in the sequence length. The book gives two real numbers to hang that on: the smallest GPT-2 has a context vector embedding size of 768, and exercise 3.3 notes it supports a context length of 1,024 tokens. At those, one layer's score matrix alone is 805,306,368 multiply-accumulates. It stops there: chapter 3 never states a layer count.

06 · What it actually did

None of the numbers in this section are in the book. They are derived here from the book's own tensor, and the figures recompute every one of them in front of you.

Start with the dynamic range. The largest weight anywhere in the 6 by 6 matrix is 0.2379 and the smallest is 0.0988, against the 0.1667 a flat average would put in every cell. Between its most and least favoured cell, the whole mechanism moves the needle by a factor of 2.408.

Entropy in bits is how many yes-or-no questions a distribution costs you on average. A flat six-way choice costs $\log_2 6 = 2.585$ bits, and a distribution that has genuinely decided something costs less. The six rows here cost 2.549, 2.519, 2.521, 2.560, 2.564 and 2.534 bits. Every one of them is between 97.4% and 99.2% of maximum entropy, and the largest gap to the ceiling anywhere in the matrix is 0.066 bits.

Switch the weights to a flat sixth each and see how far the six outputs move:

Replace the mechanism entirely with "average all six inputs equally" and no context vector moves more than 0.080. The six input embeddings sit 0.6576 apart on average. The six context vectors sit 0.0482 apart, 7.3% of the spread they came from, and a nearest-neighbour lookup puts all six of them closest to the same input embedding, starts. Six tokens went in and six near-copies of one point came out.

So what was it ranking? The diagonal of $\Omega$ is $\|x^{(i)}\|^2$, the squared length of each embedding, which means a long vector scores high against everything including itself. journey and starts have the two longest vectors in the tensor, at norms 1.2227 and 1.2071, and in five of the six rows the largest weight goes to one of them. Row 1 is the only exception, and it wins by putting its largest weight on itself, beating journey by 0.0093.

Scale any one embedding and watch its column of the matrix:

The book says a dot product is a measure of similarity, that a higher one indicates greater alignment. That is true only with a qualifier the book does not give: a dot product is alignment times both magnitudes, so a token with a long embedding attracts attention regardless of which way it points. Take with, the shortest vector at norm 0.7026, scale it by three, and its column becomes the largest weight in all six rows, including a weight of 0.687 on itself. Nothing about the sentence changed. Only a length did.

Cosine similarity is the obvious repair and it is not one. Divide each score by both norms before the softmax and the magnitude effect is gone, but on the untouched tensor the distributions get flatter in five of the six rows, and the mean row entropy rises from 2.5413 bits to 2.5673 against the same 2.5850 ceiling. Row 5 is the one exception, falling from 2.5642 to 2.5577. Worse, cosine makes every token attend hardest to itself, because a vector's cosine with itself is 1 and nothing can beat it. Removing the magnitude removes most of the little this thing was discriminating on. The dot product was not the wrong similarity measure. Similarity was the wrong question, and there is nowhere in this mechanism for a parameter to go. Every number in it is a function of the input embeddings and nothing else. It cannot get better at English, because there is nothing in it to train.

07 · What comes next

The chapter builds attention four times, and this was the first. What the next post adds, in the order the book adds it:

• Three projections. $W_q$, $W_k$ and $W_v$ give a token a question, an answer and a payload, three different learned views of the same vector. The moment $W_q \ne W_k$, the score matrix stops being symmetric by necessity, and the asymmetry section 05 measured in the weights becomes a property of the scores themselves.
• A division before the softmax, which is where "scaled dot-product attention" gets its name.
• A triangular mask, so a token cannot read the tokens after it.
• Heads, which is the same computation run several times on slices of one projection.

Everything about how modern models changed attention after 2019, the variants that replaced this arrangement wholesale, belongs to the last post in this series and is deliberately not here.

The figures on this page are a few hundred lines of dependency-free TypeScript over an eighteen-number constant, next to the 1,692 lines of the neural series' library they borrow a matrix type from. The most useful thing about writing them was that section 06's verdict was not the one I expected going in. I went looking for a mechanism that worked badly and found one that had barely done anything at all, and that gap is the argument for the three passes the chapter has left.