# Self-attention Part 7 of the tutorial "tiny-gpt". Canonical: https://learn.welldun.ai/tiny-gpt/07-attention/ The model has a window full of characters and reads only the last one. Letting it read all of them is easy; letting it decide which ones were worth reading is the idea the whole architecture is built on. --- The model can be trained now, and it is about as good as a model that sees one character can be. That ceiling is the problem. It wrote `sathe` — every pair legal, the word nonsense — because after producing `sat` it had already forgotten `sa`. Meanwhile there is a window of characters sitting right there, cut and batched and encoded, and the model reads exactly one of them. ## Just average them, then The obvious repair. Instead of taking the last character's row, take the average of every row in the window so far. That is strictly more information, and it is three lines of code. It also nearly works. Predicting after `the cat sat on the m`, an average gives the `m` — the character that actually determines what comes next — a twentieth of the say. The `the` from twenty positions back gets exactly as much. Everything in the window matters, and almost none of it matters equally. So the average has to be **weighted**. And the weights cannot be fixed in advance — not "always favour the recent", because which positions matter depends entirely on what they contain. In *the cat sat on the mat because it was soft*, working out what `it` refers to means looking at `cat` and `mat`. Neither is adjacent. `was` is adjacent and carries nothing. Any rule based on position alone gets this exactly backwards, which is why the weights have to be computed from the content every time. ## One number for every pair Look at what a weight actually is. It belongs to a *pair* — a position doing the looking and a position being looked at — and it says how much the first should take from the second. So it has to be computed from the two of them jointly, and the only material either one has is its content: the row of numbers the embedding table handed its character. Two rows in, one number out. The standard arithmetic for that is the **dot product**: multiply the rows entry by entry, add everything up — big for rows that agree, near zero for rows with nothing in common. A score for how well two rows go together. So the obvious move: weight position *j* for position *i* by dotting their rows, `x[i] · x[j]`. It has a flaw you can find with a pencil. Dot two *strangers'* rows and the products have no reason to agree — positive here, negative there, mostly cancelling. Dot a row *with itself* and every product is a square: never negative, nothing cancels. Four entries are enough to see it: | entry | row `a` | row `b` | `a` × `b` | `a` × `a` | |---|---|---|---|---| | 1 | 0.9 | −0.4 | −0.36 | 0.81 | | 2 | −0.5 | 0.7 | −0.35 | 0.25 | | 3 | 0.3 | 0.8 | 0.24 | 0.09 | | 4 | −0.8 | −0.2 | 0.16 | 0.64 | | | | **sum** | **−0.31** | **1.79** | Stranger against stranger wobbles near zero. Self against self piles up — and a row of sixteen everyday-sized entries piles to around **+16**, one per entry, for any row, before anything has been learned. So every position starts the contest for its own attention with a sixteen-point lead over every stranger, and softmax turns big leads into near-certainty: wired this way, each position spends almost all of its attention on itself and learns nothing from the window. The lab runs the check, and the numbers land where the squares say they must. ## What you look for is not what you are The fix is to stop comparing what positions *are*. Put each row through a learned matrix before comparing — and use a *different* matrix on each side: ```python Q = x @ W_q # queries — what each position is looking for K = x @ W_k # keys — what each offers to be found by V = x @ W_v # values — what each hands over if chosen ``` `x` is the window itself: one row per position, each row the embedding of that position's character. So each multiplication makes every position's vector at once — and each from its own row alone. Positions still have not met; the meeting is the dot products, next. Small enough to do by hand: a width-2 head and a two-position window. Make up the rows and two of the matrices — ``` x1 = [0.8, -0.6] W_q = [[ 1.0, 0.0], W_k = [[ 0.0, 1.0], x2 = [0.2, 1.0] [-1.0, 1.0]] [ 1.0, 0.0]] ``` A row times a matrix is one dot product per column. The first position's query: ``` q1 = [ 0.8*1.0 + (-0.6)*(-1.0), 0.8*0.0 + (-0.6)*1.0 ] = [1.4, -0.6] ``` Its key, through the other matrix the same way, is `k1 = [-0.6, 0.8]`; the second position's key is `k2 = [1.0, 0.2]`. Keep these — they meet in the next section. Same row in, three different vectors out. And the guaranteed squares are gone: a query dotted with a key multiplies two *different* rows, so matching yourself is no easier than matching a stranger — the self-share falls back to an even split — and who matches whom is decided entirely by the learned numbers. The value matrix is the same idea once more: what a position passes on need not be its raw row either. Those three matrices are the only things in attention that get learned. Everything after them is arithmetic that could not be otherwise. What you look for, what you offer, and what you hand over are three different things. That is the entire reason there are three matrices. ## Everything against everything One cell first: row *i*, column *j* holds position *i*'s query dotted with position *j*'s key — one number for how strongly what *i* is looking for matches what *j* offers. With the hand-built numbers, the first position looking at the second: ``` q1 · k2 = 1.4*1.0 + (-0.6)*0.2 = 1.28 ``` and looking at itself, `q1 · k1 = 1.4*(-0.6) + (-0.6)*0.8 = -1.32`. Set that against the raw rows: `x1 · x1 = 1.00`, `x1 · x2 = -0.44`. Raw, the self-match wins, as the squares said it must. Through the two matrices, the stranger wins. Who matches whom is now the learned numbers' decision and nothing else. Do that for every pair and the result is a square: one row per position doing the looking, one column per position being looked at. In code the whole square is a single multiplication, `scores = Q @ K.T` — the `.T` lays the keys on their side so every query-row meets every key-row. Three things happen to that square, and the figure steps through them — press **▶** to watch one row make the whole journey. The second checkbox replays the failure from above: tick *compare x with x* and the diagonal lights up — every row keeps at least 68% for itself, most 87% or more. Untick it and the advantage is gone. ## The future is off limits Position four is being asked what comes after position four. If it is allowed to look at position five, it can simply read the answer — and it will, because copying is easier than predicting, and the loss cannot tell the difference until you try to generate anything. So everything above the diagonal is forbidden — **masked**, in the standard word. In code the mask sets those scores to negative infinity *before* the softmax, because `exp(-∞)` is exactly zero: forbidden positions end up with no weight at all, and the permitted ones keep their relative sizes untouched. Two consequences, and the lab checks both: - Every row adds to 100%. Attention distributes a fixed amount of interest; it never creates more. - The first position can only see itself, so its output is exactly its own value, unchanged. And with that, the machine has all its parts: three learned matrices, the score grid, the mask, and the shares about to buy their blend. One complete unit of this is called a **head** of attention, and the width of its query, key and value vectors is the head's width. The name describes nothing — it is simply what everyone calls it — and the next part runs several at once. ## The divisor that makes it trainable Now untick *divide by √16* in the figure: every looking row sharpens, the largest shares jumping from the 0.32–0.75 range to past 0.77. The cause has nothing to do with language. A dot product adds up as many products as the vectors are wide, so wider heads mean bigger scores — measured over twenty thousand random pairs: | head width | spread of raw scores | after dividing by √width | |---|---|---| | 2 | 1.42 | 1.00 | | 8 | 2.84 | 1.00 | | 32 | 5.62 | 0.99 | | 128 | 11.25 | 0.99 | The spread grows exactly like the square root of the width, so dividing by exactly that cancels it. Skip it and the scores get big, softmax turns big scores into a spike, and a spike is flat — nudge any of those numbers and almost nothing changes, so almost no gradient comes back. The head stops learning before it has learned anything. One divisor separates a head that trains from one that sits still. ## What comes out Each row of shares buys a blend: the output is the share-weighted average of the values, one new vector per position. In numbers: suppose the second position's shares came out `[0.9, 0.1]` — nine tenths on the first position, a tenth on itself — and the value matrix handed over `v1 = [0.5, -1.0]` and `v2 = [1.0, 0.0]`. Its output is ``` 0.9*[0.5, -1.0] + 0.1*[1.0, 0.0] = [0.55, -0.90] ``` — one new width-2 vector, mostly what its neighbour handed over. Same shape as went in, entirely different content: position four no longer holds *the character `t`*, it holds *the character `t`, having read `the ca` behind it*. That is the thing that was missing. Whether the model uses it well is a matter of training, but for the first time in this tutorial the information is at least in the room. Nothing in the figure or the lab is trained, so which position attends to which is arbitrary and reading meaning into it would be a mistake. What the untrained version does show — the triangle, the rows summing to one, position zero returning its own value, and what the divisor does — is true of every head that has ever been trained. ## Build it yourself Six cells: the average that nearly works, the self-matching failure and the three matrices that fix it, the grid of every query against every key, the mask and its two guarantees, the measured reason for the divisor, and the whole head in six lines.