# Many heads at once
Part 8 of the tutorial "tiny-gpt".
Canonical: https://learn.welldun.ai/tiny-gpt/08-multi-head/
A head asks one question of the past and answers with one weighted average — and averaging shreds: two different pasts can leave identical blends. So run several heads at once and lay their answers side by side.
---
Each position now leaves attention carrying one answer, gathered from its past
by one **head** — the last part's whole machine: three learned matrices asking
who matches whom, a mask vetoing the future, and a blend of what the chosen
positions hand over, weighted by the shares. The question is whether one
answer is enough.
Take *the cat sat on the mat because it was soft*, mid-sentence. Continuing it
well needs the recent grammar — an adjective is underway. Working out what `it`
means needs `cat` and `mat`, much further back. Two different needs, pointing
at different places, at the same moment.
## One average can only blend
A head could split its attention — half on the grammar, half on `mat`. Here is
why that is not the same as having both.
A head's answer is a weighted average of **values** — the vectors each chosen
position hands over, from the third matrix of the last part. Shrink them to
two entries each so the arithmetic stays visible. Suppose a head attends
50/50 to two positions whose values are `[2, 0]` and `[0, 2]`: the average is
`[1, 1]`. Now suppose the values had been `[1, 1]` and `[1, 1]` — the average
is `[1, 1]` again.
Two different pasts, one identical summary. Whatever runs after the head sees
`[1, 1]` and cannot know which world it is in. The information was in hand,
and the averaging destroyed it — no training fixes that, because it is
arithmetic, not a bad habit.
An average can show where a head looked. It cannot keep separate what it
found.
Kept side by side instead — `[2, 0, 0, 2]` against `[1, 1, 1, 1]` — the two
worlds stay distinguishable. That is the whole design hint.
## Several heads, side by side
Run more than one complete head on the same input. Each gets its own `W_q`,
`W_k`, `W_v`, so each has its own idea of what to look for, what to offer, and
what to hand over. Nothing inside a head changes at all.
Then glue the answers side by side: the output's first columns are head one's
answer, the next columns head two's. Which-came-from-which survives as
position in the vector.
And that is the result — nothing combines the heads. No average, no picking
the larger; either would just be a different shredder. The glued vector is
the position's new description, and whatever reads it next takes the whole
width and *learns* how much weight each head's lane deserves. Combining is
deferred to a trainable layer instead of settled by a fixed rule.
## Specialists at the price of one generalist
Running four heads sounds like four times the machinery. It is not, because
the heads are made narrower to pay for their number.
Every position's description is a row of numbers — eight of them, in the lab
below — and that length is the model's **width**: the width every head reads,
and the width the glued answer must rebuild so it can stand where the old row
stood.
Two widths are in play now, so keep them apart. A head's answer is as long as
its value vectors — its *head width*, from the attention page. Cut each of
four heads down to head width 2 and each answers with two numbers per
position. Glued side by side, lengths add:
```
head 1 head 2 head 3 head 4
[a b] [c d] [e f] [g h]
↓ glued side by side ↓
[a b c d e f g h] ← eight: the model width again
```
Four twos make eight, so the glued answer is exactly the size of the row it
replaces.
Count the learned numbers. One full-width head is three 8×8 matrices — 192
numbers. Four quarter-width heads are twelve 8×2 matrices — 192 numbers. The
same budget, sliced into four independent rulebooks instead of one.
Multi-head attention adds nothing. It splits what was already there into
several narrower questions asked at once.
Each narrow head has less room — two numbers of answer instead of eight — so
the trade is depth of one view against number of views. Training decides how
each head spends its slice, and trained models really do split the work:
heads specialise, one tracking nearby grammar while another reaches back for
references.
## The axis the glue goes on
The glue is one call, and it has a wrong setting that does not crash. Glued on
the feature axis, four `(6, 2)` answers become `(6, 8)`: six positions, each
described by all four heads. Glued on the default axis they become `(24, 2)` —
the four answers stacked end to end as if they were twenty-four positions.
Nothing complains at the glue. It complains later, wherever a `(6, 8)` was
expected, which is the quiet kind of wrong worth having seen once. The lab
does it deliberately.
One more trap for later, when this is written with a library: PyTorch only
tracks parameters it can see, and heads kept in a plain Python list are
invisible to it — they would simply never train. `nn.ModuleList` exists for
exactly this.
## Gathering is not thinking
Attention — one head or many — only ever *collects*: every output is a
weighted average of value notes, laid side by side. Nothing yet chews on what
was collected. A position that has just learned it sits after `the ca` still
needs something to *conclude* `t` from that — and there is no piece in the
model that does it.
That piece, and the trick of stacking the whole arrangement into layers, is
the next part.
## Build it yourself
Six cells: the head packed into a function, the blend that shreds, two heads
disagreeing about the same input, the glue and its wrong axis, the 192 = 192
count, and the finished multi-head in three lines.