The transformer block

Keep one picture in mind for this whole part. The model’s working material is one row of numbers per position — that position’s description. Attention just rewrote every row as a blend of the rows it chose to read; that was the gathering. From here on nothing looks at two positions at once. Everything below happens to each row on its own: it arrives holding gathered evidence, and it has to leave holding more than evidence.

Because gathering is not thinking. A position that ends the attention layer holding “an a, shortly after cat and sat” has collected the facts and done nothing with them. A model that only collects is a filing cabinet.

This part builds the piece that thinks, finds out why thinking once is not enough, and makes the answer to that — stacking — survivable.

One if at a time

Thinking, at its smallest, is a rule: if the evidence looks like this, then conclude that. The thinking piece is built from exactly that shape.

One unit owns two rows of weights — the model’s own numbers, shaped by training like every matrix so far:

• a pattern: the evidence it looks for,
• a contribution: what it adds to the position’s description when it finds it.

How well the evidence matches the pattern is their dot product — the same measuring stick attention scored with. The match is then clipped at zero:

fires = max(pattern · evidence, 0)

The clip is the bend (its standard name is ReLU): plotted, a straight line bent flat at zero. A poor match does not fire a little — it does not fire at all, and the unit stays out of the way. With pattern [1, −1]:

evidence	pattern · evidence	after the bend
[0.8, −0.6] — the pattern	+1.4	1.4
[−0.8, 0.6] — its opposite	−1.4	0
[0.5, 0.5] — unrelated	0.0	0

And whatever fires, the unit acts on: it adds fires × contribution to the description. An if, followed by its then. Here are three units under your hand — the first one is the table’s:

A bank of ifs

One rule is not thinking. The feed-forward layer is 128 of these units — four times the model’s width, by convention — all reading the same evidence, each with its own pattern and its own contribution. Stack the 128 patterns as the columns of one matrix, W1, and the 128 contributions as the rows of another, W2, and the whole bank runs in one line:

The widget’s three units, written that way:

W1 = [[ 1.0,  0.0, -1.0],        W2 = [[ 1.0,  0.5],
      [-1.0,  1.0, -1.0]]              [-0.5,  1.0],
                                       [ 0.5, -1.0]]

Walk the table’s evidence through. [0.8, -0.6] @ W1 asks all three ifs at once: [1.4, -0.6, -0.2]. The bend silences the two poor matches, [1.4, 0, 0], and @ W2 sums what fired: 1.4 * [1.0, 0.5] = [1.4, 0.7] — the widget’s own output. At full size the only change is the count:

def feed_forward(z):
    return np.maximum(z @ W1, 0) @ W2   # every if, then every fired then

The bend is what makes the bank worth having. Delete it and every “if” becomes “always” — and 128 always-on contributions collapse into a single matrix: (x @ W1) @ W2 equals x @ (W1 @ W2), checked to the digit in the lab. However many linear layers you stack, they are one matrix wearing costumes. One kink breaks the collapse, and depth starts buying something.

No position sees another anywhere in this — mixing is attention’s job, exclusively:

Why once is not enough

One round of gather-and-think lets every position collect facts and draw a conclusion. What it cannot do is collect anyone else’s conclusions — they did not exist until the round was over.

Take the sentence from the attention part: the cat sat on the mat because it was soft. In one round, the position holding it can gather cat and mat and think its way to a suspicion about which one is meant. Acting on that suspicion — going back for what was said about the mat — needs a second round of gathering, because only now is there a suspicion to act on.

So the gather-and-think pair gets stacked, conclusions building on conclusions, one round per storey. And stacking is where the trouble starts.

Add, never replace

A learned layer has no reason to come out exactly signal-preserving. Suppose each of twenty-four storeys is merely a little timid — passing on about 0.7 of what it receives. If every storey replaces what it was given, the signal arrives at size 0.0001, because 0.7²⁴ is 0.000192. Nothing survives; one dud storey anywhere would have the same effect.

So a block never replaces. It adds what it computed to what it received:

z = z + feed_forward(z)

A notebook passed down a line of people, where each may add a margin note and none may erase. A useless note — all zeros — costs nothing, and the original is still there at the bottom of the stack. The same lane matters in reverse: the slope flowing back during training always has the untouched z path, however unhelpful the notes were.

Tidy after every addition

Adding has its own failure: the numbers only ever accumulate. The same twenty-four storeys, adding instead of replacing, deliver a signal of size 108 — nothing lost, but far from any scale the next storey expects. Same family of trouble as the √width divisor in attention.

The repair is to rewrite each position to a standard size after every addition: subtract its mean, divide by its spread. Mean 0, spread 1, same information — this is layer norm, and it works on each position’s own numbers only, never across positions.

The block

All of it, in the order tidy → gather → add, tidy → think → add:

def block(z):
    z = z + multi_head(layer_norm(z), heads)
    z = z + feed_forward(layer_norm(z))
    return z

Shape in equals shape out — the lab runs three blocks deep on the same input and the shape never moves. That property is the entire trick: anything that keeps its shape can be stacked as deep as the budget allows, and the depth of a model is just how many of these sit in a pile.

What the stack still cannot know

A row can now cross twenty-four storeys and arrive intact, gathered and thought about at every one. But nothing in any of it — not the heads, not the ifs — ever asks where a character sits. The next part proves that blindness with one experiment, fixes it with one table, and assembles the whole model.

Build it yourself

Six cells: the gathering packed into functions, one if-unit and then the bank of them, the collapse the bend prevents, the stack that dies by replacing, the drift that tidying holds at 1.00, and the block — three deep, shape unchanged.