Skip to content
Tutorials ▸

A tiny GPT, built from scratch

Start by counting which letter follows which, and end with a transformer. Every piece hand-written before any library version is allowed near it.

A real bigram table: for each of the sixteen commonest characters in part one, how often every other character followed it. Nothing learned this — it is a tally. Part one builds it from any text you type.

By the end: You will have built a character-level language model a piece at a time, and will know why each piece had to exist — because you will have watched the version without it fail.

Start with part 01

The 12 parts

  1. 01 The problem Every language model does one thing: guess the next symbol, then do it again. What the task is, how far the big models take it, and what a small one is genuinely good for. next-token prediction · autoregressive generation · model scale
  2. 02 Counting pairs A language model with no parameters, no training and no neural network — built by tallying which character follows which, and generating from it. language model · distribution · sampling · bigram
  3. 03 Turning characters into numbers A neural network cannot read a letter. Two small lookup tables turn text into numbers and back, and getting them wrong corrupts everything downstream quietly. tokenization · vocabulary · encode and decode · round-trip check
  4. 04 Context windows, and where the answers come from Everything from here on has to be trained, and training needs a dataset. In the other tutorial you made one by hand, a drawing at a time. Here there is nobody to do that — so the dataset has to come out of the text itself. training data · context window · block size · shift-by-one targets · batching
  5. 05 Embeddings, softmax and the loss A network can only multiply and add. Hand it the number 9 for “t” and it works out that t is nine times a, which is nonsense. So each character gets a whole row of numbers instead, and training corrects them. embedding table · logits · softmax over a batch · cross-entropy
  6. 06 The training loop The model is 169 numbers and a loss saying how bad they are. Training is turning each number the right way, a little at a time — and the rule for which way turns out to be readable straight off the percentages. gradient · gradient descent · training loop · mini-batch noise
  7. 07 Self-attention The model has a window full of characters and reads only the last one. Letting it read all of them is easy; letting it decide which ones were worth reading is the idea the whole architecture is built on. self-attention · query key value · causal masking · scaled dot product
  8. 08 Many heads at once A head asks one question of the past and answers with one weighted average — and averaging shreds: two different pasts can leave identical blends. So run several heads at once and lay their answers side by side. multi-head attention · concatenation · head size · parameter budget
  9. 09 The transformer block Attention gathers and never thinks. Thinking turns out to be a bank of if-then rules you can watch fire — and once one round is visibly not enough, stacking becomes a survival problem with a two-part fix. feed-forward · residual connections · layer norm · transformer block
  10. 10 Assembling the model The stack of blocks treats its window as a bag — shuffle everything before the last character and it concludes the same thing. One more table fixes that, one more matrix turns descriptions back into scores, and the model is whole: 12,800 numbers, every one already met. positional embeddings · lm head · model assembly · parameter count
  11. 11 Training it properly The model trains with the same loop as ever. What "properly" adds is a number that cannot flatter — a held-back slice the model never trains on — and enough text that the score on it means something. validation split · overfitting · early stopping · gradient checking
  12. 12 Generating text The weights are fourteen thousand trained numbers that have never said a word. The loop that makes them speak is the counting model's loop with one upgrade, one dial — and one trap that looks like a sensible policy. generation loop · temperature · greedy decoding · sampling