Every model before this one read its input the way a tape head reads a tape: one step at a time, dragging a summary behind it, forgetting. Attention threw that away. It lets every element look directly at every other, all at once, and learn — from nothing but its own mistakes — which of them were worth looking at. Here it learns to read a façade.
Here is an elevation: a row of bays, each one a form. An arch, a slot, a fin, a grid — the vocabulary a façade is built from.
Now hand the wall a brief: give me the arch. Something has to reach across the whole elevation, compare that brief against every bay at once, and hand back a form. Not the nearest bay. Not the last one drawn. The relevant one.
The arcs are how much each bay matters to the brief. What comes back on the right is not a copy pasted from the row — it is the row's forms, added together in proportion. Ask for a bay that's on the wall and the proportions collapse onto it, so the sum is that bay.
Every bay publishes a key: a short description of what it can answer. The brief becomes a query. Compare them with nothing more than a dot product and you have a score for each bay in the row.
Push the scores through a softmax and they become proportions that sum to one. The bay's value is its form — the actual silhouette. The answer is those forms, mixed by those proportions.
So attention never chooses. It blends. Drag the brief across the style space and watch the returned silhouette flow between bays, the way a design moves between references. That continuity is the only reason any of this can be learned: you cannot take a derivative through a choice.
Before the softmax runs, the scores are divided by a number — in a real transformer, the square root of the key dimension. It reads like housekeeping. It is the dial between a mood board and a plagiarism.
Divide by something large and every score flattens toward the same value. The wall returns the average of all its bays: a soft, symmetrical blob that is nobody's form, a design that referenced everything and committed to nothing.
Divide by something small and the largest score runs away with all the weight. The softmax hardens into a switch, one bay wins outright, and the answer is a straight copy — while the gradient to every other bay quietly dies.
Until now the keys were handed out. In a transformer they are not: the query and the key are each produced by a matrix that begins as noise and is learned.
The head below is training, right now, on one task: given a family, return that bay's form. Nothing in it knows what an arch is. Nothing points it at the matching bay. It has only the error in the silhouette it handed back.
Watch the returned form. At the first step attention is spread evenly and the answer really is the average of the whole wall — a soft, symmetrical nothing. Within a few hundred steps it has sharpened onto the bay that was asked for. The dark line is how much of its attention lands on the right one; the faint outline behind the form is the bay it should have handed back.
Your wall. Click a bay to change its family; drag its handle to push the form away from its type. Then brief the wall and watch where it looks.
Put two bays of the same family on the wall and the attention has no way to prefer one. It splits, and what comes back is the average of both forms — a bay that exists nowhere on your elevation, returned with complete confidence. That is not a bug in the code. It is what a lookup keyed only on type is, and it is precisely why real transformers give every token a position as well as an identity.