The loss landscape
You have climbed this mountain before. In Search it was called fitness and you went up; here it is called error and you go down. Everything a neural network will ever do — every coat it generates, every attention head that learns where to look — begins as a ball rolling downhill on a terrain made of its own mistakes.
A landscape made of error
A model so small you can see all of it: a line, y = a·x + b, trying to fit fourteen points. Two knobs, so every possible line is a position on a plane — and at every position we can measure how wrong that line is. That measurement, painted as height, is the loss landscape.
Drag the ball. You are the optimizer now. The inset shows the line your position describes; the valley floor is the best line that exists.
The gradient knows the slope
You found the valley by looking at the whole map. A network never sees the map — at any position it knows only the local slope: the gradient. Gradient descent is one rule repeated forever: step downhill, proportional to steepness.
The proportion is the learning rate, and it is the first hyperparameter you will ever tune. Too small and the ball crawls. Right, and it settles. Past a critical value — for this valley, exactly 1.0 — every step overshoots by more than it descended, and the ball is flung off the mountain. Add momentum and it coasts through shallow stretches: the same word, and the same physics, as the swarm's.
A neuron draws a line, a layer draws a fold
One neuron is embarrassingly simple: it draws a single straight boundary across the plane and shades the two sides. That is all it can ever say.
But stack a few and something happens: each contributes its own line, and the layer above combines the folds. Two neurons make a wedge, four make a pocket, eight make a region with genuine curvature. Expressiveness isn't in the neuron — it's in the folding. Scramble deals new random neurons; watch what one can say versus eight.
Watch a tiny network learn
Now put it all together and let it run: a two-layer network, eight neurons wide, descending its own loss landscape live — a landscape with a few thousand dimensions instead of two, but the same rule: step downhill.
The shaded field is the network's current opinion about every point on the plane. Drag the data and the opinion reorganizes under your finger. Try XOR — the pattern no single line can separate — and watch the folds arrive. The strip at the bottom is the ball rolling: loss, falling.
The learning studio
Yours now. Paint the data, choose the network, set the rate — and break it, twice, on purpose. Push the learning rate past the edge and watch the loss detonate: the ball flung off the mountain, live. Then the subtler failure: keep holdout on, add noise, and give a big network too few points. It will score perfectly on what it saw and fail on what it didn't.
That gap has a name — memorizing, not learning — and you have met it twice already: the swarm that converged and couldn't re-find the moved summit, the cascade that agreed and was wrong. Confidence and adaptability, one dial, one last time.
Everything after this page is elaboration. The VAE descending its reconstruction error, the diffusion model descending its denoising error, the attention head learning where to look — each is this same ball, rolling on a landscape with a million dimensions instead of two, moved by the same rule: step downhill, proportional to steepness, at a rate you must choose.