Interactive notebook
Gradient Descent
Why the learning rate is the first knob you touch — and the first one that ruins a run.
A convex quadratic — the easy case. Every path ends at the same minimum.
- SGD
- Momentum
- Adam
step 0loss 12.8640‖∇‖ 7.7331position (-3.20, 2.20)
Click anywhere on the surface to start from there. Then try nudging the learning rate up until it breaks.
Click the surface or focus it and use the arrow keys to move the start. Colors run cream (low loss) to deep rust (high). The bands are contour lines — where they bunch together, the surface is steep. Every path here follows the true analytic gradient, not a finite-difference approximation.