Interactive notebook
Learning-Rate Schedules
Big steps early, small steps late: why schedules beat any single learning rate.
Learning rate over training
Loss reached
Descent paths (same surface, same start)
Ease smoothly from the full rate down to nearly zero. No cliff-edge transitions, and it lands softly.
- Constant
- Step decay
- Cosine anneal
- Warmup + cosine
All four runs use plain SGD on the same ravine from the same starting point, with the same number of steps. The only difference is how the learning rate changes along the way. Push the peak rate up and watch which schedules survive it — that robustness is the real argument for warmup.