Interactive notebook
Q-Learning Gridworld
How a value table turns random flailing into a policy, one reward at a time.
Those two numbers measure different things: the first includes the random moves ε forces the agent to make, so it stays low by design. The second switches exploration off and asks what the agent has actually learned.
Right now the agent knows nothing — no map, no rules, not even which way the goal is. Press Run.
The agent only ever receives a number: +1 at the flag, −1 in a pit, and a small penalty for every step (which is why it learns to hurry rather than wander). Each cell’s shading is V(s) = max Q(s,a), its estimate of how good it is to stand there. Notice the value spreading backwards from the goal — that’s the discount γ propagating reward one cell at a time, and it’s the whole mechanism.
Set ε = 0 and reset: with no exploration the agent commits to the first mediocre thing it finds and never discovers better. Set ε = 1 and it never exploits what it learned. That tension is the exploration/exploitation tradeoff, and there is no setting that escapes it.