The Bellman equation is a recursive relationship used in dynamic programming and reinforcement learning to express the value of a state or action in terms of immediate reward and expected future value. It provides the foundation for methods such as value iteration and Q-learning. By decomposing long-term optimization into smaller subproblems, it supports decision-making over sequences of actions.
In reinforcement learning, the following identity satisfied by the optimal Q-function: \\\\\\[Q(s, a) = r(s, a) + \\\\\\\E\_\s'\\|s,a\ \\\_\a'\ Q(s', a')\\\\\\] Reinforcement learning algorithms apply this identity to create Q-learning using the following update rule: \\\\\\[Q(s,a) \\\Q(s,a) + \\\ \\\\\[r(s,a) + \\\\\\\\\_\\\\\a_1\\ Q(s',a') - Q(s,a) \\\\\] \\\\\\] Beyond reinforcement learning, the Bellman equation has applications to dynamic programming. See the Wikipedia entry for Bellman equation.