Momentum is an optimization technique that accelerates gradient-based learning by accumulating part of past update directions. It helps smooth noisy gradients, speed progress along consistent directions, and reduce oscillation during training.
A sophisticated gradient descent algorithm in which a learning step depends not only on the derivative in the current step, but also on the derivatives of the step(s) that immediately preceded it. Momentum involves computing an exponentially weighted moving average of the gradients over time, analogous to momentum in physics. Momentum sometimes prevents learning from getting stuck in local minima.