The vanishing gradient problem occurs when gradients in early layers of a deep neural network become very small during training. As a result, those layers learn slowly or not at all, making deep or recurrent networks difficult to train.
The tendency for the gradients of early hidden layers of some deep neural networks to become surprisingly flat (low). Increasingly lower gradients result in increasingly smaller changes to the weights on nodes in a deep neural network, leading to little or no learning. Models suffering from the vanishing gradient problem become difficult or impossible to train. Long Short-Term Memory cells address this issue. Compare to exploding gradient problem.