Накопление градиента — техника обучения, при которой градиенты вычисляются на нескольких mini-batches перед обновлением параметров модели. Она может имитировать больший размер batch, когда ограничения памяти не позволяют обработать полный batch за один раз.
A backpropagation technique that updates the parameters only once per epoch rather than once per iteration. After processing each mini-batch, gradient accumulation simply updates a running total of gradients. Then, after processing the last mini-batch in the epoch, the system finally updates the parameters based on the total of all gradient changes. Gradient accumulation is useful when the batch size is very large compared to the amount of available memory for training. When memory is an issue, the natural tendency is to reduce batch size. However, reducing the batch size in normal backpropagation increases the number of parameter updates. Gradient accumulation enables the model to avoid memory issues but still train efficiently.