Параллелизм данных — это scaling technique, при которой одна и та же модель реплицируется на нескольких devices, а разные подмножества input data обрабатываются параллельно. Он может ускорять training или inference и поддерживать larger batch sizes при условии, что модель помещается на каждом device.
A way of scaling training or inference that replicates an entire model onto multiple devices and then passes a subset of the input data to each device. Data parallelism can enable training and inference on very large batch sizes; however, data parallelism requires that the model be small enough to fit on all devices. Data parallelism typically speeds training and inference. See also model parallelism.
A way of scaling training or inference that replicates an entire model onto multiple devices and then passes a subset of the input data to each device. Data parallelism can enable training and inference on very large batch sizes; however, data parallelism requires that the model be small enough to fit on all devices. Data parallelism typically speeds training and inference.