Pipelining is a form of model parallelism that splits model processing into sequential stages across different devices. It improves throughput by allowing different stages to process different batches at the same time.
A form of model parallelism in which a model's processing is divided into consecutive stages and each stage is executed on a different device. While a stage is processing one batch, the preceding stage can work on the next batch. See also staged training.