Model parallelism is a method for distributing parts of a model across multiple devices when the model is too large or computationally intensive for one device. It differs from data parallelism, which replicates the model and splits the data.
A way of scaling training or inference that puts different parts of one model on different devices. Model parallelism enables models that are too big to fit on a single device. To implement model parallelism, a system typically does the following: 1. Shards (divides) the model into smaller parts. 2. Distributes the training of those smaller parts across multiple processors. Each processor trains its own part of the model. 3. Combines the results to create a single model. Model parallelism slows training. See also data parallelism.