Shard — logical partition of data, model parameters or training examples. Sharding distributes work across machines or devices and supports data parallelism, model parallelism and scalable training or storage.
A logical division of the training set or the model. Typically, some process creates shards by dividing the examples or parameters into (usually) equal-sized chunks. Each shard is then assigned to a different machine. Sharding a model is called model parallelism; sharding data is called data parallelism.