Also known as: Pre training Β· Pre Training
Pre-training is the initial training phase in which a model learns general patterns from a large dataset. The resulting model may later be adapted through fine-tuning, instruction tuning, prompt tuning, distillation, or parameter-efficient methods.
The initial training of a model on a large dataset. Some pre-trained models are clumsy giants and must typically be refined through additional training. For example, ML experts might pre-train a large language model on a vast text dataset, such as all the English pages in Wikipedia. Following pre-training, the resulting model might be further refined through any of the following techniques: - distillation - fine-tuning - instruction tuning - parameter-efficient tuning - prompt-tuning