Also known as: Pre training · Pre Training
Pre-training — это начальная фаза обучения, в которой модель изучает общие закономерности на крупном наборе данных. Полученная модель затем может быть адаптирована с помощью fine-tuning, instruction tuning, prompt tuning, distillation или параметро-эффективных методов.
The initial training of a model on a large dataset. Some pre-trained models are clumsy giants and must typically be refined through additional training. For example, ML experts might pre-train a large language model on a vast text dataset, such as all the English pages in Wikipedia. Following pre-training, the resulting model might be further refined through any of the following techniques: - distillation - fine-tuning - instruction tuning - parameter-efficient tuning - prompt-tuning