Also known as: model distillation (knowledge distillation, teacher-student models)
Дистилляция модели — техника, при которой меньшая или более эффективная модель обучается приближать поведение более крупной или сложной модели. Она используется для снижения стоимости inference, latency или сложности deployment при сохранении полезной производительности.
The process of reducing the size of one model (known as the teacher ) into a smaller model (known as the student) that emulates the original model's predictions as faithfully as possible. Distillation is useful because the smaller model has two key benefits over the larger model (the teacher): - Faster inference time - Reduced memory and energy usage However, the student's predictions are typically not as good as the teacher's predictions. Distillation trains the student model to minimize a loss function based on the difference between the outputs of the predictions of the student and teacher models. Compare and contrast distillation with the following terms: - fine-tuning - prompt-based learning See LLMs: Fine-tuning, distillation, and prompt engineering in Machine Learning Crash Course for more information.
Model distillation is a technique that allows a smaller student model to learn from a larger teacher model. The student model is trained to mimic the output of the teacher model, and it can then be used to generate new data or make inferences. Model distillation is often used to make large models more efficient or to make them more accessible to devices with limited resources. It can also be used to improve the generalization of models by reducing overfitting.