Also known as: model distillation (knowledge distillation, teacher-student models)
Model distillation is a technique in which a smaller or more efficient model is trained to approximate the behavior of a larger or more complex model. It is used to reduce inference cost, latency, or deployment complexity while preserving useful performance.
The process of reducing the size of one model (known as the teacher ) into a smaller model (known as the student) that emulates the original model's predictions as faithfully as possible. Distillation is useful because the smaller model has two key benefits over the larger model (the teacher): - Faster inference time - Reduced memory and energy usage However, the student's predictions are typically not as good as the teacher's predictions. Distillation trains the student model to minimize a loss function based on the difference between the outputs of the predictions of the student and teacher models. Compare and contrast distillation with the following terms: - fine-tuning - prompt-based learning See LLMs: Fine-tuning, distillation, and prompt engineering in Machine Learning Crash Course for more information.
Model distillation is a technique that allows a smaller student model to learn from a larger teacher model. The student model is trained to mimic the output of the teacher model, and it can then be used to generate new data or make inferences. Model distillation is often used to make large models more efficient or to make them more accessible to devices with limited resources. It can also be used to improve the generalization of models by reducing overfitting.