Knowledge distillation is a model compression and transfer technique in which a smaller or simpler model learns to reproduce the behavior of a larger or more complex model. It is used to improve efficiency, reduce deployment cost, or transfer capabilities while preserving useful performance.
The process of transferring knowledge from a large machine learning model to a smaller one.