Knowledge distillation

How it is used

Often just called distillation; the verb is distill, as in 'we distilled the 70B model into a 3B model'. Used to cut inference costs or to run models on phones. Licenses matter here, because some model licenses forbid using outputs to train other models.

Example sentences

1. Knowledge distillation gave us a model ten times smaller with only a 2% drop in accuracy.

2. We used knowledge distillation to build an on-device version of the classifier.

Common collocations

• teacher model
• student model
• distill a model
• distillation loss

Related terms

• quantizationStoring a model's weights, and sometimes its activations, with fewer bits, for example 8-bit or 4-bit numbers instead of 16-bit, to make it smaller and faster.
• fine-tuningExtra training of an already pretrained model on a smaller, task-specific dataset, so that it does better on that task or follows a particular style.
• inferenceThe stage in which a trained model is used to produce outputs for new inputs, for example answering a user's question or classifying a new photo.
• modelThe result of training: a program with learned parameters that takes an input, such as an image or a sentence, and produces an output, such as a label, a number or a text.

This word in other fields

The same English word can have a different meaning in another field:

• knowledge distillation · Software Engineering

More terms: Fine-Tuning and Adaptation

• transfer learning
• base model
• LoRA
• RLHF