Okuzeka
Software Engineering vocabulary · Machine Learning and Data Processing

Knowledge distillation

Meaning in software engineering, with examples

nounAdvanced term

In software engineering, knowledge distillation means: Training a smaller 'student' model to copy the outputs of a larger 'teacher' model, so that it keeps much of the quality while being faster and cheaper to run.

In Turkish: bilgi damıtma, model damıtma

How it is used

Often just called distillation; the verb is distill, as in 'we distilled the 70B model into a 3B model'. Used to cut inference costs or to run models on phones. Licenses matter here, because some model licenses forbid using outputs to train other models.

Example sentences

  1. Knowledge distillation gave us a model ten times smaller with only a 2% drop in accuracy.

  2. We used knowledge distillation to build an on-device version of the classifier.

Common collocations

  • teacher model
  • student model
  • distill a model
  • distillation loss
  • quantizationStoring a model's weights, and sometimes its activations, with fewer bits, for example 8-bit or 4-bit numbers instead of 16-bit, to make it smaller and faster.
  • fine-tuningExtra training of an already pretrained model on a smaller, task-specific dataset, so that it does better on that task or follows a particular style.
  • inferenceThe stage in which a trained model is used to produce outputs for new inputs, for example answering a user's question or classifying a new photo.
  • modelThe result of training: a program with learned parameters that takes an input, such as an image or a sentence, and produces an output, such as a label, a number or a text.

This word in other fields

The same English word can have a different meaning in another field:

More terms: Machine Learning and Data Processing

Practise with graded reading

Reading at the right level is the fastest way to make new words stick. Pick free English books by level: