Quantization

How it is used

Used to run large models on fewer or smaller GPUs, on laptops and on phones. There is usually a small loss in quality, which teams measure before shipping. People say a model is quantized to 4 bits and name formats such as INT8.

Example sentences

1. After 4-bit quantization the model fits on a single GPU, and accuracy dropped by less than a point.

2. Quantization made inference almost twice as fast on our servers.

Common collocations

• 4-bit quantization
• post-training quantization
• quantized model
• quantization error

Related terms

• inferenceThe stage in which a trained model is used to produce outputs for new inputs, for example answering a user's question or classifying a new photo.
• GPUShort for graphics processing unit: the chip that renders a game's images by running shaders and drawing millions of pixels every frame. The CPU handles most of the game logic.
• knowledge distillationTraining a smaller 'student' model to copy the outputs of a larger 'teacher' model, so that it keeps much of the quality while being faster and cheaper to run.
• weightsThe learned numbers on the connections of a neural network, which decide how strongly each input affects the next layer. Together they hold what the model has learned.
• LoRAShort for low-rank adaptation: a fine-tuning method that keeps the original weights frozen and trains only small added matrices, which saves a lot of memory and storage.

This word in other fields

The same English word can have a different meaning in another field:

• quantization · Artificial Intelligence & Machine Learning

More terms: Machine Learning and Data Processing

• artificial intelligence
• machine learning
• model
• dataset
• training data
• label
• train
• supervised learning
• neural network
• deep learning
• loss function
• overfitting