Inference

How it is used

Contrasted with training. Engineers run inference, speed it up and talk about inference cost, inference servers and batch versus real-time inference. It comes up constantly in infrastructure and budget discussions.

Example sentences

1. Now that the app has a million users, inference costs us more than training.

2. We run inference on CPUs at night and on GPUs during peak hours.

Common collocations

• run inference
• inference cost
• inference speed
• real-time inference
• batch inference

Related terms

• deployTo put a trained model into a real system so that users or other software can send it requests and receive predictions.
• latencyThe time between sending a request to a model and receiving its response, usually measured in milliseconds or seconds.
• GPUShort for graphics processing unit: a chip that does thousands of simple calculations in parallel, which makes it the main hardware for training and running neural networks.
• quantizationStoring a model's weights, and sometimes its activations, with fewer bits, for example 8-bit or 4-bit numbers instead of 16-bit, to make it smaller and faster.
• endpointA URL where a deployed model can be reached over a network: other software sends a request to it and receives the model's prediction or answer.

This word in other fields

The same English word can have a different meaning in another field:

• inference · Software Engineering
• inference · Philosophy

More terms: Deployment and MLOps

• MLOps
• data drift
• roll out