Latency

How it is used

Teams track average latency and tail latency such as p95 or p99. For LLMs they also measure time to first token. Latency is often traded off against throughput, cost and model quality.

Example sentences

1. The bigger model is more accurate, but latency went from 300 milliseconds to two seconds.

2. Our p99 latency spikes every time the cache is cleared.

Common collocations

• low latency
• p99 latency
• reduce latency
• latency budget
• time to first token

Related terms

• inferenceThe stage in which a trained model is used to produce outputs for new inputs, for example answering a user's question or classifying a new photo.
• endpointA URL where a deployed model can be reached over a network: other software sends a request to it and receives the model's prediction or answer.
• quantizationStoring a model's weights, and sometimes its activations, with fewer bits, for example 8-bit or 4-bit numbers instead of 16-bit, to make it smaller and faster.
• GPUShort for graphics processing unit: a chip that does thousands of simple calculations in parallel, which makes it the main hardware for training and running neural networks.

This word in other fields

The same English word can have a different meaning in another field:

• latency · Software Engineering
• latency · Cloud Computing & DevOps
• latency · Networking & Telecommunications
• latency · Game Development

More terms: Deployment and MLOps

• deploy
• MLOps
• data drift
• roll out