Attention

How it is used

The core idea of the transformer. Engineers talk about self-attention, attention heads, attention layers and attention maps. Because its cost grows quickly with input length, a lot of research aims to make attention faster.

Example sentences

1. The attention maps suggest the model focuses on the drug name when it answers.

2. Each layer has 16 attention heads.

Common collocations

• self-attention
• attention head
• attention layer
• attention mechanism
• attention map

Related terms

• transformerA neural network architecture, introduced in 2017, that uses attention to process all parts of a sequence in parallel. It is the basis of today's LLMs.
• layerOne level of a neural network: a group of units that takes the output of the previous level, transforms it and passes the result on to the next one.
• context windowThe maximum amount of text, measured in tokens, that a language model can take into account at once, including the prompt, any documents and its own answer.
• neural networkA model made of many connected units arranged in layers, loosely inspired by the brain. Each connection has a weight that is adjusted during training.

This word in other fields

The same English word can have a different meaning in another field:

• attention · Software Engineering
• attention · Psychology

More terms: Neural Networks and Deep Learning

• deep learning
• parameter
• weights
• activation function
• backpropagation