Transformer

How it is used

Used as a noun and before other nouns: a transformer model, transformer layers, a transformer-based system. Engineers talk about encoder-only, decoder-only and encoder-decoder transformers. They are used for images and audio too, not only text.

Example sentences

1. We replaced the old recurrent model with a small transformer and cut errors by a third.

2. Most current LLMs are decoder-only transformers.

Common collocations

• transformer model
• transformer architecture
• decoder-only transformer
• transformer-based

Related terms

• attentionA mechanism that lets a model decide which other parts of its input are most relevant to each part, and give those parts more weight when it builds its output.
• LLMShort for large language model: a very large neural network trained on huge amounts of text to predict the next token, which lets it answer questions, write, summarize and code.
• layerOne level of a neural network: a group of units that takes the output of the previous level, transforms it and passes the result on to the next one.
• neural networkA model made of many connected units arranged in layers, loosely inspired by the brain. Each connection has a weight that is adjusted during training.

This word in other fields

The same English word can have a different meaning in another field:

• transformer · Software Engineering
• transformer · Electrical & Electronics Engineering
• transformer · Energy & Renewables

More terms: Neural Networks and Deep Learning

• deep learning
• parameter
• weights
• activation function
• backpropagation