Okuzeka
Software Engineering vocabulary · Machine Learning and Data Processing

Pretraining

Meaning in software engineering, with examples

/ˌpriːˈtreɪnɪŋ/nounCommon term

In software engineering, pretraining means: The first and largest training stage, in which a model learns general patterns from a huge amount of mostly unlabeled data before it is adapted to specific tasks.

In Turkish: ön eğitim

How it is used

For LLMs, pretraining usually means predicting the next token on trillions of tokens of text, and it is by far the most expensive stage. It is followed by post-training steps such as fine-tuning and RLHF. Also written pre-training; the adjective is pretrained.

Example sentences

  1. Pretraining took eight weeks on a few thousand GPUs.

  2. We don't do pretraining ourselves; we fine-tune open models.

Common collocations

  • pretraining data
  • pretraining run
  • large-scale pretraining
  • pretrained model
  • fine-tuningExtra training of an already pretrained model on a smaller, task-specific dataset, so that it does better on that task or follows a particular style.
  • base modelA model that has only been pretrained, before any fine-tuning or instruction training. It continues text well but does not reliably follow instructions or chat.
  • LLMShort for large language model: a very large neural network trained on huge amounts of text to predict the next token, which lets it answer questions, write, summarize and code.
  • transfer learningReusing what a model learned on one task or dataset as the starting point for a different but related task, instead of training a new model from scratch.

This word in other fields

The same English word can have a different meaning in another field:

More terms: Machine Learning and Data Processing

Practise with graded reading

Reading at the right level is the fastest way to make new words stick. Pick free English books by level: