Okuzeka
Artificial Intelligence & Machine Learning vocabulary · Large Language Models

Token

Meaning in artificial intelligence & machine learning, with examples

/ˈtoʊkən/nounCore term

In artificial intelligence & machine learning, token means: A small unit of text that a language model reads and writes, often a whole word, part of a word or a punctuation mark. Model limits and prices are counted in tokens.

In Turkish: token, belirteç

How it is used

Used constantly in LLM work: input and output tokens, tokens per second, cost per million tokens, maximum tokens. In English text a token is on average a bit shorter than a word; Turkish, with its long words, usually needs more tokens per word.

Example sentences

  1. The prompt is about 3,000 tokens, so it fits easily in the context window.

  2. With most providers, output tokens cost more than input tokens.

Common collocations

  • input tokens
  • output tokens
  • tokens per second
  • token limit
  • cost per token
  • tokenizerThe component that splits text into tokens and turns them into numbers a model can read, and that turns the model's output numbers back into text.
  • context windowThe maximum amount of text, measured in tokens, that a language model can take into account at once, including the prompt, any documents and its own answer.
  • LLMShort for large language model: a very large neural network trained on huge amounts of text to predict the next token, which lets it answer questions, write, summarize and code.
  • embeddingA list of numbers (a vector) that represents a word, sentence, image or other item in such a way that items with similar meanings end up close to each other.

This word in other fields

The same English word can have a different meaning in another field:

More terms: Large Language Models

Practise with graded reading

Reading at the right level is the fastest way to make new words stick. Pick free English books by level: