Okuzeka
Artificial Intelligence & Machine Learning vocabulary · Natural Language Processing

Tokenizer

Meaning in artificial intelligence & machine learning, with examples

/ˈtoʊkənaɪzər/nounCommon term

In artificial intelligence & machine learning, tokenizer means: The component that splits text into tokens and turns them into numbers a model can read, and that turns the model's output numbers back into text.

In Turkish: tokenizer, token ayırıcı

How it is used

Every model comes with its own tokenizer, and using the wrong one breaks the results. Engineers look at a tokenizer's vocabulary size and how efficiently it handles a language; some split Turkish words into many small pieces. The process is called tokenization.

Example sentences

  1. This tokenizer splits long Turkish words into many pieces, so our prompts cost more.

  2. Make sure you load the tokenizer from the same checkpoint as the model.

Common collocations

  • load the tokenizer
  • tokenizer vocabulary
  • subword tokenizer
  • tokenization
  • tokenA small unit of text that a language model reads and writes, often a whole word, part of a word or a punctuation mark. Model limits and prices are counted in tokens.
  • preprocessingThe steps that clean and prepare raw data before a model uses it, such as removing duplicates, filling in missing values, resizing images or normalizing text.
  • natural language processingThe area of AI that deals with human language, building systems that read, understand, translate, summarize or generate text and speech.
  • embeddingA list of numbers (a vector) that represents a word, sentence, image or other item in such a way that items with similar meanings end up close to each other.

This word in other fields

The same English word can have a different meaning in another field:

More terms: Natural Language Processing

Practise with graded reading

Reading at the right level is the fastest way to make new words stick. Pick free English books by level: