AI glossary · Fundamentals
Tokenization
In one sentence
Tokenization is the process of splitting text into tokens, the pieces a language model understands, and mapping each one to a number in its vocabulary. It's the first step before a model can read or write anything.
Also known as: tokenisation · tokenizer · text tokenization · subword tokenization
What is tokenization?
A language model only works with numbers. Tokenization is the bridge between your text and those numbers: it chops the sentence into pieces, the tokens, and swaps each piece for its number in a fixed vocabulary. When the model replies, the process runs in reverse.
Put it to work
UTIO's AI, through an API
From €29/mo + VAT
For developers and agencies building their own products.
See the AI APIThe tool that does this is called a tokenizer. It isn’t something you can swap at will: each model is trained with a specific tokenizer and depends on it. If the tokenizer splits a language badly, the model spends more tokens saying the same thing and may perform worse in that language.
Related to this term
AI API
- Transcription, voice, text, image, OCR and embeddings
- Per-client keys with a spending cap
- Prepaid wallet: €50 minimum top-up
From €29/mo + VAT Panel account, plus usage
See the AI APIHow it works
Modern tokenizers work with subwords: not single letters, not whole words, but pieces in between. That gives two advantages: a manageable vocabulary and the ability to write any word, even one never seen before, by joining pieces together.
The most common method is BPE (byte pair encoding):
- Start from individual characters or bytes.
- Find the pair of pieces that most often appear side by side in a large body of text and merge them into a new piece.
- Repeat until the vocabulary reaches the chosen size, typically tens or hundreds of thousands of pieces.
There are variants such as WordPiece and Unigram, often implemented with the SentencePiece library. They all aim for the same thing: common sequences should take up few pieces.
In practice, a common word like “thanks” may be a single token, while “dishwasher-safe” or an unusual surname gets split into several.
Example
An online bike parts shop in Sheffield uses a model to answer customer questions. Its product pages are full of technical codes such as “Shimano BR-M6100” or “11-51T cassette”. The model’s tokenizer doesn’t know those combinations and breaks them into lots of small pieces.
The result: the pages use more tokens than their length suggests, and the model sometimes mixes up similar codes. The shop fixes this by looking up codes exactly in its catalogue and passing the model only the right product page, using RAG, instead of hoping it will “remember” them.
Tokenization at UTIO
On UTIO’s plans you don’t need to think about tokenization: agents are sold by minutes or conversations, not by tokens.
If you’re building with the AI API, tokenization happens inside each capability. For instance, the embeddings endpoint turns your text into vectors for searching your content; it tokenizes the text before calculating them. Per-unit rates by model tier are published in the panel. Knowing how text gets split helps you prepare the documents your AI agent will work with.
Frequently asked questions
Do all models tokenize text the same way?
No. Each model family has its own tokenizer and vocabulary. The same text can produce a different number of tokens in two models, so token counts and per-token prices can't be compared directly across providers.
What is BPE?
BPE (byte pair encoding) is one of the most widely used tokenization methods. It starts from single characters and keeps merging the pairs that occur together most often in the training text, until it has a vocabulary of common pieces: syllables, word stems, whole words.
Is tokenization in AI the same as tokenization in payments?
No. In payments, tokenization replaces sensitive data, such as a card number, with a code that's worthless to an attacker. In AI it means splitting text into pieces for a model. Same word, different job.
Related terms
- Token
A token is the chunk of text a language model works with: a word, part of a word or a punctuation mark. Models read, count and bill in tokens, not in words or characters.
- LLM
An LLM (large language model) is an AI model trained on vast amounts of text to predict the next word. That single skill lets it draft, summarise, translate, classify and hold a conversation.
- Embeddings
An embedding is a list of numbers that represents the meaning of a piece of text, an image or audio. Items with similar meaning have embeddings that sit close together, which lets you search by meaning rather than by exact words.
- Context window
A context window is the maximum amount of text, measured in tokens, that a language model can take into account at once: instructions, conversation, documents and its own reply. Anything outside it, the model simply doesn't see.
- Natural language processing (NLP)
Natural language processing (NLP) is the branch of artificial intelligence that lets machines work with human language: read, analyse, classify, translate and produce text or speech the way people use it.
Updated on