UTIO
☎️ Try the demoSign up

AI glossary · Fundamentals

Token

In one sentence

A token is the chunk of text a language model works with: a word, part of a word or a punctuation mark. Models read, count and bill in tokens, not in words or characters.

Also known as: AI token · LLM token · text token

How a sentence is split into tokens “Hi, what time do you open tomorrow?”. The sentence is split into 9 tokens: “Hi” (id 13347), “,” (id 11), “what” (id 1412), “time” (id 892), “do” (id 656), “you” (id 499), “open” (id 1825), “tomorrow” (id 16986), “?” (id 30). Common words take one token each and punctuation takes its own. The ID numbers are illustrative: every model has its own vocabulary. “Hi, what time do you open tomorrow?” Hi 13347 , 11 what 1412 time 892 do 656 you 499 open 1825 tomorrow 16986 ? 30 7 words → 9 tokens
Each token has a number (ID) in the model's vocabulary; the ones shown are illustrative. In English, a token averages roughly three quarters of a word; languages such as Spanish or French usually need a few more tokens for the same text.

What is a token?

A language model doesn’t read the way you do. Before it processes a sentence, a tool called a tokenizer splits it into pieces. Each piece is a token with its own number in the model’s vocabulary. Common words usually take a single token. Rare words, names and unusual word forms get split into two or more. Spaces and punctuation count too.

Put it to work

AI agents that look after your customers

From €49/mo + VAT

Voice, WhatsApp and email with the phone line included: agent, numbers and CRM in one fee.

See plans and pricing

That’s why the size of a text, as far as an AI is concerned, is measured in tokens rather than words, and why two texts with the same word count can “weigh” differently.

Related to this term

UTIO agents

  • Available 24/7 in 26 languages
  • Set it up yourself for €0: built from your website
  • 14-day free trial, no card needed

From €49/mo + VAT

See plans and pricing

How it works

There are three steps:

  1. Split. The tokenizer breaks the text up using the vocabulary the model learned during training.
  2. Number. Each token becomes a number. The model only ever works with those numbers.
  3. Generate. The model predicts the next token, then the next, until the answer is complete, and the result is turned back into text.

Tokens set two practical limits. The first is the context window: how many tokens the model can take into account at once, counting your instructions, any documents you pass it and the conversation so far. The second is cost: most providers charge per input token and per output token.

Language matters. Tokenizers are trained mostly on English, so a rule of thumb of one token for roughly three quarters of a word holds well in English, while other languages tend to need more tokens for the same message.

Example

A dental practice in Leeds wants an assistant to answer patients’ questions. Its fee guide and opening hours run to about 3,000 words. If the whole document is sent to the model with every question, each query drags thousands of input tokens along, even when the patient only wants to know whether it opens on Saturdays.

The usual fix is to send only the passage that answers the question. That’s what RAG does: it searches first and passes the model only what’s relevant. Fewer tokens per question, sharper answers and a smaller bill.

Tokens at UTIO

On UTIO plans you don’t buy tokens. The voice agent comes with included minutes and the WhatsApp agent with a monthly allowance of AI conversations: units you can understand without doing tokenizer maths. The AI is part of the monthly fee, together with the phone line, the numbers and the CRM.

If you’re a developer using the AI API, per-unit rates depend on the model tier and are published in the panel. There you’ll also meet “token” in its other sense: the access key created for a client, with its spending cap and rate limit.

Knowing what a token is helps explain why a well-designed AI agent doesn’t hand the model everything it knows, only what it needs for each answer.

Frequently asked questions

How many words is 1,000 tokens?

In English, roughly 750 words, since a token averages about three quarters of a word. Other languages, such as Spanish or French, usually get fewer words per 1,000 tokens. For an exact count, use the token counter your model provider offers.

Is a token the same as an API token?

No. For a language model, a token is a piece of text. Many platforms also call the key that identifies an API caller a “token”. Same word, two different meanings.

Why do models count tokens instead of words?

Because a model never sees letters or words, only numbers. Each token is an entry in its vocabulary with its own ID. Working in chunks lets it handle new words, names and typos without needing an infinite dictionary.

Related terms

  • RAG

    RAG (retrieval-augmented generation) is a technique in which a language model first retrieves the relevant passages from your own documents and then writes its answer from them, instead of relying on what it memorised in training.

  • AI agent

    An AI agent is a system that uses a language model to understand a request, decide which steps to take and carry them out with real tools, such as your calendar or CRM, until the task is done rather than just replying.

Updated on

← Back to the glossary

UTIO is the AI agent line of Universal Telecom, an operator registered with the CNMC.