AI glossary · Models
Quantization
In one sentence
Quantization is a technique that stores an AI model's parameters as lower-precision numbers, for example 8 or 4 bits instead of 16. The model takes up less memory and often runs faster, in exchange for a loss of quality that can be small.
Also known as: model quantization · quantisation · LLM quantization · quantized model
What is quantization?
At heart, an AI model is a huge collection of numbers: its parameters. Each one is usually stored with 16 bits of precision. Quantization rounds them off so they can be stored in fewer bits, much as a photo is compressed to make the file smaller.
Put it to work
UTIO's AI, through an API
From €29/mo + VAT
For developers and agencies building their own products.
See the AI APIThe maths is simple. A 7-billion-parameter model at 16 bits (2 bytes each) takes up about 14 GB in weights alone. At 4 bits, about 3.5 GB. That gap decides whether the model fits on a consumer graphics card, a laptop or only an expensive server.
On top of that, moving less data between memory and processor speeds up inference, the stage where the model is actually used to answer, which is where the bottleneck usually sits.
Related to this term
AI API
- Transcription, voice, text, image, OCR and embeddings
- Per-client keys with a spending cap
- Prepaid wallet: €50 minimum top-up
From €29/mo + VAT Panel account, plus usage
See the AI APIHow it works
There are two main approaches:
- Post-training quantization. Take the trained model and convert its weights to lower precision, sometimes using a small calibration dataset to choose how to round. It’s the most common approach because it’s quick and cheap.
- Quantization-aware training. The model is trained or re-tuned knowing it will run with fewer bits, so it learns to compensate for the rounding. It loses less quality but costs more.
You can quantize just the weights or the intermediate calculations as well. Modern methods protect the most sensitive values so rounding affects them as little as possible.
Quantization often goes hand in hand with small language models and with open-source AI, whose models are frequently released in pre-quantized versions for running on your own hardware.
Example
A law firm in York wants to use AI to summarise case files but would rather the documents never left the office. It buys a computer with a strong graphics card, but the full version of its chosen model won’t fit in the card’s memory.
A 4-bit quantized version fits and runs smoothly. Before adopting it, the firm compares summaries from the quantized and full versions on twenty old files. The difference is negligible for its purposes, so it goes ahead, with a solicitor reviewing every summary.
Quantization at UTIO
If you use UTIO’s AI API, there’s nothing to quantize and no graphics cards to manage: the model runs on our infrastructure. What you choose is the model tier for text generation (basic, standard or superior), which balances cost against capability in the same way quantization balances memory against quality.
Quantization matters if you run models on your own hardware. In that case, always test the quantized version on your real cases before you put it in front of a customer or inside an AI agent.
Frequently asked questions
How much quality does a quantized model lose?
It depends on the model, the method and the number of bits. At 8 bits the difference is often hard to spot; at 4 bits many models still do well on everyday tasks; below that, the drop becomes more noticeable. The safe approach is to test the quantized version on your own cases before you use it.
What do GGUF, GPTQ and AWQ mean?
They're quantization formats and methods. GGUF is the file format used by llama.cpp to run models on ordinary computers; GPTQ and AWQ are methods for quantizing weights while losing as little quality as possible. You'll see them in the names of models released already quantized.
Related terms
- Model parameters
A model's parameters are its internal numeric values, mainly the weights of a neural network, which are adjusted during training and store what it has learned. Their count is used as a rough measure of model size.
- Small language model (SLM)
A small language model (SLM) is a language model with far fewer parameters than an LLM. It knows less about everything, but it replies faster, costs less and can run on a modest server, a laptop or even a phone.
- Inference
Inference is when an already-trained AI model is put to use: it receives a new input, such as a question, an audio clip or an image, and computes a response from what it learned, without changing its parameters.
- Open-source AI
Open-source AI is AI whose model is released so anyone can use, study, modify and share it. In practice, many so-called open models publish only their weights, not the training data or all of the training code.
- Mixture of experts (MoE)
Mixture of experts (MoE) is an AI model architecture split into several sub-networks called experts. For each token, a router switches on only a few of them, so the model can be very large without using all of its compute at every step.
Updated on