AI glossary · Models
Mixture of experts (MoE)
In one sentence
Mixture of experts (MoE) is an AI model architecture split into several sub-networks called experts. For each token, a router switches on only a few of them, so the model can be very large without using all of its compute at every step.
Also known as: MoE · sparse mixture of experts · MoE model · expert routing
What is mixture of experts?
In a conventional, “dense” LLM, every token passes through all of the model’s parameters. As the model grows, every answer costs more. Mixture of experts breaks that link: the model has a large number of parameters in total but uses only some of them for each token.
Put it to work
AI agents that look after your customers
From €49/mo + VAT
Voice, WhatsApp and email with the phone line included: agent, numbers and CRM in one fee.
See plans and pricingThat’s why MoE models come with two figures: total parameters, which reflect how much knowledge the model can hold, and active parameters, which determine how much computation each token takes. An MoE can offer the capacity of a large model at a per-token compute cost closer to a mid-sized one.
The idea dates back to the 1990s, but it took off in language models with releases such as Mistral AI’s Mixtral 8x7B in late 2023. Several well-known models, open and closed, now use variants of this architecture.
Related to this term
UTIO agents
- Available 24/7 in 26 languages
- Set it up yourself for €0: built from your website
- 14-day free trial, no card needed
From €49/mo + VAT
See plans and pricingHow it works
Inside each transformer block, the layer that processes tokens after attention is replaced by several parallel layers, the experts. A small extra component is added, the router:
- A token arrives at the MoE layer.
- The router scores which experts suit that token best and picks a few, often two.
- Only those experts process the token; their outputs are combined using weights.
- The next token may go to different experts.
During training, care is taken to keep the load balanced, so the same few experts don’t end up doing all the work.
Example
A software company in Exeter that makes practice management systems for vet clinics wants to add an assistant to its product. It compares two models of similar capability: one dense, one MoE.
In testing, the MoE replies faster on the same hardware, because each token involves less computation. But to run it on its own servers, the company needs graphics cards with more memory, since every expert has to be loaded. So the question isn’t “which is better” but which costs less for them: memory or response time. If they use the model through an API, the provider does that sum and they only see the price.
How it affects your business
You don’t need to choose an architecture to use AI. But understanding MoE explains two things you’ll see on model spec sheets: why some list “total” and “active” parameters, and how a very large model can still respond quickly.
Response speed matters most in real-time conversations, such as those handled by UTIO’s voice agent. And if you run models on your own infrastructure, the memory they take up can be reduced through quantization.
Frequently asked questions
Is each expert in an MoE an expert in a subject, like law or medicine?
No, despite the name. The model learns its own specialisation during training and it's rarely intuitive: one expert might fire mostly on punctuation, another on certain code patterns. There's no 'accounting expert' you can pick.
Does an MoE model need less memory?
Not necessarily. It does less computation per token, but every expert has to be loaded because the router might pick any of them. So an MoE can be fast and still need a lot of memory to run.
Related terms
- LLM
An LLM (large language model) is an AI model trained on vast amounts of text to predict the next word. That single skill lets it draft, summarise, translate, classify and hold a conversation.
- Transformer
A transformer is a neural network architecture that processes every word in a text at once and uses an attention mechanism to work out which parts relate to each other. It's the foundation of almost every language model in use today.
- Model parameters
A model's parameters are its internal numeric values, mainly the weights of a neural network, which are adjusted during training and store what it has learned. Their count is used as a rough measure of model size.
- Inference
Inference is when an already-trained AI model is put to use: it receives a new input, such as a question, an audio clip or an image, and computes a response from what it learned, without changing its parameters.
- Quantization
Quantization is a technique that stores an AI model's parameters as lower-precision numbers, for example 8 or 4 bits instead of 16. The model takes up less memory and often runs faster, in exchange for a loss of quality that can be small.
Updated on