AI glossary · Models
Multimodal AI
In one sentence
A multimodal model is an AI model that can take in or produce several types of data, such as text, images, audio or video, and connect them. For example, it can read a photo of an invoice and answer questions about it.
Also known as: multimodal model · multimodal LLM · MLLM · vision-language model
What is multimodal AI?
A “modality” is a type of data: text, image, audio, video. The first LLMs handled text only. A multimodal model adds other modalities and, crucially, links them: it understands that “the stain in the corner” refers to a specific spot in the photo you’ve just sent.
Put it to work
AI agents that look after your customers
From €49/mo + VAT
Voice, WhatsApp and email with the phone line included: agent, numbers and CRM in one fee.
See plans and pricingSome multimodal models work on the input side, taking images or audio and replying in text; others can also generate images or speech. Several well-known families, such as recent versions of GPT or Gemini, offer capabilities of this kind, though they vary from version to version.
Related to this term
UTIO agents
- Available 24/7 in 26 languages
- Set it up yourself for €0: built from your website
- 14-day free trial, no card needed
From €49/mo + VAT
See plans and pricingHow it works
The underlying idea is to bring every modality into the same internal “language” of numbers:
- Encode. A specialised component turns each image or audio clip into a sequence of vectors, similar to text tokens.
- Mix. Those vectors go into the model alongside the text, and the attention mechanism of the transformer relates them to one another.
- Respond. The model generates text and, if it’s built for it, images or audio too.
To make this work, the model is trained on large numbers of paired examples: photos with captions, recordings with transcripts.
Example
A small insurance broker in Norwich receives claims via WhatsApp: a photo of the dented car, a voice note explaining what happened and sometimes a photo of a handwritten accident report.
A multimodal system can describe the visible damage, transcribe and summarise the voice note and pull the drivers’ details from the report photo. The claims handler gets one tidy file instead of five loose attachments. The final assessment is still theirs: the model helps prepare the case, it doesn’t decide the payout.
How it affects your business
Your customers don’t just type: they send photos, voice notes and scanned documents. A system that can read several formats saves you from transcribing everything by hand.
At UTIO, these capabilities are offered separately behind a single key in the AI API: audio transcription, text to speech, text, image, OCR for scanned documents and embeddings. You combine them like building blocks. The voice agent follows the same chained approach: it transcribes what your customer says, processes it with a language model and replies with a synthetic voice.
Frequently asked questions
What's the difference between a multimodal model and several models chained together?
A multimodal model handles the different data types inside a single model. The alternative is to chain specialised models: one transcribes audio, another reasons over the text and a third generates speech. Both approaches are valid; a chain lets you pick the best component for each step.
Can a multimodal model watch videos?
Some can, usually by analysing individual frames together with the audio track. Capabilities vary a lot between models and versions, so check which formats each one supports in its documentation.
Related terms
- LLM
An LLM (large language model) is an AI model trained on vast amounts of text to predict the next word. That single skill lets it draft, summarise, translate, classify and hold a conversation.
- Computer vision
Computer vision is the branch of AI that lets a system interpret images and video: recognising objects, reading text, spotting defects or describing a scene. Today it mostly relies on neural networks trained on large numbers of images.
- Speech to text
Speech to text (STT) is the technology that turns spoken audio into written text. It's the software component, usually a model reached through an API, that does the listening in a voicebot, a dictation tool or a transcription service, either live or on recordings.
- OCR
OCR (optical character recognition) is the technology that turns the text in an image, a scanned PDF or a photo into digital text you can edit, search and process. With AI, it also understands the document's layout and structure.
- GPT
GPT (generative pre-trained transformer) is OpenAI's family of language models. The name sums up how they work: they generate text, are pre-trained on huge amounts of data and are built on the transformer architecture.
Updated on