AI hallucination

What is an AI hallucination?

The term is borrowed from psychology, but in AI it means something more mundane: the model states something untrue, with the same confidence it uses for things that are true. It might be a date, a section of a law that doesn’t exist, a quote attributed to someone who never said it, or a shop’s opening hours that it imagined.

One of the most cited cases happened in the US in 2023: a lawyer filed a brief citing court rulings that a chatbot had generated. The rulings didn’t exist, and the lawyer was sanctioned.

Hallucinations aren’t a one-off bug in a particular model. They follow from the way every LLM works.

How it works

A language model generates text by choosing, token by token, whatever is most likely to come next. It doesn’t consult a database of facts. So:

• If the fact isn’t in what it learned, or appeared only rarely, the model fills the gap with something plausible.
• If the question takes something for granted, such as “what time does your Leeds branch open?” when there’s no Leeds branch, the model tends to play along.
• If the temperature is high, there’s more randomness in word choice, and more risk with it.

What reduces them most:

1. Give it the sources: with RAG, the model answers from your documents rather than from memory.
2. Tell it not to answer anything that isn’t in those sources, and let it say “I don’t know”.
3. Review anything important before it reaches a customer.

Reasoning models and fine-tuning can improve accuracy on some tasks, but they don’t make the problem go away.

Example

A small chain of opticians in Glasgow trials a generic chatbot on its website. A customer asks whether their health cash plan is accepted, and the chatbot says yes, quoting a specific discount. Neither the arrangement nor the discount exists: the model inferred them from the idea that “opticians often work with health plans”.

The chain changes tack: it uploads its real list of accepted plans and their terms as a knowledge base and instructs the assistant to say so when a plan isn’t on the list, then pass the query to the team.

Hallucinations at UTIO

UTIO’s agents answer from your knowledge base. In the support vertical, anything not in the documentation goes unanswered rather than invented: the agent takes a note and opens a ticket in the CRM. Autonomy is also gradual: you can start in draft mode, with a person approving every reply.

The content agent applies the same principle to what you publish: no figure goes out without a source; if it doesn’t have one, it pauses the campaign and tells you exactly which sentence needs backing up. That way an AI agent speaks for you without making up what it doesn’t know.