AI bias

What is AI bias?

A model learns from examples. If the examples are lopsided, the model learns the imbalance as if it were a rule. There’s no malice in the algorithm: it faithfully mirrors its data, gaps and prejudices included.

A widely cited case is an experimental recruitment tool that marked down CVs from women. It had learned from previous years’ hires, mostly men, and treated that pattern as a sign of a good candidate. The company dropped the tool. In the same way, speech recognition can make more mistakes with accents that were under-represented in its training data.

How it happens

Bias can creep in at several stages:

1. In the data. A dataset with few examples from one group, or one that reflects unfair decisions made in the past. This is the most common source.
2. In the design. Choosing what to predict and which variables to use. A postcode, for instance, can act as an indirect proxy for ethnicity or income.
3. In training and evaluation. If you only measure average accuracy, a model can work well for most people and badly for a minority without anyone noticing.
4. In use. Automation bias: people accept the machine’s recommendation without questioning it, even when it looks off.

Language models add another front: they can repeat stereotypes found in the text they were trained on, for example by assuming the gender that goes with a job.

Example

A lettings agency in London uses a tool to rank prospective tenants. When it breaks the results down by applicant type, it finds that self-employed people nearly always end up at the bottom, even with high incomes. The model had learned that “salaried employee” meant “good tenant”, because that’s what the history showed.

The agency changes its criteria, adds examples of creditworthy self-employed tenants and decides that a person will always make the final call, with the ranking used only as support.

What it means for your business

If you use AI with customers, there are two things to avoid: treating some of them worse than others, and letting it make important decisions unsupervised. The first is tackled by reviewing results; the second with clear rules and a person in charge. The EU AI Act takes the same line, with stricter obligations the higher the risk.

At UTIO, the voice agent and WhatsApp agent follow the rules you set: what they may do, which information they answer from and when they hand over to a person. Autonomy is gradual (no AI, ask first, draft or automatic) and starts in draft. That way you can check how the agent responds to different kinds of customer before giving it more room.