Benchmark

What is a benchmark?

When a provider launches a new model, it usually comes with a table of scores. Those scores come from benchmarks: the same exam for every model, with the same questions and the same marking scheme.

There are many kinds. Some test general knowledge with multiple-choice questions. Others cover maths, coding, reading comprehension or following instructions. Others check whether the model makes things up or shows bias. And there are leaderboards where people compare two answers without knowing which model wrote each.

Benchmarks are useful for getting your bearings, but they have limits. Scores change with every version, models end up “learning” the popular tests, and a high score doesn’t guarantee the model will do well on your task.

How it works

1. Define the task and gather a set of questions with correct answers.
2. Fix how questions are asked: the same instructions, the same number of examples and the same settings, such as temperature.
3. Run every model over all the questions.
4. Score the results automatically (right or wrong answer, code that passes tests), with human raters, or with another model as judge.

Two well-known problems: contamination, when test questions have leaked into training data, and saturation, when every model scores near the maximum and the test no longer tells them apart. That’s why newer, harder benchmarks keep appearing, for example for reasoning models.

Example

A language school in Oxford wants an assistant to answer students’ questions about timetables, levels and enrolment. Looking at public leaderboards, it’s drawn to the model at the top of a reasoning ranking.

Before deciding, it builds its own benchmark: fifty real student questions with the correct answers. It compares three models. The winner isn’t the leaderboard champion but a model that replies faster, sticks to instructions and doesn’t invent class times. That small in-house exam tells the school more than any ranking.

How it affects your business

If you’re going to use AI with customers, your best benchmark is your own conversations. Collect real questions with the right answers and test against them. Look too at what leaderboards don’t measure: does the model admit it doesn’t know instead of hallucinating? How long does it take? What does it cost?

With UTIO you can run that test without committing: sign-up includes a free trial with no card required, as set out on the pricing page, so you can see how the AI agent handles your business before you decide.