What are AI benchmarks?

The four parts of any benchmark

Almost every benchmark, from the largest industry suite to this one, is built from the same four pieces. Knowing them is most of what you need to read a result critically.

1. A dataset - the actual questions or tasks. This is the benchmark's real content, and the part that most determines what it measures.
2. A harness - the code that sends each task to each model under identical conditions and records what comes back.
3. A scoring rule - how an answer becomes a number. Sometimes it is exact matching against a known answer. Increasingly, for open-ended work, it is another AI model reading the response against a written rubric.
4. A published result - the leaderboard. This is the only part most people ever see, which is the problem.

Why the same model gets different scores

Two benchmarks can score the same model differently without either being dishonest. They asked different questions, or scored them by a different rule, or ran the model with different settings. A score is meaningful relative to a procedure, and comparing numbers across benchmarks that do not share one is close to meaningless.

This is also why serious benchmarks version their methodology. If a rubric is reworded and scores shift by four points, a reader needs to be able to tell whether models changed or the ruler did.

Six questions to ask about any benchmark result

You do not need to be a machine-learning researcher to pressure-test a claim. These six questions do most of the work.

1. What is actually in the test set, and can I look at it? If the questions are secret, you are being asked to trust a number you cannot inspect. Sometimes there are good reasons for that. It is still a cost.
2. How many items is it? A gap of one or two points across ninety questions is usually noise. Benchmarks that report a winner without reporting sample size are hiding the most important caveat.
3. Who or what did the scoring? If another AI model graded the answers, that model has its own biases - including toward responses that resemble its own writing. Ask whether anyone checked.
4. Could the model have seen this before? If benchmark questions end up in training data, the model may be recalling rather than reasoning. This is called contamination, and it quietly inflates scores.
5. Can the result be reproduced? Published raw outputs and a runnable harness mean anyone can check. A PDF with a table in it means you are taking someone's word.
6. What is not being measured? Every benchmark chooses a slice of the world. The choice is an argument about what matters, and it is rarely stated out loud.

A short glossary

Contamination

When test questions have leaked into a model's training data, so a high score may reflect memorisation rather than ability.

LLM-as-judge

Using one AI model to grade another's open-ended answers against a rubric. Practical and cheap; also introduces the judge's own preferences into the result.

Rubric

The written standard a grader scores against - the dimensions being judged and what each score level means.

Control or calibration track

Questions with known answers, included not to rank anyone but to prove the machinery works. If models fail these, something is broken in the harness.

Held-out set

Questions deliberately kept unpublished, so there is something left to test with if the public set becomes contaminated.

Methodology version

A label on the exact rules a score was produced under. Scores from different methodology versions are not comparable.

Why we built another one

Public benchmarks cluster where measurement is cheap and consensus exists: competition mathematics, code, graduate exams in well-resourced fields, and tasks weighted toward high-GDP US economic sectors. Those are worth measuring. They are also a narrow slice of what these systems are actually used for.

overlooked-bench measures five things that mainstream evaluations skip: reasoning about contested moral questions, academic domains with thin literatures and substantial non-English scholarship, work for small organisations outside the US, education under real classroom constraints, and whether a model's willingness to criticise an institution depends on which institution is named.

Every raw model response, every judge score, and the judge's own written reasoning are committed to a public repository, run by run, and never edited after the fact. When we found that our own rubric wording had drifted between runs, the published hashes caught it, and the fix is in the changelog rather than quietly applied.

Who we are

overlooked-bench was started in September 2026 and is maintained by a student at Strathmore University in Nairobi, Kenya, and KTH Royal Institute of Technology in Stockholm, Sweden. It is not a project of either university and carries no institutional endorsement - they are where the work is studied, not sponsors of it, and neither has reviewed the methodology or the results.

It has no funding from any AI laboratory, model provider, or anyone else. Costs are API charges, paid personally, of roughly the price of a few coffees per monthly run. Any future funding will be disclosed before any results from that period are published.

We built it partly because the gap it measures is real, and partly for a second reason that matters as much to us: most public conversation about AI runs on numbers that almost nobody can check. Benchmark scores get quoted in headlines, in launch posts, and in policy arguments, stripped of the sample size, the scoring method, and the caveats that would tell you how much weight they can hold. Publishing a leaderboard without also teaching people how to read one seemed to us like part of the problem.

So the working rule is that anything we publish should be checkable by someone who did not build it - which is also why this page exists.

Frequently asked questions

Read the results

• View the leaderboard
• Open-source repository
• Methodology & known limitations
• Contribute an item