overlooked-bench - benchmarking what the benchmarks miss

Frontier benchmarks measure a narrow slice of the world: high-GDP US sectors, competition mathematics, and tasks with a scored right answer. This project measures the rest - moral reasoning on contested questions, academic domains with thin literatures, work for small and non-US organisations, education beyond the university track, and whether a model's willingness to criticise an institution depends on which institution it is.

Every raw model response and every judge score is committed to the repository, run by run, and never edited after the fact. If a number here is wrong, it is wrong in a way you can go and check. Methodology | Funding & disclosures | Raw run data

Loading runs...

Leaderboard

The overall index is the unweighted mean of the five headline category scores. The calibration track is excluded from it by design - it exists to prove the harness works, not to rank anyone. Click a column to sort.

Institutional criticism - treatment asymmetry

Prompts are built in matched groups: within a group every prompt asks for the same analysis in the same structure, and only the institution named changes. The asymmetry index is the mean score spread inside those groups. A model that treats matched prompts alike scores near zero. A high score means the model's willingness to criticise depends on who is being criticised - it is a measure of consistency, and carries no claim about which treatment was the correct one.

Model	Mean criticism score	Asymmetry index	Groups measured

Charts

Generated directly from summary.json by charts/generate_charts.py. Never hand-edited.

Run diagnostics

Check these before trusting anything above. A truncation or calibration warning means the run has a harness problem, not that a model regressed.