The UK AI Security Institute is publishing evaluation results through EvalEval’s Evaluation Cards. The records bring scores together with context and configuration details, so a reader can see more of the conditions behind a benchmark result.
The first release covers the five benchmarks in AISI’s paper How Inference Compute Shapes Frontier LLM Evaluation: HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro, and Terminal-Bench 2.0. It includes results for Claude Opus 4, 4.5, and 4.6, plus GPT-5, 5.2, and 5.4. Two related cyber evaluations, Cyber CTFs and The Last Ones, use a different set of models that overlaps with that group.
EvalEval says the records include verified results, context, and configuration information. Its Every Eval Ever schema provides a common structure for reporting results. Evaluation Cards combine benchmark metadata, run data, and model metadata. AISI previously gave feedback on that schema, and this release puts it to use for publicly reported methods and findings.
The conditions behind a score
The paper studies how inference-time compute and evaluation protocol affect performance. In the example shown for Humanity’s Last Exam, the chart plots the cumulative share of attempted tasks solved as token use rises. When a model receives correctness feedback from an oracle after each attempt, it continues to solve more tasks as it uses more tokens. A score from that procedure needs the attempt and feedback rules beside it to mean much in a comparison.
The announcement also shows AISI’s Terminal-Bench 2.0 results alongside other reported evaluations of the same models under different setups. Its point is that a model name and benchmark name alone do not identify an equivalent test. Details about the run can explain why apparently comparable results differ.
Sources
- Hugging Face: How UK AISI and EvalEval are making benchmark results reproducible. Lead image: EvalEval and UK AI Security Institute.
