Benchmarks
UK AISI shares evaluation records that show why benchmark setups matter
The UK AI Security Institute has put benchmark results and configuration details into Evaluation Cards, making comparisons easier to inspect while leaving some limits on reproducibility.
By The AI Feed Desk
Google DeepMind pilots double-blind AI evaluations
Google DeepMind is testing double-blind AI evaluations with cryptographic isolation so model owners cannot see confidential benchmarks and evaluators do not receive model weights.
By The AI Feed Desk
NVIDIA AVO reaches 100% on ARC-AGI-3 public set
NVIDIA says its AVO agent architecture completed all 183 public ARC-AGI-3 levels, showing how persistent memory and supervision change agent benchmarks.
By The AI Feed Desk
RuBench tests coding agents on native Russian repository tasks
RuBench adds 25 repository-level coding-agent tasks written natively in Russian, with withheld regression tests and product-agent runs.
By The AI Feed Desk
OpenAI retracts SWE-Bench Pro recommendation after benchmark audit
OpenAI audited SWE-Bench Pro and now estimates that roughly 30% of its tasks are broken, weakening a key coding-agent evaluation.
By The AI Feed Desk
Harvey LAB-AA shows legal agents still miss most complete deliverables
Artificial Analysis launched Harvey LAB-AA, a legal-agent benchmark where the top model fully passes only 14.2% of real-world legal tasks.
By The AI Feed Desk
AutomationBench-AA shows agents still break business guardrails
Artificial Analysis and Zapier launched AutomationBench-AA, a SaaS-agent benchmark that scores completed objectives only when business guardrails stay intact.
By The AI Feed Desk
Metacognition-Bench tests whether models notice their own mistakes
Metacognition-Bench measures whether language models detect tempting wrong reasoning paths, pairing trap-rate evaluation with adapters that flag likely free-form errors.
By The AI Feed Desk
ScarfBench shows coding agents still struggle with Java migrations
IBM Research's ScarfBench tests whether AI coding agents can preserve behavior while migrating Java applications across enterprise frameworks.
By The AI Feed Desk
Hugging Face and Every Eval Ever make model-card scores more inspectable
Community Evals and Every Eval Ever now connect model-page benchmark scores to structured provenance records.
By The AI Feed Desk
OpenAI's GeneBench-Pro makes biology benchmarks about judgment
GeneBench-Pro tests whether AI agents can handle ambiguous computational-biology analysis, not just clean benchmark questions.
By The AI Feed Desk
Cognition's FrontierCode asks whether AI code would survive review
FrontierCode evaluates coding agents on mergeability, code quality, scope, tests, and maintainer judgment instead of only functional correctness.
By The AI Feed Desk
AA-Briefcase tests agents on messy business work
Artificial Analysis' AA-Briefcase benchmark evaluates models on multi-week knowledge-work projects with documents, email, Slack data, deliverables, and graded analysis quality.
By The AI Feed Desk
Hugging Face measures whether tools are agent-friendly
Hugging Face's agent-focused benchmark tests whether software changes help coding agents finish tasks with fewer errors, tokens, and detours.
By The AI Feed Desk
DeepSWE makes coding-agent rankings a cost question
DeepSWE's June 20 leaderboard update separates frontier coding agents by pass rate, cost, output tokens, and agent steps across long-horizon software tasks.
By The AI Feed Desk
MLPerf Mobile v6.0 gives on-device LLMs a real test surface
MLCommons added standardized Android LLM tests to MLPerf Mobile v6.0, including Llama 3.2 1B, Llama 3.2 3B, and Llama 3.1 8B Instruct workloads.
By The AI Feed Desk
NVIDIA says Blackwell swept MLPerf Training 6.0
NVIDIA says Blackwell delivered the fastest time to train on all seven MLPerf Training 6.0 benchmarks, including large-scale DeepSeek-V3 and Llama workloads.
By The AI Feed Desk