A new benchmark called ORCA-bench is a useful reminder that coding ability is not the same thing as operational reliability.
The paper, submitted to arXiv on July 30, frames ORCA-bench as a production-fidelity benchmark for on-call root cause analysis. It gives agents access to an OpenTelemetry-based microservice system with metrics, logs, traces, code, and six days of telemetry, then asks them to handle 1,079 root-cause-analysis tasks.
The headline result is sobering. The authors report that even the strongest evaluated systems solve only a minority of medium and hard tasks, while weaker agents often hallucinate implausible root causes.
On-call work is not just code search
This is the right kind of test for the next wave of coding agents. Real incidents are messy. They require stitching together symptoms, deployments, traces, logs, service ownership, recent code changes, and sometimes ambiguous user reports.
A model can be excellent at generating code and still be poor at incident diagnosis. Root-cause analysis requires evidence discipline. The agent needs to gather signals, form hypotheses, reject tempting false explanations, and explain confidence without turning uncertainty into a fabricated answer.
ORCA-bench’s setup is valuable because it gives agents multiple observability channels instead of a single prompt. The paper also reports that removing code access degrades performance, which suggests that telemetry alone is not enough for reliable diagnosis.
Benchmarks should punish confident wrong answers
The most important failure mode is not low accuracy by itself. It is confident hallucination during an incident.
An on-call assistant that says “I don’t know yet, here are the three strongest clues” can still help a human responder. An agent that invents a root cause can waste time, trigger unnecessary rollbacks, or hide the real failure.
That is why ORCA-bench’s human-rescored judge and public testbed are useful. They push evaluation toward operational truth, not just answer fluency.
