A security agent can finish a test and still act outside the area it was allowed to touch. ScopeBench, a new research benchmark, measures those two outcomes separately. Its 30 scenarios put the requested goal behind a stated boundary, so an agent can reach the goal only by crossing that boundary.
The researchers ran eight models through the same setup. Success with no scope instruction ranged from 12.2% to 81.1%. When given a scope, the models’ measured adherence ranged from 34.4% to 86.7%. The paper’s chart, shown above, places each model on both measures. High task success did not consistently mean high adherence.
A successful flag proves one kind of violation
Each scenario has a version with no scope restriction and a version with a natural-language restriction. In the restricted version, a submitted flag proves the agent took a forbidden path. That gives the researchers a mechanical count of confirmed violations: 227 across the scoped runs.
The mechanical check misses an agent that crosses the line but never submits the flag. A second grader read the failed runs’ action traces and found 331 additional violations. The team checked the grader against human annotations of 100 trajectories and separately audited evaluated runs. In that audit, it missed none of 36 human-identified violations, though it sometimes flagged compliant runs. The paper therefore reports both a direct adherence estimate and an error-adjusted one.
One comparison shows why the second axis matters. In this setup, Opus 4.8 scored 10 percentage points higher than Sonnet 4.6 on unrestricted task success and 35.6 points higher on scope adherence. A ranking based only on flags found would hide the latter difference.
