Artificial Analysis announced AutomationBench-AA on July 6, an independent leaderboard for Zapier’s AutomationBench that tests whether AI agents can complete SaaS workflows without breaking business rules.
The benchmark uses 657 tasks across 40 simulated app environments, including categories such as Finance, HR, Marketing, Operations, Sales, and Support. Models interact through REST APIs and must discover the endpoints they need through structured tool calls.
Artificial Analysis says the tasks are graded programmatically against nearly 12,000 assertions. Some assertions are objectives the agent must complete; others are guardrails that already pass and must not be broken.
The headline score penalizes unsafe completion
AutomationBench-AA’s most useful design choice is that completing work is not enough. The headline score measures the share of objectives a model completes without violating guardrails.
That makes the benchmark closer to how business automation fails in practice. An agent that updates the right customer record but violates approval rules, exposes the wrong data, or changes unrelated fields is not simply less accurate. It is operationally unsafe.
At launch, Artificial Analysis says Claude Fable 5 and Claude Opus 4.8 lead with scores of 48.6% and 48.5%. Gemini 3.5 Flash follows at 42.6%, and GPT-5.5 xhigh is reported at 42.1%.
Those are Artificial Analysis results on its private subset, not a universal ranking for every agent workload.
Every tested model breaks rules
The more important result is that every model tested triggers guardrail violations.
Artificial Analysis reports guardrail violations ranging from 0.46 per task for Gemini 3.5 Flash to 1.26 per task for Qwen3.7 Plus. It also says Gemini 3.5 Flash completes 15.0 objectives per guardrail violation, the best ratio in the launch set.
The benchmark also shows uneven domain difficulty. Artificial Analysis says finance workflows are the hardest category, with agents completing roughly half the proportion of objectives they complete in Support and Operations tasks.
That gap is plausible and important: finance tasks usually have stricter state, permission, and compliance consequences.
