A new AgentCheck paper argues that tool-using agents are often tested under the easiest assumption: the tools work.
The July 13 arXiv paper introduces AgentCheck, an open-source workbench for testing LLM agents over MCP. The system runs an agent against its real tools, records the tool responses, injects a controlled fault into one response, and then replays the same setup after a mitigation is added.
That makes the result more useful than a one-off failure report. The developer can see whether the agent failed because of the injected fault, then rerun the same fault to confirm whether the fix actually helped.
The failure mode is silent confidence
The paper’s main result is uncomfortable for agent builders: the failures are often not crashes.
AgentCheck tests 120 scenarios across 12 fault types. The paper says the best of five tested agents passed 105 scenarios, while the weakest passed 77. The authors report that agents often failed by confidently using incorrect tool outputs rather than stopping, asking for clarification, or flagging uncertainty.
The fault types include visible errors such as timeouts and permission failures, plus harder cases such as stale data, contradictions, wrong answers, silent empty results, schema drift, and prompt-injection-like tool responses.
That maps closely to production agent risk. An agent that crashes on a timeout is annoying. An agent that silently trusts stale account data or a poisoned tool result can create a much harder incident.
MCP makes the test surface concrete
AgentCheck uses MCP as the intervention layer. That is timely because MCP is becoming a common way to connect agents to files, databases, browsers, internal tools, and SaaS systems.
The workbench treats an MCP server as a place where responses can be held constant, perturbed, and replayed. Matching tool calls can come from cache until the agent diverges, while later calls go live. That lets the test isolate the fault rather than mixing it with normal nondeterminism.
The paper also uses deterministic pass/fail checks first, with an LLM judge only for secondary diagnostic labels. That is the right bias. If a tool response contains stale data or a hidden instruction, the first question is whether the agent did the unsafe thing, not whether another model can write a plausible critique.
