RuBench is a new repository-level coding-agent benchmark built around tasks written natively in Russian rather than translated from English.
The arXiv paper, submitted July 7, describes RuBench 1.0 as 25 tasks mined from recent fix commits in five live open-source repositories: aiohttp, aiogram, Laravel, NestJS, and Fastify. The tasks span Python, PHP, TypeScript, and JavaScript.
The paper says each task statement was written from scratch in Russian in the style of a customer request. The benchmark withholds the upstream maintainer regression tests used for grading and publishes a SHA-256 manifest for the grading oracles.
The benchmark is small, but the setup is useful
Most repository-level coding benchmarks are English by design. That leaves a gap for teams whose users, support tickets, or product requirements arrive in another language.
RuBench tries to measure that setting directly. It evaluates product configurations, not just raw model APIs: Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, plus Codex CLI with GPT-5.5, with three independent runs each.
The paper reports that the best configuration resolves 78.7% of tasks. It is careful about the sample size: at 25 tasks, only gaps to the weakest model are statistically resolvable.
That caveat is important. RuBench should not be treated as a general model ranking. It is better read as a design example for multilingual, product-level coding-agent tests.
Product behavior can change what is measured
The paper’s most practical finding is about a fifth, hors-concours configuration: Claude Code with Fable 5.
The author says trajectory audits found that on 5 of 25 tasks, or 20%, an official safeguard fallback silently rerouted routine HTTP-protocol fixes to Opus 4.8. That means the deployed product, not only the named model, became the measured unit.
That distinction matters for any coding-agent benchmark. A user may think they are evaluating a model, while the product is also applying routing, safeguards, hidden policies, or fallback behavior.
For teams buying coding agents, that means model labels are not enough. They need run logs, routing transparency, and repeatable evaluation conditions.
