Credited, Yet Failing Strict Clean Pass
241 of 431
Scored a success, failing the contract.
Tau task scoring credits 431 runs. 241 of them, 55.9%, fail Strict Clean Pass on the same trace.
AIMLSystems 2026
Do agents keep your data correct when tools fail?
ACID-Bench injects tool failures into Tau-Bench retail and airline tasks, then checks the resulting database state instead of the agent's reply.
- Scenarios
- 52
- Fault Families
- 5
- Recorded Runs
- 780
- Configurations
- 5
Results
Task success can hide a broken contract.
The same 780 recorded runs, scored two ways: Tau-Bench task success, and Strict Clean Pass, which also needs a valid run, the scenario's own checks and no integrity violation.
Valid, Fault Fired, Integrity Violation
197 of 241
Of those 241, 197 are valid runs where the fault fired and a contract requirement failed. The gap is not explained by invalid evidence alone.
- Strict Clean Pass234 of 780
- Strict Clean Pass, Task Failed44
- Label Disagreements, Both Directions285
- Runtime Failures0
Correction to one printed figure. The paper counts 525 of 780 runs (67.3%, 95% interval 63.9 to 70.5) with an integrity violation. That count includes 126 runs where the injected fault never fired, because the agent never reached the failing call (39 of them handed off to a human first). On those runs the flag only records that the required steps were not completed. Without them the count is 399 of 780 (51.2%, 95% interval 47.6 to 54.6). The figures above are not affected. Details in ERRATA.md.
Configurations
Strict Clean Pass, per configuration.
Across all 780 runs, Strict Clean Pass is 234, or 30.0% (95% interval 26.9 to 33.3 by run, not adjusted for repeated scenarios). Per configuration it spans 36 to 57 of 156.
Dots are rates, lines are 95% intervals by scenario.
| Configuration | Route | Strict Clean Pass | 95% Interval |
|---|---|---|---|
| DeepSeek V4 FlashNon-Thinking, Direct | Direct | 53 / 15634.0% |
23.1 to 45.5 |
| DeepSeek V4 FlashThinking-Max, Direct | Direct | 57 / 15636.5% |
26.3 to 47.4 |
| DeepSeek V4 ProThinking-Max, Direct | Direct | 49 / 15631.4% |
21.2 to 42.3 |
| MiniMax M2.7Reasoning-Low, OpenRouter | OpenRouter | 39 / 15625.0% |
16.0 to 35.3 |
| Kimi K2.6Reasoning-High, OpenRouter | OpenRouter | 36 / 15623.1% |
13.5 to 33.3 |
Strict Clean Pass for five tested configurations (52 author-designed scenarios, 3 runs each). Intervals resample scenarios. Configurations differ in provider route, reasoning settings and token limits, so this is not a model ranking. Within this release, most pairwise differences are not distinguishable; see KNOWN_ISSUES.md.
How It Works
One fault, injected exactly where it matters.
A deterministic proxy sits between the agent and the Tau-Bench retail and airline tools. It injects one declared fault on the matching call, records what really happened next to what the agent was told, and checks the resulting state. No LLM judge supplies a label.
Partial Commit
A later operation fails after earlier writes committed. The agent must keep the earlier effects and finish or reconcile the rest.
Ghost Success
The write goes through, but the agent receives an error. It must resolve the doubt without repeating the write.
Stale Read
A read is stale relative to a declared concurrent change. The agent must refresh before the dependent write.
Crash Restart
The call runs, is rolled back to the state before it, and the agent gets an error. It must account for earlier writes. To the agent and the database this looks the same as a write that failed outright; only the targeted tools, the error code and a cleared retry cache differ.
Consequence Fault
The write path adds evidence, verification, budget or harm constraints. The agent must meet those safeguards as well as the state checks.
Scope: 52 author-designed scenarios in two domains, a deterministic scripted user, and no independent expert validation of the contracts. Results describe the tested configurations under these contracts. See KNOWN_ISSUES.md.
Questions
What people ask about ACID-Bench.
- What is ACID-Bench?
- ACID-Bench is a fault-injection benchmark for LLM tool agents that change state. It has 52 scenarios over Tau-Bench retail and airline tasks. Each scenario injects a tool failure (a partial commit, a ghost success, a stale read, a crash and restart, or a consequence fault) and then checks the database the agent leaves behind with deterministic, state-based checks and no LLM judge.
- How is it different from Tau-Bench?
- Tau-Bench credits a run when the task's reward is reached. ACID-Bench also asks whether the resulting records are correct after a tool failure. Across 780 recorded runs, Tau-style scoring credits 431 and 241 of those fail Strict Clean Pass. ACID-Bench reuses Tau-Bench's environments at a pinned commit.
- Is it a model ranking?
- No. The five tested configurations differ in provider route, reasoning settings and token limits, so the per-configuration table is descriptive. Most pairwise differences are not distinguishable in this release.
- Where are the code and the data?
- The official artifact and reference implementation is on GitHub (MIT license). The 780 recorded runs are on Hugging Face (CC BY 4.0). The archived release has a Zenodo DOI, 10.5281/zenodo.24168719.
- How do I cite it?
- Cite the paper, "ACID-Bench: Fault-Injection Contracts for State-Changing Tool Agents", Andrei Vince, accepted at AIMLSystems 2026. A BibTeX entry is in the Cite section below.
- Is it the same as "Agentic Transaction: Towards ACID-Compliant Agent Systems"?
- No. That is a different project with a similar name. ACID-Bench is the benchmark described on this page.
- Can I train on it?
- Benchmark data should not appear in training corpora. Every dataset row and the repository carry a canary string so that contamination can be detected.
Cite ACID-Bench
Measure what the task did to the system.
@inproceedings{vince2026acidbench,
title = {ACID-Bench: Fault-Injection Contracts for State-Changing Tool Agents},
author = {Vince, Andrei},
booktitle = {AIMLSystems 2026},
year = {2026}
}
Code and evaluator: github.com/andreivince/acid-bench. Recorded runs: huggingface.co/datasets/andreivince/acid-bench-traces.