Unbiased reliability
pass@k and pass^k estimators, bias-corrected per task.
Statistics for agent evaluation
How much of a leaderboard is signal?
Agent leaderboards report one number per system, from one run. AgentAudit is the toolkit, the benchmark and the empirical study that tell you how much of that number you can trust, including the parts of our own hypothesis the data did not support.
- 185 tests
- 97% coverage
- ruff clean
- NumPy + SciPy only
- €0 spent
Try it
Same agents. Same benchmark. Run it again.
Six agents whose true success rates differ by just one point each. Every press of the button re-runs the whole benchmark once, exactly as a leaderboard would. Watch who comes first.
The evidence
Four numbers worth remembering
Where could the top ten truly rank?
SWE-bench Verified, public submissions. Bars are 95% bootstrap intervals on the leaderboard rank.
Distinguishable only when the gap is large
Share of agent pairs a paired test can tell apart, by score gap: raw p < 0.05 vs Holm-adjusted over pairs.
More runs made the interval more wrong
Simulated coverage of a nominal 95% interval for an agent's mean score. Treating all runs of a task as one binomial sample shrinks the interval but not the error.
Peeking at the result
False-positive rate when two identical agents are compared and the experiment may stop early.
Where the variance lives
Local open-weight models on AgentAudit-Bench: crossed agent × task × run decomposition.
Seven local models, 120 tasks, 10 runs each
Score with 95% interval. Run on one laptop, at zero cost.
The toolkit
One library. Eight ways to doubt a number.
Honest uncertainty
Wilson, Clopper-Pearson, beta-binomial, task, run and two-stage bootstrap, BCa.
Crossed decomposition
Agent × task × run ANOVA with a generalizability coefficient.
Paired, exact tests
Sign-flip permutation, cluster bootstrap, McNemar, Holm and BH.
Rank distributions
P(top-k), pairwise P(i > j), split-half Kendall τ.
Power and cost
Minimum detectable effect, cost-optimal runs per task, tasks vs repeats.
Stop early, safely
Anytime-valid betting confidence sequences with equivalence margins.
Where agents break
Deterministic taxonomy and failing-step localisation. Tokens, never money.
# numpy + scipy only
pip install agentaudit
agentaudit demo
agentaudit analyze trajectories.jsonl --html report.html
agentaudit power --task-var 0.10 --run-var 0.15 --delta 0.05 --cost-task 10AgentAudit-Bench
A benchmark that audits itself
Six simulated worlds, tasks generated from a seed with controllable horizon, distractors and tool failure rate. Success is decided by the final state of the world, never by what the agent says.
Every task is audited: an oracle solves it, a do-nothing agent does not, and generation is deterministic.
- fs virtual file system
- sql SQLite database
- crm REST API
- booking room reservations
- forms web forms
- tabular data and calculator
Playground
Audit your own agents in the browser
Drop in a JSONL file of trajectories and get intervals, rank uncertainty and a power plan. The exact Python library runs through WebAssembly. Nothing leaves your machine.
Launch playgroundHonesty
What this does not show
- The real-model study uses 1B to 7B quantised local models on one laptop. Absolute scores do not transfer to frontier agents; generality rests on the simulations and the public re-analysis.
- The environments are simulated and cleaner than real software tasks.
- Tasks are treated as i.i.d. draws from the generator's population; intervals refer to that population.
- The SWE-bench re-analysis assumes every public submission was evaluated on all 500 Verified instances.
Cite
If it helps your work
@software{agentaudit2026,
author = {G{\'o}mez Pulido, Francisco Javier},
title = {AgentAudit: statistically rigorous evaluation of LLM agents},
year = {2026},
url = {https://github.com/fragompul/agentaudit}
}