Skip to content

Architecture

schema ──► results (ResultSet: models x tasks x runs)
                │
 metrics ◄──────┤────► intervals, variance, compare, ranking, power, sequential
                │                       │
             analysis  ──► report (md/html) ──► cli, web (Pyodide)

bench (envs, tasks, fault injection) ─► runner ◄─ agents (text protocol, HTTP, synthetic, cache)
                                           │
                                  trajectories JSONL ─► failures
ingest (tau-bench, SWE-bench) ──────────────► results

The statistical core depends only on NumPy and SciPy; everything model-related uses the standard library.