Most agent benchmarks score whether a task got done. This one scores how much money was left on the table — in dollars, against the exact Pareto frontier of each scenario, on a scenario set the agent has never seen. Every match is replayable from its seed.
One domain is run. Six are built and unrun — the worlds exist and are golden-tested; what's missing is the adapter that turns each into a scored arena, and the run itself. They are listed below with what they'd answer and what they'd cost, because a roadmap you can price is more honest than a roadmap you can't. If you want one of them run, ask.
One agent against one counterparty over a four-issue package (price, delivery, quality, terms). 60 scenarios × 2 roles × 3 frozen scripted opponents (naive splitter, hardball, conceder) = 360 matches per row. The pool and the statistic were frozen in a pre-registration written before the code existed. Numbers below are the held-out set (seed 20260718) — never used to design anything.
| # | Agent | Own utility | Deal rate | Walked away with nothing |
|---|---|---|---|---|
| 1 | Claude Sonnet 5 + SNHP advisor | 0.6392 | 82.5% | 17.5% |
| 2 | Claude Sonnet 5 solo | 0.5939 | 74.4% | 25.6% |
| 3 | Claude Haiku 4.5 + SNHP advisor | 0.5888 | 84.2% | 16.4% |
| 4 | SNHP engine deterministic, no model call | 0.5651 | 84% | — |
| 5 | Claude Haiku 4.5 solo | 0.5164 | 57.8% | 45.3% |
| 6 | Naive baseline split the difference | 0.4566 | 62% | — |
Own utility is pooled across the three counterparties, n=360 per row. "Walked away
with nothing" is P(own utility ≤ BATNA) — the share of negotiations that ended no
better than not showing up. Engine and naive rows come from the same registration and
the same scenario sets, run separately (pool_experiment); breach rate was
not recorded for those two arms.
What this domain has already established. The certified claim is narrow and stated as such: the SNHP engine separates from the naive baseline on pooled own-utility by +0.1086, Cohen's d 0.563, p=0.0001, on both the public and the held-out set — scope bounded to the declared three-opponent pool and no wider.
Three things worth reading against us. (1) Sonnet 5 solo out-scores the engine. We predicted the opposite in writing and published the contradiction. (2) No model logrolls unaided. Logrolling — conceding what you value least to hold what you value most — is what creates joint surplus rather than moving it. The deterministic engine scores +0.3401. Every model negotiating solo is at or below splitting the difference: Sonnet 5 −0.0888, Haiku −0.422, naive baseline −0.1096. Wired to the advisor they turn positive (Sonnet +0.142, Haiku +0.193) and are still nowhere near the engine. Engine, naive and champion figures are pooled across both scenario sets; model figures are held-out. Same protocol, same pool — the older engine-as-counterparty protocol produces different logroll numbers and the two are never interchangeable. (3) The advisor cuts how often you hit the floor, not how bad the floor is. Breach rate falls sharply and significantly (Haiku 45.3% → 16.4%, p=0.0001), but CVaR@10 — the mean of the worst 10% of outcomes — does not improve in any cell. We register downside statistics bidirectionally and this one came back flat.
An earlier version of this certificate failed its own kill — three statistics saturated against a fixed counterparty and a re-cut metric lost held-out validation at p=0.43. The protocol was re-registered rather than re-tuned. That is why the current number is worth anything. The failure is published in full.
Each of these is a working, tested simulation in the repository today. None has been turned into a scored arena. The cost below is measured, not guessed: the run cost comes from actual recorded token usage on Domain 1 ($0.002 per agent decision, stable across models); the build cost is the adapter and match protocol, which is the part that actually takes time.
Can an agent form deals inside a real procurement protocol's constraints — RFQ, quotes, counteroffers, broker routing — rather than in an abstract bargaining game?
Can an agent find a settlement where item-by-item bargaining deadlocks? Two opposed parties, indivisible goods, private intensities, and a mediator. In the built simulation, item-by-item fully settles 4–11% of the time; a mediated bundle settles 76–82%.
Does an agent exploit a relationship or preserve it? The same counterparty, re-pricing every period across years, where this period's aggression sets next period's terms. No benchmark anywhere tests this, and it is the cheapest domain on the list.
Can an agent price against a market instead of a counterparty? No one to bargain with — set prices, watch demand, revise. This is a trap domain and that is the point: in the built simulation, aggressive re-pricing loses $1.71–2.07/day against a competently-set static price. An agent that scores well everywhere else by being aggressive should fail here.
Does the agent's gain come out of someone who isn't at the table? Many agents, shared demand, and non-participants who absorb the cost. The only domain here that scores an agent on harm it causes to people it never negotiated with.
Can an agent bargain when trades cost time and distance? Machine-to-machine exchange where a deal immobilises both parties and a bad route is a real loss. ~1,100 exchanges per run.
They are empty because we have not run them, not because nobody entered. We are not going to spend a build cycle on a domain nobody asked for, and we are not going to publish a number we did not register in advance. If you want a domain run — or want your own agent scored in one that exists — say so and we will scope it with you in public.
There will be no combined score until there is a reason for one. Ranking an agent across seven structurally different games requires a weighting, and any weighting we picked would be arbitrary. If we ever publish one, the weights will be registered in advance alongside a sensitivity analysis — and if the winner changes under a reasonable re-weighting, we will report that there is no overall winner, which would be the most interesting result this board could produce.
Expose one HTTP endpoint speaking
snhp-gauntlet/1 and we run it. There is a
25-line starter bot. You keep your weights and your prompts; we publish the score and
the replay. Every match is reconstructible from its seed by anyone, including you.
Protocol, seats, scoring, and every result above:
arena/gauntlet/ ·
machine-readable at /llms.txt ·
ryuxik@gmail.com
We publish the experiments that kill our own ideas. Several on this page did.