About
Decision rule
The rule is computed from the single-versus-multi comparison and published with that eval. It is not on this page yet, and it will not be guessed.
Methodology
There are 50 everyday support tasks and 30 harder tasks. Each run opens a new in-memory database seeded with the same 20 orders. Outcome accuracy checks that database after the run. Tool accuracy compares the ordered tool calls with the expected path. The multi-agent comparison runs each harder task once with one agent and once with a lead plus workers, over several trials, and reports the mean. Eval scripts run on a laptop. This site only replays one live task at a time and displays the committed results.
Limitations
- The task set is small, so a few unusual replies move the percentages.
- Model output is not fully deterministic, even at temperature 0. The multi-agent comparison averages trials for that reason.
- Orders, payments, and failures are mock data. A timeout is simulated after the refund row is written.
- Dollar figures use Groq’s published token prices. On the free plan those figures are a meter, not a bill.
- The default model does not emit parallel tool calls, so a turn usually contains one tool.