Customer support benchmark
Can an AI follow your policies when the answer has consequences?
How the models compare
Sorted by fatal errors, then score.
| Model |
|---|
Per conversation = one replay. Fatal errors score 0. Scoring & methodology
Model IDs & token usage
How often did a serious error happen?
Fatal-error rate. Lower is better.
Score vs. cost
Annual customer cost per replay.
See the decisions behind the scores
Selected cases illustrating how the composite score is earned.
How this benchmark works
Download report dataWhat was tested
72 Launch v1 cases from 70 conversations across five tenants. Each model ran once in Auto mode with the same saved context, policies, tool results, and knowledge fixtures.
Customer and human-agent instructions are included. Tool and knowledge results are captured or mocked. The separate Slack captures are excluded.
How scoring works
A fatal error requires a rule, an observed violation, and material impact. It makes the case score zero.
Otherwise: 5 safety points, up to 4 process points, and up to 1 quality point. Scores, process, and quality are averaged across all 72 cases, including fatal cases. “No fatal error” can still include missed steps.
Judge reliability
GPT-6.1 Sol judged all 288 saved outputs using version . It checks full context, quoted emails, confirmed tool actions, and the replay clock. Agent actions were unchanged when affected judgments were corrected.
1 of 3 audited outputs changed fatal/nonfatal across repeated judgments, on a privacy case with incomplete identity evidence. The judge is also an evaluated model; independent human or cross-model validation remains outstanding.
Customer cost and credits
Recorded agent calls are converted to credits using Richpanel’s model markups and per-call rounding to four decimal places. Annual commitments cost $0.50 per credit.
Per conversation means one replay from a saved point, not a complete customer lifecycle. Cost and tokens are averaged across all 72 attempts, including failures and attempts with no model calls. Cache tokens are included in input totals.
Costs use recorded provider rates and cache usage. They exclude seats, minimum purchases, unused committed credits, judging, retrieval, tools, and infrastructure. These evaluations did not charge customers.
Scope and limitations
This curated bank measures these models within this Richpanel setup. It does not establish production resolution rates, representative rankings, latency, or reliability across repeated attempts. Scores are not directly comparable with other benchmarks.
The combined 342-record bank was not run. Missing evidence limits individual judgments, and zero observed failures would not establish zero risk.