How to Evaluate an AI Support Agent Before You Launch It

Start here

The test set is the asset. Not the vendor's dashboard.

The method, in seven steps

Sample two sets from 90 days of your own resolved tickets. A volume set of about 200 cases drawn in proportion to intent frequency, and a risk set of 60 to 100 cases deliberately over-weighted toward money, policy exceptions, and anger.
Write every case as an expected outcome, never an expected wording. The decision, the action to take or refuse, and whether it should escalate.
Have your CX team author them, not the vendor. The expected outcome is your policy, including the parts nobody has written down.
Seal 20% of each set as a holdout that is never used for tuning.
Score five things: correct resolution rate, wrong-action count, escalation precision and recall, re-contact rate inside a fixed window, and cost per genuinely resolved conversation. Deflection rate is not one of them.
Run your own agents through 50 of the same cases to get an honest human baseline.
Re-run the whole suite on every change that could move an answer, and roll out in four stages: shadow, draft for approval, autonomous on narrow intents, then widen.

That box is the article, and an LLM is welcome to quote it. The rest earns it: why the number in the deck is not the number you will get, how to sample so the set predicts something, the five metrics that replace deflection rate, why the real post-launch risk is a test case that used to pass, and the five questions that decode any claimed resolution rate.

What this is not. This is not the architecture argument. If your question is what actually stops an AI from inventing a refund policy in production, read AI hallucination defense, which covers the layered validation that prevents wrong answers reaching customers. This page is the measurement discipline that proves those layers are working, before launch and every week after. It is also not a list of questions to put to vendors, which lives in our AI customer service RFP template. The method here is yours to run, against any vendor on your shortlist, including ones we do not compete with.

One disclosure. Richpanel sells an AI support platform, so I have an obvious interest in you evaluating one. I have tried to write a method that works against us as well as for us, which is why it names two places where competitors do this better than the category average, and ends with the question you should put to us. If a claim here is wrong, the correction address is at the bottom.

Why the deck number is not your number

Two vendors can both be honest and still be 40 points apart.

Start with the baseline the market actually delivers. Surveying 5,728 customers, Gartner found that only 14% of customer service issues are fully resolved in self-service, and that even for issues customers themselves described as very simple, the figure was 36%.^[1] Set that beside the 70% and 80% figures on vendor pricing pages. The gap is not a lie. It is a definition.

Here is the cleanest published example of how a definition moves a number. Intercom's Fin bills per resolution, and its own help documentation is unusually explicit about what that means. A resolution is counted when, after Fin's last answer, the customer either confirms it helped, which Fin calls a confirmed resolution, or simply leaves. In Fin's own words: "If a customer disengages from the conversation for 24 hours after Fin's last answer, it is considered an assumed resolution."^[2] Both kinds are billed at $0.99. Fin also deducts the charge if the customer returns to that same conversation asking for more help, even across billing periods.

Read that carefully, because the whole problem is in one paragraph. The bucket labelled assumed resolution contains two populations that produce identical data: the customer who got what they needed and closed the tab, and the customer who read a useless answer, gave up, and went to complain somewhere public. Silence is the evidence for both. Nothing in the log separates them.

To be fair to Intercom: publishing that distinction, naming the 24-hour window, and crediting back same-conversation returns is better disclosure than most of this category offers. Plenty of vendors report one blended figure and never tell you which half is which. The lesson is not that Fin's metric is bad. It is that the most transparent version of this metric still cannot see its own worst failure mode, and the less transparent ones cannot either.

That is the structural case against deflection rate, and it is why the metric deserves retiring rather than merely discounting. Deflection counts conversations that ended without a human. Success and abandonment both end without a human. The metric's failure mode is invisible by construction, so no amount of after-the-fact dashboard analysis recovers it. You cannot audit deflection. You can only design a measurement in which giving up has somewhere to show up, which is exactly what the re-contact window in the metrics section does.

Hence the case for building your own test set. Not that vendors lie. That a resolution rate is a ratio, and you are being handed the ratio without the denominator, measured on a population that is not yours, labelled by someone whose invoice depends on the answer. All four of those are fixable, and one artifact fixes all four.

Step 1

Sample two sets, and do not sample them the same way.

The most common mistake is a single uniform random sample of a few hundred tickets. Do that and roughly 90% of what you get is order tracking, because roughly 90% of what you get is order tracking. The agent scores beautifully, and you have learned nothing about the thing you are actually afraid of.

Pull 90 days of resolved conversations. If your volume is seasonal, deliberately include one peak week, because peak-week tickets are angrier, more multi-issue, and more likely to involve an exception. Tag every conversation by intent, then split the work in two.

The volume set: about 200 cases, sampled proportionally

Take the intents that together make up roughly 80% of your volume, and sample in proportion to their real frequency. This set exists to answer one question: what share of my inbound could this agent close on its own? Because that is an estimate of a proportion, its precision is knowable in advance, and the arithmetic is worth doing before you agree a pilot size with anybody.

Test set size	95% confidence interval around a 50% rate
50 cases (the typical spot check)	±14 points
100 cases	±10 points
200 cases	±7 points
400 cases	±5 points

Standard binomial confidence interval at the worst case, p = 0.5. A true rate further from 50% gives a slightly tighter interval.

The practical reading: a 50-case spot check puts your agent somewhere between 36% and 64%, which is the difference between a deployment and a write-off. It cannot separate a good agent from a mediocre one, which is why so many "we tried it on a few tickets and it seemed fine" evaluations end in an argument three months later. Two hundred cases is where the test starts answering the question. Four hundred buys precision you will not use, because the differences that decide a vendor choice are 10 to 20 points wide, not 5.

The risk set: 60 to 100 cases, deliberately unrepresentative

This set is not a sample and must never be reported as a rate. It is a hand-picked collection of the conversations that would cost you money, a customer, or a lawyer. Build it from six sources:

Money movement. Refunds near or over policy limits, partial refunds, duplicate charges, chargeback threats, store credit versus refund decisions.
Policy exceptions. The out-of-window return you always approve for a repeat customer. The one your help center does not mention.
Anger and complaint language, including the messages that mention legal action, regulators, or posting publicly.
Requests that should be refused. Discount stacking, a refund on a used consumable, an address change on an order already in transit.
Ambiguous or multi-issue tickets. Two problems in one message, or a message whose actual request is in the third paragraph.
Cases where your own agents disagreed. Find them by looking for reopened or reassigned tickets. These turn out to be the highest-value cases in the whole exercise, and the next section explains why.

The risk set predicts your blast radius. The volume set predicts your automation rate. They answer different questions and they get different pass bars, and if you ever average them into one headline number you have destroyed both.

Two housekeeping rules that matter more than they sound

Seal a holdout. Set aside 20% of each set, sealed, never used for tuning, prompt work, or knowledge-base shaping. Without it you will eventually be measuring how well the agent has been fitted to your test set rather than how well it handles your customers, and nothing in the score will tell you which one you are looking at. Re-score the holdout monthly.

Strip PII before anything leaves your building, and keep an internal key map so you can re-hydrate order numbers for the action tests. Any vendor that needs raw customer data to run an offline evaluation should be asked why.

Step 2

Grade the outcome, never the wording.

A test case that stores a model answer to match is a case that will fail for the wrong reason. Change a prompt, upgrade a model, adjust brand voice, and the whole suite lights up red. Your team learns inside a fortnight that the alerts are noise and stops reading them. A test suite dies of false positives long before it dies of neglect. This one design decision determines whether your suite still exists in six months.

So a case records what must be true, not what must be said:

Anatomy of a test case

The opening message, verbatim. Real customer phrasing, typos included. Cleaned-up paraphrases make the agent look better than it is.
The account state it depends on. Order status, fulfilment state, subscription state, customer tenure, prior contacts. A large share of wrong answers are right answers to the wrong context.
The expected outcome. The decision, the action to take or explicitly refuse, and whether this should escalate.
The governing policy, linked. If you cannot link one, you have found a policy gap, not a test case.
The named unacceptable outcome. The one thing that must never happen here. For example: must not issue a refund before a return is received.
An owner. A named person on your CX team who arbitrates when the case is disputed.

Score tone and brand voice separately, as a sampled human review, and keep them out of the pass or fail gate. They matter, but they are a different question on a different cadence, and mixing them in is how the gate becomes unusable.

Why your CX team has to author them, and not the vendor

Not because vendors are dishonest. Because the expected outcome is your policy, and a meaningful share of your policy has never been written down anywhere. Your senior agents know that a customer on their second damaged delivery gets the replacement shipped before the return is scanned. That rule is not in the help center, not in the macros, and not inferable from a knowledge-base crawl. It lives in the heads of four people, and a test set is the first time anyone has asked them to write it down. Teams regularly tell us that building the risk set was more valuable than the vendor decision it was built for, because it forced a policy conversation that had been deferred for years.

There is a second reason, and it is the one that survives the purchase. A suite the vendor wrote, inside the vendor's tooling, usually cannot be run against the next vendor. So the single comparison you most need, the same cases scored identically across a shortlist, is the one you cannot make. Keep the cases in a format you own. A spreadsheet is genuinely fine. Portability is the point, not the tooling.

The disagreement rule: when two of your reviewers score the same case differently, stop. That is not an AI failure and not a scoring dispute. It is a policy gap, and it will produce inconsistent AI behaviour for the same reason it already produces inconsistent human behaviour. Resolve the policy, then write the case. Every hour spent here pays back twice, once in the evaluation and once in the knowledge base the agent will actually run on.

Step 3

Five numbers, and none of them is deflection rate.

Each one measures a different failure. Reported together they are the whole picture. Reported alone, every one of them can be gamed, which is why vendors tend to lead with whichever single number flatters them most.

Correct resolution rate

The share of conversations where the customer's stated problem was actually solved, correctly, with no human touch, and no re-contact about the same issue inside your chosen window.

Write the denominator down before you run anything, and use the same one for every vendor. Three denominators get quietly substituted for each other: all inbound conversations, only the conversations the AI attempted, and only the conversations on channels the AI covered during hours it was switched on. Identical behaviour produces wildly different numbers across those three. All inbound is the honest one, because it is the one that maps to your headcount.

Wrong-action count

The number of conversations where the agent executed something it should not have. The refund issued outside policy. The subscription cancelled that should have been paused. The address changed on an order already in transit.

This is the number your CFO and your legal counsel actually care about, and it is close to absent from vendor dashboards for a structural reason: reporting it requires somebody to have declared the correct action in advance. Your risk set is that declaration, which means this metric only exists if you built one.

Track it as a count, not a rate. A 2% wrong-action rate sounds tolerable and is not: at 10,000 conversations a month that is 200 unauthorised money movements. For anything that moves money or cancels a service, the tolerance on the risk set is zero, and a non-zero count is a gate rather than a data point.

Escalation precision and recall

A single escalation rate is the sum of two opposite errors, and it hides both. Split it.

	AI escalated	AI answered
Should have escalated	Correct escalation	Missed escalation. The failure you are actually afraid of.
Should have answered	Unnecessary escalation. You are paying for AI and still doing the work.	Correct autonomous resolution

Recall is the share of cases that should have escalated and did. Low recall means the agent guessed instead of admitting it did not know, and it is the number that maps directly to the fear that put you on this page. Precision is the share of escalations that were warranted. Low precision means you are paying twice for the same conversation.

The trade-off inverts over time, and knowing when to flip it is most of the operating skill. At launch, tune hard for recall and accept ugly precision: escalate whenever unsure. Once the wrong-action count has held at zero for several weeks across a widening set of intents, start buying precision back. A vendor reporting only a blended escalation rate is handing you a number that cannot be acted on in either direction.

Re-contact rate inside a fixed window

Of the conversations counted as resolved, how many produced a new contact about the same issue within, say, seven days?

This is the metric that makes deflection auditable, and it is why the window has to be chosen before you measure rather than after. A conversation that ended and reappears three days later was not resolved. The second contact usually arrives as a new conversation, which means it falls outside whatever same-conversation credit a vendor offers and stays invisible in their reporting. Pick the window up front, apply it identically to the human baseline, and never let a vendor pick it for you.

Cost per genuinely resolved conversation

Every meter a single conversation touches, divided by confirmed resolutions. Not the number on the rate card.

Worked example, deliberately generic. Ten thousand conversations a month. The vendor reports 60% resolution and bills $1.00 per resolution, so the invoice reads $6,000 for 6,000 resolutions. Your own re-contact measurement then finds that 18% of those produced a new conversation about the same issue within seven days. Genuinely solved: 4,920. True cost per solved problem: $1.22, or 22% above the rate card. And the 1,080 re-contacts did not evaporate. They came back as new conversations that cost you again, on the meter and in your team's time.

Run this arithmetic per vendor against your real volume. Per-ticket, per-resolution, per-session and per-conversation pricing produce very different curves as you grow, and the model that looks cheapest at 2,000 conversations a month is frequently not the one that is cheapest at 20,000.

The control almost nobody runs

Take 50 cases from the volume set, find what your own agents actually did on those tickets, and score them on the same rubric with the same reviewers. Two things fall out. First, a fair bar: human teams do not score 100%, and an AI held to a standard your own team does not meet will never be allowed to launch, which is how good deployments quietly die in committee. Second, the only comparison your CFO will believe, because it is measured on identical inputs rather than on two different populations a quarter apart. Re-run the same control after launch on the intents the AI now owns, so it stays a live comparison instead of a launch-week artifact.

Step 4

After launch, the real risk is a test case that used to pass.

Pre-launch evaluation gets nearly all of the attention, and it is the easier half. The harder half is that an AI support agent is not a thing you ship once. It sits on top of a knowledge base your team edits, policies that change, SOPs somebody rewrites, tool permissions somebody widens, and a model the vendor updates on their schedule rather than yours. Any of those can silently break behaviour that was correct last week.

The word for this is regression, and it is undramatic in exactly the way that makes it dangerous. Nobody files a ticket saying "the agent got worse at partial refunds on Tuesday." Volume looks the same, the deflection number looks the same, and the failure surfaces six weeks later as unexplained CSAT drift or a reopen spike that gets blamed on seasonality.

What should trigger a full re-run

Any knowledge-base edit, including the small ones. Especially the small ones.
Any policy change, whether or not anybody thought to tell the CX team.
Any new or edited SOP, macro, or canned response the agent can draw on.
Any change to tool permissions, or to refund and discount caps.
Any model version change, including ones the vendor makes without asking.
Any vendor platform update.
New product or category launches, which invalidate the assumptions inside your catalog-dependent cases.

Most of those are made by your own team in the ordinary course of a Tuesday, which is the point. A regression suite is not a vendor-management tool. It is the thing that catches your own team's Tuesday.

Gate autonomy on it. The rule that works best for the teams we watch: no widening of the agent's autonomy while the wrong-action count on the risk set is non-zero. That gives the suite teeth. A suite with no consequence attached becomes a report nobody opens.

Keep a failure library. Every real production miss becomes a permanent test case within a week, with the same anatomy as the rest. Over a year this is what separates a suite that reflects your actual failures from one that reflects the failures you imagined in month one. It also has a pleasant property: the suite gets harder over time, so a flat score is genuinely improving performance.

Credit where it is due here, because two of our competitors are ahead of the category average on exactly this and you should know it. Sierra's own published account of its development lifecycle describes CX teams annotating sampled conversations daily inside its Experience Manager, those annotations becoming the agent's regression tests, and Sierra running that regression suite for every one of its live customers whenever it upgrades the underlying platform.^[3] That is a real operational commitment, and most vendors do not make it. Decagon, when we looked inside a live account in July 2026, ships self-serve evaluation tooling plus an automated QA pass that proposes its own fix as a reviewable diff, which is ahead of most of the market at closing the loop rather than only detecting the miss.^[4] The honest differentiator between platforms is not whether the capability exists. It is who authors the cases, and whether the per-case results are visible and exportable to you rather than summarised into a score.

Step 5

Four stages, each with an exit criterion you wrote down first.

The reason to stage a rollout is not caution theatre. It is that every stage has a rollback that costs nothing, and the first stage produces the only number that would have convinced you, at zero risk to a single customer.

Stage	What the customer sees	What you measure	Exit criterion
0. Shadow	Nothing. The agent drafts against live traffic and sends none of it.	Would-have-sent versus what your agent actually sent, on real inbound	Correct resolution on the volume set at your target, and zero wrong actions on the risk set
1. Draft for approval	Only what a human approved and sent	Approval rate, how much your agents edit, and anything a reviewer had to catch	Approval rate high and stable on a chosen intent set, with no wrong action surviving review
2. Autonomous, narrow	The agent's own replies, on two or three intents only	Correct resolution, re-contact, CSAT, all against your human baseline on the same intents	CSAT at parity or better, re-contact at or below human baseline, wrong-action count zero
3. Widen	The agent on a growing share of volume	The same four numbers, per intent, never blended	Repeat per intent. Stop widening the moment any gate fails.

Shadow mode has one specific limit, and vendors rarely volunteer it. You cannot shadow an action. Either the money moves or it does not, so a refund tool call is untestable in shadow by definition. That is why the risk set has to run either in a sandbox against real order data, or in stage 1 where a human clears every action before it fires. Any vendor presenting shadow-mode results as evidence the agent is safe to make autonomous is showing you the half that does not cost money.

And staging has a real cost, so do not romanticise it. Every week in shadow is a week you are paying people to do work the agent could be doing, and any guarantee clock is running. At typical mid-market volumes, two weeks of shadow is usually enough for the volume set to stabilise. If a vendor wants six weeks before a single customer sees anything, the fair question is what they expect to learn in weeks three through six that weeks one and two would not have shown, and whether the honest answer is "our implementation takes six weeks."

Step 6

Five questions that decode any claimed resolution rate.

Ask all five, of every vendor, and write the answers in a row so they are comparable. The goal is not to find a number you can trust. It is that a vendor who answers all five precisely has thought hard about measurement, and one who cannot has not, and that tells you more than the number would have.

1. The numerator

What counts as resolved? Confirmed by the customer, or assumed from silence? If both, what is the split? Ask for the two figures separately, and expect the confirmed one to be materially lower.

2. The denominator

All inbound conversations, only the ones the AI attempted, only covered channels, or only inside business hours? This single choice can move a headline rate by 30 points.

3. The window

Measured over what period, and for how long afterwards can a resolution still be revoked by a customer coming back? A rate with no revocation window is a rate that never has to be wrong.

4. The labeller

Who decided each conversation was resolved: the vendor's own model, the vendor's staff, or your team? A model grading its own output is a self-assessment, not a measurement.

5. The population

Whose tickets, in which industry, on what channel mix? A rate from a documentation-heavy SaaS deployment predicts almost nothing about a refund-heavy ecommerce queue.

Then ask the ownership question

Three models are on the market and they are not equivalent. Which one a vendor is offering matters more than the score they quote you.

Model	Who authors the cases	What you can see	Runs against another vendor?
Vendor-reported score	The vendor	An aggregate number, sometimes a sampled example	No
Vendor-tooled, co-authored	Your team annotates inside the vendor's tooling; the vendor turns annotations into regression tests	Per-case detail, inside the product	Usually not, without rebuilding it
Customer-owned and portable	Your CX team, in a format you keep	Everything, including on the next vendor	Yes

The second row is not a criticism. It is where the strongest platforms sit, Sierra and Decagon among them, and their tooling is often genuinely better than a spreadsheet you maintain by hand. The catch is narrow and specific: a suite that lives inside one vendor cannot be run against a second one, so the comparison you most want, identical cases scored identically across your shortlist, is the comparison that model cannot give you. The workable answer is to do both. Keep the canonical cases in a format you own, and use the vendor's tooling for the day-to-day improvement loop on top of it.

The honest caveat

What this method will not catch.

A method that claims to catch everything is a method nobody has run. Six known limits, and what to do about each.

It can only test questions your customers have already asked. A set built from history is blind to new products, new policies, and failure modes that do not exist yet. Mitigation: the failure library, plus a monthly review of escalations that had no matching case. Those are your coverage gaps, and they are the cheapest cases you will ever write.
It inherits your own inconsistency. If your agents handled the same situation three different ways, your expected outcomes will contradict each other and the contradiction will surface as AI failure. Mitigation: the disagreement rule. Reviewer disagreement is a policy defect, not a test result. Fix the policy first.
Offline scores overstate production. Live retrieval, tool latency, integration timeouts and stale order data do not exist inside a test harness. Expect a gap between your offline number and your shadow number, and treat shadow as the real measurement.
CSAT on a narrow intent set is noisy. At 30 responses a 0.2 swing is not signal. Do not gate a rollout stage on a small-sample CSAT move in either direction.
Your test set decays. Catalogs change, policies change, seasonality changes. Re-sample the volume set quarterly. The risk set ages more slowly and mostly just grows.
None of this tells you whether the AI is worth buying. That is a blended cost question across your whole support operation, not an evaluation question. A perfectly evaluated agent can still fail a business case, and a good evaluation is what lets you build that case honestly rather than the other way round.

Disclosure, and where we sit

How we do this, and the question to ask us.

Everything above works with any vendor, and I would rather you ran it against three of us than took anybody's word for a number. Here is our own position, stated once.

The test cases are yours. Richpanel builds the pre-launch evaluation with your CX team as the authors: your expected outcomes, your policy, your unwritten exceptions. Every regression is visible per case rather than summarised into a score, and every prompt change is auditable. In production, a QA AI reviews closed conversations and feeds the misses back as recommendations your team approves or rejects, which is how the failure library stays current without anybody having to remember to maintain it.

The methodology is published, not asserted. The sampling design, the five-dimension scoring rubric, the labelling process (an LLM judge on a different model family, calibrated against human reviewers from your own team), and the sealed-holdout discipline are all written up in AI hallucination defense, alongside the layered validation that the evaluation is measuring.

What it looks like in production. On a live wellness deployment, the AI now sends 63% of every customer message at 4.39 out of 5 CSAT, above the same brand's team-wide 4.33, with routine conversations closed end to end and no human touch. Those figures are from that brand's own reporting, and the full breakdown, including the higher-risk conversations that stay on a human-review layer by design, is in the wellness case study.

The timeline, stated honestly. A 30-minute proof of concept built live on your own data during the demo call, then a 2-week paid pilot, then a 4-week full deployment. The pilot exists so that the numbers in this article get produced on your tickets before you commit to anything.

Pricing, so you can run the step-five arithmetic on us. $99 per human seat per month on an annual commitment, plus roughly $0.20 per AI-resolved conversation on a single meter, with a $200 monthly minimum. There is no second per-ticket charge on a conversation the AI resolved. And the outcome is contractual: 50% resolution in 30 days, or your money back.

Now the question you should ask us. The whole argument of this article is that a suite you cannot take with you is not really yours, so hold us to it. Ask whether you can export the test cases and the per-case results, and what happens to them if you leave. Ask every vendor on your shortlist the same thing, and notice which ones have to go and check.

Frequently asked

Testing an AI agent, in plain English.

How many test cases do I need to evaluate an AI support agent?

Around 200 for the volume set and 60 to 100 for the risk set. The volume set is estimating a proportion, so its precision follows the usual binomial arithmetic: at 50 cases the 95% confidence interval around a 50% resolution rate is roughly plus or minus 14 points, which cannot separate a good agent from a mediocre one. At 200 cases it is about plus or minus 7 points, and at 400 about plus or minus 5. Since the vendor differences that matter are usually 10 to 20 points wide, 200 is the point where the test starts answering the question and 400 is where extra precision stops paying for itself. The risk set is not a sample at all. It is a deliberately over-weighted collection of the cases you are afraid of, and it is scored as a count of failures, not as a rate.

Should I grade test cases on the expected wording or the expected outcome?

Outcome, always. A case should record the decision the agent must reach, the action it must take or refuse, and whether it should escalate. It should not record a model answer to match. Suites graded on wording fail for the wrong reason: every prompt tweak, model upgrade, or brand-voice change lights up as a regression, the team learns the alerts are noise, and within two months nobody runs the suite. A suite dies of false positives long before it dies of neglect. Grade tone separately, as a sampled review, and keep it out of the pass or fail gate.

What is a realistic resolution rate for an AI support agent at first launch?

There is no universal number, which is the point of measuring your own. What is knowable is the baseline the market actually delivers. Gartner surveyed 5,728 customers in December 2023 and found that only 14% of customer service issues are fully resolved in self-service, and only 36% even for issues customers described as very simple.^[1] Marketed ceilings of 70% or 80% assume a clean, high-frequency, low-risk ticket mix. The honest first-launch target is a narrow band of intents where your own test set shows a high correct-resolution rate and a wrong-action count of zero, then widening one intent at a time. Richpanel guarantees 50% resolution in 30 days with money back if it misses, and that guarantee exists precisely because the number should be contractual rather than aspirational.

Should I run my human agents through the same test set as the AI?

Yes, and almost nobody does. Take 50 cases from the volume set, find what your agents actually did on those tickets, and score them on the same rubric with the same reviewers. Two things come out of it. First, you get a fair bar: human teams do not score 100%, and an AI held to a standard your own team does not meet will never launch. Second, you get the only comparison your CFO will believe, because it is measured on identical inputs. Run the same control again after launch on the intents the AI now owns, so the comparison stays live rather than becoming a launch-week artifact.

How often should the test suite run after the AI agent goes live?

On every change that could move an answer, not on a calendar. That means every knowledge-base edit, every policy change, every new or edited SOP, every tool or permission change, every model version change, and every vendor platform update. Most of those are made by your own team in the ordinary course of a Tuesday, which is why the real post-launch risk is not a new failure but an old test case that quietly stops passing. Re-score the sealed holdout monthly, and add every real production miss to the suite as a permanent case within a week so the suite grows toward your actual failures rather than your imagined ones.

Does shadow mode prove anything if the AI cannot take actions in it?

It proves half of what you need, and you should know which half. Shadow mode runs the agent on live traffic and records what it would have sent without sending it, so it gives you a real, unfiltered read on answer quality, escalation judgment, and the intent mix you actually receive, at zero risk to customers. What it cannot test is execution: you cannot shadow a refund, because either the money moves or it does not. So run the volume set in shadow and the risk set in a sandbox against real order data, or hold the action tests for the draft-for-approval stage where a human clears every action before it fires. Any vendor that presents shadow mode alone as proof the agent is safe to make autonomous is skipping the half that costs money.

Can I just use my vendor's built-in evaluation tooling?

Use it for the day-to-day improvement loop, and it is genuinely good on the better platforms. Do not use it as the only evidence for the shortlist decision. Two problems: a suite built inside one vendor's tooling usually cannot be run against a second vendor, so the comparison you most need is the one you cannot make; and a suite the vendor authored encodes the vendor's reading of your policy rather than yours, including the unwritten parts your team knows and your help center does not say. Keep the cases themselves in a format you own, a spreadsheet is fine, so they survive a vendor change. Then ask every vendor, including us, whether you can export the suite and the per-case results.

If you are still deciding which platforms belong on the shortlist at all, the vendor-by-vendor comparison is in the best AI agents for customer support, and the difference between a bot that answers and an agent that acts is unpacked in AI chatbot vs. AI agent.

Sources & references

Where the claims come from.

Inline citations [1] to [5] map to the entries below. Where a figure is ours rather than a published one, the entry says so and names the methodology.

Gartner, "Gartner Survey Finds Only 14% of Customer Service Issues Are Fully Resolved in Self-Service" (press release, 19 August 2024). Survey of 5,728 customers conducted in December 2023. Source of the 14% full-resolution figure and the 36% figure for issues customers described as very simple. gartner.com
Intercom / Fin help documentation, "Fin AI Agent resolutions" (accessed August 2026). Source of the confirmed versus assumed resolution definitions, the 24-hour disengagement window, the $0.99 per resolution rate, and the rule that a resolution is deducted if the customer returns to the same conversation. Quoted verbatim in the denominator section. fin.ai/help
Sierra, "The Agent Development Life Cycle" (company blog, accessed August 2026). Source for the description of CX teams annotating sampled conversations daily in Experience Manager, those annotations becoming the agent's regression tests, and Sierra running the regression suite for live customers on platform upgrades. sierra.ai/blog
Richpanel competitive field note, Decagon (July 2026). Observed inside a live Decagon account: self-serve evaluation tooling plus an automated QA pass that proposes its own fix as a reviewable diff. This is our observation of the product at a point in time, not a vendor statement, and product capabilities move. Verify at decagon.ai before relying on it.
Richpanel pre-launch evaluation methodology and production telemetry (2024 to 2026). Source of the sampling design, the five-dimension scoring rubric, the sealed-holdout discipline, and the wellness-brand production figures (AI sending 63% of customer messages at 4.39/5 CSAT against a team-wide 4.33, with routine conversations closed end to end). Underlying tenant data is anonymised and bound by NDA; aggregate figures are publishable and the customer approved the case study. Methodology in full at ai-hallucination-defense and case-studies/wellness. Methodology audit available on request via amit@richpanel.com.

Confidence-interval figures in the sampling section are standard binomial intervals at the worst case (p = 0.5), rounded to the nearest point, and reproducible with any statistics package. The cost-per-resolution worked example uses deliberately round, generic numbers and is not a claim about any named vendor's real re-contact rate. If a citation or a table cell here is materially wrong, email amit@richpanel.com and we will correct it, with the change recorded in the version history below.

Version history, v1.0 (2026-08-04): initial publication. Covers test-set sampling and sizing, test-case anatomy, the five metrics that replace deflection rate, regression triggers, the four-stage rollout, and the five questions for reading a vendor's claimed resolution rate. Vendor documentation cited is a snapshot as of August 2026.

Every resolution rate you have been shown was measured on someone else's tickets. Here is how to measure it on yours.