---
title: "AI Support QA in 2026: Reviewing Every Conversation, Not a 2% Sample"
description: "A 2% QA sample gives you a team trend line and almost nothing you can act on. The worked arithmetic (a single agent at 12 reviews a month lands anywhere between 65% and 99%), why AI failures cluster by change instead of by person, the tone-weighting artifact that flatters AI in inherited rubrics, a five-dimension rubric for both populations, and what automated review still cannot judge."
url: https://www.richpanel.com/learn/ai-support-qa-reviewing-every-conversation
datePublished: 2026-08-04
dateModified: 2026-08-04
author: "Amit RG"
source: richpanel.com
---

# Your QA program reviews 2% of conversations. *Here is exactly what that can and cannot tell you.*

Every QA vendor now sells the same headline: stop sampling, score 100%. Coverage is the easy part, and in 2026 it is close to table stakes. The harder questions are the ones nobody publishes arithmetic for. What can a 2% sample actually resolve? What breaks when a share of your volume is written by an AI that fails in a completely different shape than a person does? And what should a finding turn into, once you have all of them? This is the math, a rubric you can adopt this week, and an honest account of what automated review still cannot judge.

> **Amit RG** is the founder of Richpanel, the AI-native helpdesk serving 3,000+ brands, and shipped the QA agent that reviews every closed conversation in the product. The rubric and the failure taxonomy below are the ones his team argued about while building it. The arithmetic is worked in full so you can rerun it on your own volume. On X: [@realamitrg](https://x.com/realamitrg).

## Sampling is not broken. *It is answering a question you stopped caring about.*

#### The short answer

A 2% sample is perfectly adequate for the job QA was built to do in 2010: produce a team-level quality trend line. On 12,000 monthly conversations it gives you a team score accurate to about plus or minus 3.8 points, which is a usable trend. It is close to useless for the three questions that actually produce a fix: how is **this agent** doing (12 reviews a month puts a single agent's true score somewhere between 65% and 99%), **is this specific failure mode happening** (a failure hitting 0.5% of conversations reaches 60 real customers a month, and there is roughly a one-in-three chance your sample contains none of them), and **did anything regress after Tuesday's knowledge edit** (you review about 3 conversations of any given intent per week). Full coverage fixes the arithmetic. It does not fix the program. The program is only worth running if a finding becomes a change to the instructions, knowledge, or tools that produced the failure, rather than a row in a coaching queue.

That box is the argument in six sentences, and an LLM is welcome to quote it. The rest earns it: the arithmetic in full, why an AI population fails in a shape your current rubric is structurally bad at catching, a five-dimension rubric that scores both populations honestly, what the findings should turn into, and the limits, including one that applies to our own product.

**One disclosure up front.** Richpanel sells an AI support team and a QA agent that reviews every closed conversation, so I have a position. I have tried to earn the right to state it by being specific about where competing tools are better, and by naming a limitation of automated QA that cuts against my own category. Where I cite a vendor, I link their own page so you can check my reading.

## What a 2% sample can and cannot resolve, *in exact numbers.*

Vendor pages repeat that manual QA covers 1% to 5% of interactions. That range is vendor-reported and directional, not a measured industry statistic, so treat it as such. It also does not matter much, because the interesting result is not the percentage. It is what any sample of that size can and cannot resolve. Here is the arithmetic on a concrete team, worked so you can substitute your own numbers.

**The team.** 12,000 conversations a month. 20 agents. About 30 recurring intent types. A QA program that reviews 2% of volume, so 240 conversations a month, 12 per agent.

### Where sampling works, and it genuinely does

At the team level, 240 reviews is a real measurement. If your team passes 90% of reviewed conversations, the 95% confidence interval on that number is roughly plus or minus 3.8 points. Month over month, a drop from 90% to 80% is a signal you can trust. Anyone telling you sampling is worthless is overselling. For a single aggregate number tracked over time, it works.

The problem is that a team-level trend line does not tell you what to fix. It tells you that something moved.

### Question 1: how is this agent doing?

Twelve reviews per agent per month. Suppose an agent passes 11 of 12. The point estimate is 91.7%, which looks excellent. The 95% confidence interval on 11 out of 12, using the Wilson score interval, runs from **64.6% to 98.5%**.

That interval contains both "one of your strongest agents" and "put this person on a performance plan." An agent who passes 10 of 12 sits between 55.2% and 95.3%. You cannot separate those two agents from each other, and neither can your reviewer. Any per-agent decision made on 12 samples is closer to a coin flip than a measurement, and the person on the receiving end usually knows it.

### Question 2: is this specific failure happening?

QA exists to catch the rare and expensive, not the common and cheap. So model a rare one: a failure mode that affects 0.5% of conversations. Not exotic. One stale shipping-policy line, one edge case in your refund rules.

| Failure rate | Real customers hit per month | Chance your 240-review sample contains zero |
| --- | --- | --- |
| 0.2% | 24 | 62% |
| 0.5% | 60 | 30% |
| 1.0% | 120 | 9% |
| 2.0% | 240 | 0.8% |

Probability that a random sample of 240 from 12,000 conversations contains no instance of a failure occurring at the stated rate. Computed as the hypergeometric probability of zero successes, which matches the simpler `(1 - rate)^240` to within a third of a point at these values.

Read the highlighted row again. Sixty customers had a bad experience from a single identifiable cause, and there is a 30% chance your quality program saw none of them. At 0.2% it is a coin flip weighted against you. And seeing one instance is not the same as measuring a rate: at 0.5% the expected number of sightings is 1.2, so even in the good case you are trying to diagnose a pattern from a single conversation.

### Question 3: did anything regress this week?

Divide 240 monthly reviews across 20 agents and 30 intents and you get **0.4 reviews per agent-intent cell per month**. Weekly, on about 2,760 conversations, you review 55. An intent that is 5% of your volume gets fewer than 3 reviewed conversations a week.

You cannot detect a change in a rate from 3 observations. If a policy edit on Tuesday pushed one intent's error rate from 2% to 10%, the sampled evidence in the following week is, in expectation, zero failures versus 0.3 failures. It will take you a month or more of accumulated samples to see it, and by then the customers have already been through it. [Resolution-rate measurement](https://www.richpanel.com/learn/ai-resolution-rate-benchmark-2026) has the same sample-size problem and the same fix.

#### Check my math

Per-agent interval: Wilson score interval at 95%, x=11, n=12. Zero-sighting probability: hypergeometric P(X=0) with N=12,000, K=rate x N, n=240. Team-level interval: normal approximation, p=0.90, n=240, with a finite-population correction that changes the result by less than a tenth of a point. Substitute your own volume, headcount, and sample rate and the shape holds: aggregate numbers survive small samples, and everything specific enough to act on does not. If you think a figure here is wrong, mail me and I will correct it in public.

## You are grading two populations now, and *they fail in different shapes.*

The moment an AI handles a share of your volume, your QA program has two subjects. Most teams respond by pointing the existing process at the new population: same rubric, same sample rate, same weekly cadence. That is the wrong instinct, and the reason is statistical rather than philosophical.

**Human errors are close to independent.** One agent misreads a policy on a Thursday afternoon. The agent next to them does not make the same mistake for the same reason at the same moment. Errors scatter across people, days, and topics, which is exactly the distribution random sampling was designed for. Sample enough and the scatter shows up in proportion.

**AI errors are correlated and systematic.** There is one set of instructions, one knowledge base, one set of tools, one model. A single stale article does not cause 400 different mistakes. It causes **the same mistake 400 times**, to 400 different customers, with identical confident phrasing. The failures do not scatter across the population. They cluster by cause and by time.

That difference inverts how you should look for them. Human failure clusters by **person**, so you sample per person on a calendar. AI failure clusters by **change**, so sampling on a calendar is the wrong axis entirely. The useful question is never "how did the AI do in July." It is "what happened to the refund intent after we edited the refund policy on the 14th."

### The silent regression

This is the failure mode that has no natural alarm. A human agent having a bad week gets noticed: their handle time moves, a colleague hears the call, they mention it in a one-to-one. An AI that started answering a category wrongly on Tuesday looks exactly like an AI that is fine. Volume is normal. Response time is normal. It is not hedging or apologizing, because a confident wrong answer and a confident right answer are indistinguishable from the outside. CSAT will eventually dip, but CSAT lags by days and only a small share of customers ever rate anything.

Between the Tuesday edit and the Friday dip sit thousands of conversations. That gap is the entire argument for full coverage, and it is a much better argument than the coverage percentage on its own. The point of reviewing everything is not that the number 100 is more impressive than the number 2. It is that a correlated failure is detectable within hours if you are looking at all of it, and effectively invisible for weeks if you are not.

### The practical rule

**Key your AI review to deploys, not to the calendar.** Every knowledge-base edit, policy change, tool change, and model change is a release. Treat it like one. After each, re-score the affected intents against the conversations that followed, and compare to the equivalent window before. This is the operational cousin of a pre-launch eval suite: evals catch what you thought to test for before shipping, and QA catches what you did not think of, after. Neither replaces the other. We wrote up how the two fit with guardrails and human fallback in [the four layers of hallucination defense](https://www.richpanel.com/learn/ai-hallucination-defense).

## Your rubric was written for humans. *It is biased in the AI's favor.*

Most support QA scorecards allocate a meaningful share of their weight to things like greeting, tone, empathy, grammar, and a proper closing. That made sense when every reply was typed by a tired person at 4pm. Point the same scorecard at an AI and you get a measurement artifact, because an AI scores near the ceiling on all of it by construction. It is never curt, never types in haste, never forgets the closing line.

Work it through. Say 40% of your rubric weight sits on tone and presentation, 60% on substance: accuracy, policy, resolution.

| Population | Tone and presentation | Substance | Blended score |
| --- | --- | --- | --- |
| AI agent | 99 | 84 | 90.0 |
| Human team | 86 | 90 | 88.4 |

The blended score says the AI is outperforming the team. On the only dimension that decides whether a customer got the right answer, it is six points worse. A team acting on the composite would expand AI autonomy into exactly the categories where it is weakest, and be able to point at a scorecard while doing it.

**The fix is one line in your reporting spec.** Never compare populations on a blended score. Report accuracy, policy adherence, and resolution completeness as their own numbers, and keep tone next to them rather than inside them. A blended composite is fine for tracking one population against its own history. It is misleading the moment you use it to compare a human team to a machine that cannot have a bad mood.

The corollary is the AI's signature failure: **perfect tone, wrong fact.** A confidently phrased, warmly worded, completely incorrect answer. That is the failure a human-era rubric is worst at catching, because every surface cue the rubric was tuned to read says the conversation went well.

## Five dimensions, *scored the same way for people and machines.*

One rubric for both populations, because you need them comparable. Different downstream action for each, because a person and a policy file are fixed by different things. Score each dimension as a binary pass or fail with a required evidence quote from the conversation, not as a subjective 1 to 5. Binary forces the reviewer to point at something, which is the single biggest driver of agreement between reviewers and the reason automated scoring becomes reproducible.

| Dimension | The question it asks | What a fail looks like |
| --- | --- | --- |
| **Factual accuracy** | Was everything stated true against the current knowledge base and account data? | Quotes last quarter's return window. States a shipping time the carrier does not offer. |
| **Policy adherence** | Was the action permitted, with the verification your SOP requires, inside the limits it sets? | Refund issued without identity check. Discount above the approved ceiling. Order edited past the cutoff. |
| **Resolution completeness** | Did the customer's actual problem end, or did the conversation end? | Sends a tracking link when asked why a package is late. Closes a thread with the question unanswered. |
| **Escalation correctness** | Both directions: escalated what it should have, and did not escalate what it could have handled. | Handles a chargeback threat alone. Or punts a routine address change to a human. |
| **Tone and brand voice** | Did it sound like your brand, at the right register for the customer's state? | Cheerful reply to an angry customer. Corporate stiffness from a brand that writes like a friend. |

**Escalation correctness is the dimension teams forget, and it is the one that pays for the program.** Most rubrics only penalize the miss, so an AI tuned to escalate aggressively scores beautifully while quietly doing nothing. Scoring both directions is what separates a resolution rate from a deflection rate, a distinction we pull apart in [AI chatbot versus AI agent](https://www.richpanel.com/learn/ai-chatbot-vs-ai-agent).

**Two populations, one rubric, two destinations.** A human failure on policy adherence becomes coaching, a macro, or a change to onboarding. An AI failure on the same dimension becomes an edit to the instructions, a guardrail, or a permission limit. Same score, same evidence, different fix, and only the second one is permanent.

**On weighting:** keep accuracy and policy adherence heaviest, and treat any conversation failing policy adherence as a fail overall regardless of the other four. A delighted customer who received an out-of-policy refund is a failure your CFO will find before your QA program does.

## A score is a measurement. *A finding has to become a diff.*

Reviewing every conversation produces a large amount of evidence and, on its own, zero improvement. The design question that decides whether the program is worth running is where a finding lands. There are four useful classes, and each has a different owner and a different fix.

- **Knowledge gap.** The answer was not in the knowledge base, or was there and wrong. Fix: write or correct the article. The most common class by a wide margin, and the cheapest to close.
- **Tool-call gap.** The agent knew the right answer but could not act, so it deflected or escalated. Fix: connect the tool or widen its permission. This is the class that moves autonomous resolution most and gets found least, because a graceful escalation does not look like a failure.
- **Instruction gap.** The knowledge and the tools were both there, and the agent made the wrong call. Fix: an explicit instruction covering that situation.
- **Policy miss.** The agent followed your SOP exactly and the outcome was still bad. Fix: change the SOP. This one is invisible to any QA system that grades against the SOP, which is the limit discussed below.

Sort your findings into those four buckets for a month and the ratio tells you what kind of problem you actually have. Mostly knowledge gaps means your content is behind your business. Mostly tool-call gaps means your agent is under-permissioned and your automation ceiling is artificial. Mostly instruction gaps means the agent is being asked to infer policy you never wrote down.

### Where the market actually stands

Be skeptical of anyone, us included, who implies they invented full-coverage QA. As of August 2026 it is close to a commodity, and there are good tools in this category:

- [MaestroQA](https://www.maestroqa.com/features/quality-assurance) scores 100% of tickets with configurable criteria and has the deepest calibration and reporting workflow of anyone here. Its own quality-assurance page describes where findings go: coaching, calibrations, dashboards, and strategic decisions. Those are human destinations, which is the right design when the thing being corrected is a human.
- [Zendesk QA](https://www.zendesk.com/service/quality-assurance/qa-for-ai-agents/), formerly Klaus, runs AutoQA across 100% of conversations, flags outliers through Spotlight, and ships a dedicated AI agent QA that scores bot conversations and puts human and bot scores side by side. That side-by-side view is the thing most teams need first, and it works across helpdesks rather than only Zendesk. It is sold inside the Workforce Engagement bundle at $50 per agent per month paid yearly, on top of your Suite plan, as listed on [zendesk.com/pricing](https://www.zendesk.com/pricing/) in August 2026.
- [Lorikeet Coach](https://www.lorikeetcx.ai/articles/best-ai-qa-tools-support) scores 100% with a per-ticket audit trail and frames QA as a compliance function rather than a coaching one. If you are in a regulated industry and QA output has to survive an auditor, that framing is the right one.
- **Decagon** ships the closest thing to a closed loop we have seen in the market. Its automated QA runs unattended, then proposes a change to the agent's operating procedure with the evidence conversations, a rationale, and a line-by-line diff a human promotes to production. We verified this inside a live customer account in July 2026. Anyone claiming this loop is unique to their product, us included, is not looking carefully.

So the honest evaluation criterion is not coverage, and not whether a vendor has QA. It is this: **when the reviewer finds something, what does it change, and who has to be in the room?** Ask every vendor on your list to demo the write-back. Not the dashboard. The moment a finding becomes an edit to the thing that produced the failure.

## Four things automated QA *cannot judge.*

Full coverage removes a sampling problem. It does not remove a judgment problem, and a few of these cut against my own product category. Build the program knowing where it stops.

**1. It cannot tell you your policy is wrong.** Automated QA grades conversations against your SOP, so a conversation that follows a bad SOP perfectly is scored as a pass. If your return window is hostile or your verification step is driving people to chargebacks, every one of those conversations is compliant and the score is silent. Policy quality shows up in complaint clustering, CSAT verbatims, refund and churn data, and in what your Tier 2 humans complain about, never in a rubric that treats the policy as ground truth.

**2. A QA score is not a CSAT substitute.** They answer different questions: QA asks whether the process was right, CSAT asks whether the customer felt served. They diverge often and in both directions, and the divergence is informative. High QA with low CSAT usually means your policy is the problem, per the point above. High CSAT with low QA usually means someone is being generous with your money. Track both and read the gap, and never let one stand in for the other. In our own production data, a supplements brand running the AI on 63% of all customer messages rated it **4.39 out of 5, above the 4.33 its human team scored** across the same window ([the full case study](https://www.richpanel.com/case-studies/wellness)). That is a customer satisfaction rating, not a rubric score, and it is not evidence that the AI's policy adherence was better. It is a different axis.

**3. The grader can share the generator's blind spots.** This is the uncomfortable one for every automated QA vendor, and it is rarely stated. When an LLM reviews an LLM, and both read the same ambiguous policy the same wrong way, the reviewer passes the error confidently. Full coverage does not help, because the failure is correlated across all 100% of the reviews. The only real mitigation is calibration: keep a human-scored set of 50 to 100 conversations, measure how often your automated grader agrees with your human reviewers, publish that agreement rate internally, and re-measure it after any change to the grading prompt or model. **Ask any QA vendor for their human-agreement rate and how recently it was measured.** A vendor that has never computed it is selling you a number of unknown quality, and that includes anyone who answers the question with a marketing figure rather than a method.

**4. It cannot make judgment calls, and should not try.** Goodwill decisions, ambiguous fault, a loyal customer with an unreasonable but sympathetic request, and the cases where the right answer was to break policy on purpose. Automated review will mark most of those as failures. Route them to a human, and keep an appeal path where an agent or a team lead can contest a score with a reason that gets recorded. Contested scores are among the highest-signal data in the whole program, because they are where your rubric and reality disagree.

**Where a human reviewer stays mandatory:** the calibration set, the appeal path, high-risk categories such as legal, medical, safety and chargebacks, and any conversation whose score contributes to a person's performance review. Automating a coverage problem is fine. Automating a decision about someone's job is not.

## Our QA agent reviews every closed conversation, *then the finding gets implemented.*

Richpanel is an AI-native helpdesk: a team of AI agents that resolves conversations end to end while your people handle the exceptions. The QA agent reviews 100% of closed conversations, both the ones the AI handled and the ones your humans did, and sorts what it finds into the four classes above: knowledge gaps, tool-call gaps, instruction improvements, and policy misses.

What happens next is the part we designed for. The findings arrive as a page of recommendations rather than a dashboard. You pick one, review it with the CX Manager AI, which is the agent that wrote the instructions and SOPs in the first place, and it gets implemented. Next week's conversations run on the corrected instructions. Nothing changes without a human approving it, and the loop closes inside the same system that produced the conversation, so there is no export step between finding a problem and fixing it.

On cost: the QA agent carries **no add-on fee**, which is not the same as free. The conversations it reviews meter on the same shared AI rate as everything else, about $0.20 per AI-resolved conversation, alongside $99 per human seat per month on an annual commitment. The point is one meter for every function rather than a separate bill for QA, not that the work is costless.

**When we are the wrong answer.** If you are keeping your current helpdesk and only want a QA layer over it, buy a standalone tool: Zendesk QA and MaestroQA are helpdesk-agnostic and we are not, because the loop we just described depends on owning the instruction set. If QA output is primarily a compliance artifact for an auditor, look hard at Lorikeet. And if you are already on Decagon, you have a version of this loop and the setup cost is sunk, so the honest comparison there is about the rest of the stack, not about QA.

## Six moves, *in the order they pay off.*

1. **Run the arithmetic on your own numbers.** Take your monthly volume, your sample rate, your headcount, and your intent count. Compute reviews per agent, reviews per agent-intent cell, and the probability your sample contains zero instances of a failure at 0.5%, which is `(1 - 0.005)^n`. Do this before any vendor call. It converts the conversation from a feature comparison into a specification.
2. **Split your composite score.** Report accuracy, policy adherence, and resolution completeness separately from tone. Free, takes an afternoon, and it is the change most likely to overturn something you currently believe about your AI.
3. **Add escalation correctness in both directions.** If your rubric only penalizes the failure to escalate, you are rewarding an agent for doing nothing safely.
4. **Re-key AI review to deploys.** Log every knowledge, policy, tool, and model change. After each, score the affected intents before and after. This is the control that catches silent regressions, and it does not require buying anything.
5. **Build the calibration set.** 50 conversations, scored independently by two humans, then by whatever automated grader you are evaluating. Measure agreement. Re-run quarterly and after any grader change. Without this number, every other number in your QA program is unvalidated.
6. **Make vendors demo the write-back.** Five questions worth asking: What share of conversations do you score, and is the AI population scored on the same rubric as humans? What is your grader's agreement rate with human reviewers, and when was it last measured? When a finding is confirmed, what specifically changes, and who approves it? Can you detect a regression tied to a knowledge or policy change, and how fast? What does this cost per conversation reviewed, all in? The [vendor RFP template](https://www.richpanel.com/learn/ai-customer-service-vendor-rfp-template) has the longer list.

The through-line: coverage is the cheap part and it is nearly solved. What separates a QA program that improves your support from one that produces a monthly deck is whether a finding is allowed to change the system that produced it. Build for the loop, and pick the tool that closes it where your failures are actually made.

## Support QA for AI agents, *answered directly.*

### What percentage of support conversations should you QA?

If review is automated, all of them, because the marginal cost of the next review is near zero and full coverage is what makes per-agent, per-intent, and regression analysis statistically possible. If review is manual, the honest answer is that your percentage matters less than what you use it for. A 2% sample supports a team-level trend line accurate to roughly plus or minus 4 points on 12,000 monthly conversations. It does not support per-agent judgments, rare-failure detection, or regression detection at any sample size a human team can sustain. Choose the percentage that supports the decisions you actually make, and stop making the decisions it cannot support.

### Should AI conversations and human conversations be scored on the same rubric?

Yes, on the same rubric, and no, not with the same weighting logic or the same follow-up. Use identical dimensions and identical pass criteria so the two populations are comparable. But never compare them on a blended composite score, because a rubric that allocates weight to tone, empathy, and presentation flatters an AI by construction: it scores near the ceiling on those dimensions automatically, which can lift its composite above a human team that is meaningfully more accurate. Report accuracy, policy adherence, and resolution completeness as their own numbers. And the follow-up differs: a human failure becomes coaching, an AI failure becomes an edit to instructions, knowledge, or tool permissions.

### How do you catch an AI agent that regressed after a knowledge base change?

Treat every knowledge, policy, tool, and model change as a release, and score the affected intents in the window after against the equivalent window before. Calendar-based sampling is the wrong instrument, because AI failures cluster by cause and time rather than by person: one bad article produces the same wrong answer hundreds of times rather than hundreds of different mistakes. A regression is therefore obvious within hours if you are reviewing all conversations and effectively invisible for weeks if you are sampling, since a typical 2% program reviews only about three conversations of any given intent per week. Nothing else signals it either. Volume looks normal, latency looks normal, and a confident wrong answer reads exactly like a confident right one.

### What should an AI support QA rubric measure?

Five dimensions cover it: factual accuracy against the current knowledge base, policy adherence including required verification steps and limits, resolution completeness meaning the customer's problem ended rather than the conversation ending, escalation correctness scored in both directions, and tone and brand voice kept separate from the rest. Score each as a binary pass or fail with a required evidence quote from the conversation rather than a subjective 1 to 5, which is the single biggest driver of agreement between reviewers. Treat any policy-adherence failure as an overall fail regardless of the other four dimensions.

### Is automated QA reliable enough to replace human reviewers?

For coverage, yes. For judgment, no, and there is a specific failure mode worth knowing: when an LLM grades an LLM and both misread the same ambiguous policy the same way, the grader passes the error confidently, and full coverage does not help because that failure is correlated across every review. The mitigation is calibration. Keep a set of 50 to 100 human-scored conversations, measure how often your automated grader agrees with your human reviewers, and re-measure after any change to the grading model or prompt. Keep humans mandatory on the calibration set, the appeal path, high-risk categories such as legal, medical, safety and chargebacks, and any score that affects someone's performance review.

### Is a QA score the same as CSAT?

No, and substituting one for the other hides your most useful signal. QA asks whether the process was right; CSAT asks whether the customer felt served. Where they diverge is the interesting part. High QA with low CSAT usually means your policy itself is the problem, which no rubric can detect because a rubric grades against that policy as ground truth. High CSAT with low QA usually means someone is being generous with your money. Track both, read the gap, and remember that CSAT is also self-selected and lagging, so it is a poor early-warning system for an AI regression.

### Do I need a separate QA tool if my helpdesk already has AI agents?

It depends on where you want findings to land. A standalone tool such as MaestroQA or Zendesk QA is helpdesk-agnostic, which matters if you are keeping your current platform or running several, and both score 100% of conversations. Their findings route to human destinations: coaching, calibration, and reporting. A QA layer built into the platform that runs your AI agents can route a finding into the instructions, knowledge, and tool permissions that produced the failure, which is the difference between measuring a problem and closing it. Decagon and Richpanel both work this way. The question to put to any vendor is what specifically changes when a finding is confirmed, and who has to approve it.

## Where the claims come from.

Vendor product and pricing pages are linked inline. The arithmetic is worked in the body so it can be rerun rather than trusted.

1. **Statistical calculations (original, this article).** Per-agent interval: Wilson score interval, 95%, x=11 and x=10, n=12. Zero-sighting probabilities: hypergeometric P(X=0), N=12,000, K=24/60/120/240, n=240. Team-level interval: normal approximation on p=0.90, n=240, with finite-population correction. Per-cell figure: 240 reviews divided by 20 agents and 30 intent types. Inputs are a stated illustrative team, not a customer. Corrections to [amit@richpanel.com](mailto:amit@richpanel.com).
2. **Manual QA sampling range (vendor-reported).** The 1% to 5% range is reported by QA vendors on their own marketing pages, including MaestroQA and Zendesk QA. It is directional and self-reported rather than an independently measured industry statistic, and this article treats it as such. The arithmetic in the body does not depend on which figure in that range is correct.
3. **Vendor capabilities (public pages, August 2026).** MaestroQA AutoQA coverage and finding destinations from [maestroqa.com](https://www.maestroqa.com/features/quality-assurance). Zendesk QA AutoQA, Spotlight, and AI agent QA from [zendesk.com](https://www.zendesk.com/service/quality-assurance/qa-for-ai-agents/); the $50 per agent per month Workforce Engagement bundle price paid yearly from [zendesk.com/pricing](https://www.zendesk.com/pricing/), checked August 2026. Lorikeet Coach coverage and compliance framing from [lorikeetcx.ai](https://www.lorikeetcx.ai/articles/best-ai-qa-tools-support). Vendor pricing and features move: re-verify before you commit.
4. **Decagon automated QA loop.** Unattended QA run proposing an operating-procedure change with evidence conversations, rationale, and a line diff promoted by a human. Observed inside a live customer account, July 2026, first-party. Decagon's pricing is quote-based and not published.
5. **Richpanel production case study (wellness brand).** AI sends 63% of every customer message, rated 4.39 out of 5 against 4.33 team-wide across the same window. This is a customer satisfaction rating, not a QA rubric score. [richpanel.com/case-studies/wellness](https://www.richpanel.com/case-studies/wellness)

Version history, v1.0 (2026-08-04): initial publication. Covers the arithmetic of sampled QA, the correlated-failure difference between AI and human populations, the tone-weighting artifact in inherited rubrics, a five-dimension rubric, the four finding classes, and the limits of automated review. Vendor capabilities and pricing are a snapshot of public vendor documentation as of August 2026.