The math nobody publishes
What a 2% sample can and cannot resolve, in exact numbers.
Vendor pages repeat that manual QA covers 1% to 5% of interactions. That range is vendor-reported and directional, not a measured industry statistic, so treat it as such. It also does not matter much, because the interesting result is not the percentage. It is what any sample of that size can and cannot resolve. Here is the arithmetic on a concrete team, worked so you can substitute your own numbers.
The team. 12,000 conversations a month. 20 agents. About 30 recurring intent types. A QA program that reviews 2% of volume, so 240 conversations a month, 12 per agent.
Where sampling works, and it genuinely does
At the team level, 240 reviews is a real measurement. If your team passes 90% of reviewed conversations, the 95% confidence interval on that number is roughly plus or minus 3.8 points. Month over month, a drop from 90% to 80% is a signal you can trust. Anyone telling you sampling is worthless is overselling. For a single aggregate number tracked over time, it works.
The problem is that a team-level trend line does not tell you what to fix. It tells you that something moved.
Question 1: how is this agent doing?
Twelve reviews per agent per month. Suppose an agent passes 11 of 12. The point estimate is 91.7%, which looks excellent. The 95% confidence interval on 11 out of 12, using the Wilson score interval, runs from 64.6% to 98.5%.
That interval contains both "one of your strongest agents" and "put this person on a performance plan." An agent who passes 10 of 12 sits between 55.2% and 95.3%. You cannot separate those two agents from each other, and neither can your reviewer. Any per-agent decision made on 12 samples is closer to a coin flip than a measurement, and the person on the receiving end usually knows it.
Question 2: is this specific failure happening?
QA exists to catch the rare and expensive, not the common and cheap. So model a rare one: a failure mode that affects 0.5% of conversations. Not exotic. One stale shipping-policy line, one edge case in your refund rules.
| Failure rate |
Real customers hit per month |
Chance your 240-review sample contains zero |
| 0.2% | 24 | 62% |
| 0.5% | 60 | 30% |
| 1.0% | 120 | 9% |
| 2.0% | 240 | 0.8% |
Probability that a random sample of 240 from 12,000 conversations contains no instance of a failure occurring at the stated rate. Computed as the hypergeometric probability of zero successes, which matches the simpler (1 - rate)^240 to within a third of a point at these values.
Read the highlighted row again. Sixty customers had a bad experience from a single identifiable cause, and there is a 30% chance your quality program saw none of them. At 0.2% it is a coin flip weighted against you. And seeing one instance is not the same as measuring a rate: at 0.5% the expected number of sightings is 1.2, so even in the good case you are trying to diagnose a pattern from a single conversation.
Question 3: did anything regress this week?
Divide 240 monthly reviews across 20 agents and 30 intents and you get 0.4 reviews per agent-intent cell per month. Weekly, on about 2,760 conversations, you review 55. An intent that is 5% of your volume gets fewer than 3 reviewed conversations a week.
You cannot detect a change in a rate from 3 observations. If a policy edit on Tuesday pushed one intent's error rate from 2% to 10%, the sampled evidence in the following week is, in expectation, zero failures versus 0.3 failures. It will take you a month or more of accumulated samples to see it, and by then the customers have already been through it. Resolution-rate measurement has the same sample-size problem and the same fix.
Check my math
Per-agent interval: Wilson score interval at 95%, x=11, n=12. Zero-sighting probability: hypergeometric P(X=0) with N=12,000, K=rate x N, n=240. Team-level interval: normal approximation, p=0.90, n=240, with a finite-population correction that changes the result by less than a tenth of a point. Substitute your own volume, headcount, and sample rate and the shape holds: aggregate numbers survive small samples, and everything specific enough to act on does not. If you think a figure here is wrong, mail me and I will correct it in public.