EvalPower

AI Evaluation Sample Size & Budget Calculator

"It looks better on 50 examples" isn't evidence. Here's the sample size that would be.

Current success rate (%)

Lift you need to detect (pp)

Confidence level (%)

Statistical power (%)

Slices evaluated separately

Cost per model call ($)

Samples needing human review (%)

Cost per human review ($)

Samples needed per arm

2,626

Sample size scales with the inverse square of the lift you want to detect — halving the detectable difference roughly quadruples the samples. Decide the smallest lift that would actually change your decision before you spend anything.

Total samples: 5,252
Total evaluation cost: $1,681
Human review cost: $1,576

What you entered

Current success rate (%): 80Lift you need to detect (pp): 3Confidence level (%): 95Statistical power (%): 80Slices evaluated separately: 1Cost per model call ($): 0.02Samples needing human review (%): 20Cost per human review ($): 1.5

1
Comparison being powered
80% baseline vs 83.0% target= 3 pp lift
2
Samples per arm at 95% confidence / 80% power
(z₍α₎+z₍β₎)² × (p₁q₁+p₂q₂) ÷ Δ²= 2,626 per arm
3
Total samples (2 arms × 1 slice)
2,626 × 2 × 1= 5,252 samples
4
Model call cost
5,252 × $0= $105
5
Human review (20% of samples)
1,050 × $2= $1,576 → total $1,681

Result

Samples needed per arm: 2,626

To detect a 3 pp change from a 80% baseline at 95% confidence and 80% power you need 2,626 samples per arm — 5,252 in total, costing about $1,681 ($1,576 of that human review).

Eval samples needed to detect a quality change

How many labelled examples you need to tell two model versions apart, at 80% power. Smaller lifts need dramatically more samples, and every extra slice multiplies the total.

Baseline	Lift to detect	Confidence	Slices	Per arm	Total samples	Eval cost
70%	10 pp	95%	1	291	582	$186
80%	5 pp	95%	1	903	1,806	$578
80%	5 pp	99%	1	1,344	2,688	$860
80%	3 pp	95%	1	2,626	5,252	$1,681
80%	5 pp	95%	4	903	7,224	$2,312
90%	2 pp	95%	1	3,211	6,422	$2,055

Halving the lift you want to detect roughly triples the samples. Cost assumes $0.02 per model call with 20% of samples going to human review at $1.50 each.

An AI evaluation sample size calculator answers a question teams routinely skip before shipping a model change: how many test cases are actually needed to tell a real improvement from noise, and what will running that evaluation cost? The common practice — trying a new prompt or model on a few dozen examples, eyeballing the outputs, and declaring it better — has no statistical standing. With small samples, differences of several percentage points appear and vanish purely by chance, which is how teams ship regressions while believing they shipped improvements.

The calculation applies the standard test for comparing two proportions. Given your current success rate, the smallest lift you would want to detect, a confidence level, and statistical power, it computes the samples required per arm using the two-proportion formula, then multiplies by two arms and by the number of slices you evaluate separately. That slice multiplier matters more than it first appears: if you need per-language, per-customer-segment, or per-difficulty-tier confidence rather than one blended number, each slice needs its own adequately powered sample, and the total multiplies accordingly. Cost is then built from two components — model calls across all samples, and human review on the share of samples a grader must judge, which is usually the dominant expense because human time costs far more per item than inference.

The most important property to internalize is that sample size scales with the inverse square of the effect you want to detect. Halving the detectable lift roughly quadruples the samples and therefore the budget. Detecting a ten-point improvement is cheap; detecting a one-point improvement is enormously expensive. This is why the first question should never be 'how many samples can we afford' but 'what is the smallest change that would actually alter our decision?' If a two-point gain would not change whether you ship, there is no reason to fund an evaluation powered to find it. The source research frames this exactly as joining statistical power with grader cost, human adjudication, slices, and regression frequency — and the frequency point compounds, since an evaluation you run on every release multiplies this cost across the year. Practical mitigations include using an LLM judge calibrated against human labels to cut the human-review share, reusing a fixed benchmark set across releases, and reserving full-power evaluation for changes that genuinely warrant it.

Precision is quadratically expensive

Sample size scales with 1/Δ², so halving the lift you want to detect quadruples the samples and the budget. Detecting a 10-point change might take a few hundred samples; detecting 1 point can take tens of thousands. Always start by asking what the smallest decision-relevant difference is — funding precision beyond that is pure waste.

Human review usually dominates the bill

Model calls are cheap per sample; a human grader is not. If 20% of samples need human judgment at $1.50 each, that line typically exceeds total inference cost by a wide margin. Calibrating an LLM judge against a human-labelled subset, then using it for the bulk of grading, is the standard way to cut this without abandoning rigour.

Frequently asked questions

I want to detect a 3-point lift from an 80% baseline at 95% confidence and 80% power — how many samples?

About 2,626 samples per arm, so roughly 5,250 in total for a single slice. At $0.02 per model call plus human review on 20% at $1.50, that's around $105 in inference and $1,576 in review — approximately $1,680 for one properly powered comparison.

Why can't I just test on 50 examples?

Because with 50 samples the confidence interval around an 80% success rate spans roughly ±11 points. An observed 5-point 'improvement' sits comfortably inside noise — you'd see swings that large from random sampling alone. Small-sample eyeballing tells you whether something is catastrophically broken, not whether it's better.

How do I reduce the cost without losing rigour?

Widen the detectable lift to the smallest difference that would actually change your decision — that's quadratic savings. Calibrate an LLM judge against a human-labelled subset and use it for bulk grading. Reuse a fixed benchmark set across releases. And reserve fully powered evaluations for changes that genuinely warrant them.

Why does the number of slices multiply the cost?

Because a sample powered for your overall population isn't powered for each subgroup within it. If you need separate confidence per language, customer tier, or difficulty band, each slice requires its own adequately sized sample — so five slices is roughly five times the samples and cost of a single blended comparison.