SigmaCalc

A/B Testing Calculator

Find out whether your variant's improvement is real, or just noise from a small sample.

Visitors — A (control)

Conversions — A (control)

Visitors — B (variant)

Conversions — B (variant)

Rate A: 5%
Rate B: 6.5%
Lift: +30%

p-value

0.15

Not statistically significant at α = 0.05

What you entered

A: 50/1000B: 65/1000

1
Conversion rate A
50 ÷ 1000= 0.05
2
Conversion rate B
65 ÷ 1000= 0.065
3
Pooled conversion rate
(50 + 65) ÷ (1000 + 1000)= 0.0575
4
Standard error of the difference
√[p(1−p)(1/1000 + 1/1000)]= 0.0104
5
z-statistic
(0.065 − 0.05) ÷ 0.0104= 1.4408
6
Two-tailed p-value
from z = 1.4408= 0.1496

Result

p-value: 0.15

Variant B converts at 6.5% versus 5% for A, a 30% lift — with a p-value of 0.15, that difference is not statistically significant at the conventional 0.05 threshold.

A/B test significance at common conversion rates

Each row compares two variants using a two-proportion z-test. Lift is the relative improvement of B over A.

A conversions / visitors	B conversions / visitors	Rate A	Rate B	Lift	Z-score	P-value	Significant?
100 / 1,000	120 / 1,000	10.00%	12.00%	20.00%	1.4293	0.1529	No
200 / 2,000	210 / 2,000	10.00%	10.50%	5.00%	0.5213	0.6022	No
250 / 5,000	300 / 5,000	5.00%	6.00%	20.00%	2.1932	0.0283	Yes
50 / 500	75 / 500	10.00%	15.00%	50.00%	2.3905	0.0168	Yes

Rows one and three are the pair worth studying: both show a 20% lift, but only the second is significant, because 5,000 visitors per arm provides the evidence that 1,000 does not. A 20% lift on 1,000 visitors each way returns p = 0.1529 and would be a mistake to ship. Row four shows the opposite case - a large enough effect can reach significance on only 500 per arm. Decide the sample size before starting and stop at it: checking repeatedly and stopping when p first drops below 0.05 inflates false positives badly. Significance is also not the same as a worthwhile effect.

Don't peek too early

Checking results daily and stopping as soon as you see significance inflates your false-positive rate dramatically — decide on a sample size or time window in advance and stick to it, rather than stopping the moment p dips below 0.05.

Statistical significance isn't practical significance

A tiny, real lift can still be statistically significant with enough traffic, but not worth the engineering cost to ship. Always weigh the lift's size and business impact alongside the p-value, not instead of it.

Frequently asked questions

Version A: 120 conversions out of 3,000 visitors. Version B: 150 out of 3,000. Is B better?

A = 4.0%, B = 5.0%, lift = +25%. The z-test gives p = 0.047 — just barely significant at the 0.05 level. With borderline results like this, consider running longer for more confidence or evaluating whether a 1 percentage point lift justifies the change.

How long should I run an A/B test?

Until you reach your pre-determined sample size, NOT until you see significance. Use the sample size calculator first: decide on minimum detectable effect, significance level (usually 0.05), and power (usually 0.80). Typical web tests need 1,000–10,000+ visitors per variant.

How is this different from the chi-square calculator?

Both test whether the difference between groups is real. This calculator is tailored for conversion rates (2 variants, one metric), shows lift percentage and practical framing. The chi-square calculator handles any categorical comparison, including 3+ groups.

What is 'statistical power' and why does it matter?

Power is the probability of detecting a real effect when one exists — typically set to 80%. Low power means you might run a test, see no significance, and wrongly conclude there's no difference when there actually is one. Increase power by increasing sample size.