A/B Test Significance Calculator

The calculator runs a two-proportion z-test, the standard test for comparing two conversion rates:

• Conversion rates: p₁ = conversions A ÷ visitors A and p₂ = conversions B ÷ visitors B.
• Pooled rate, assuming for the moment that A and B are the same: p = (conversions A + conversions B) ÷ (visitors A + visitors B).
• Standard error of the difference: SE = √[p(1 − p) × (1/n₁ + 1/n₂)].
• Test statistic: z = (p₂ − p₁) ÷ SE. The p-value is the chance of a z at least this far from zero if there were no real difference.

Worked example: the default inputs

• A: 250 conversions from 5,000 visitors = 5.00%. B: 300 from 5,000 = 6.00%. That is a 20% relative lift.
• Pooled rate: 550 ÷ 10,000 = 0.055.
• SE = √(0.055 × 0.945 × (1/5,000 + 1/5,000)) ≈ 0.00456.
• z = 0.01 ÷ 0.00456 ≈ 2.19, which gives a two-sided p-value of 0.0283.

0.0283 is below 0.05, so at 95% confidence B is a significant winner. At 99% confidence it is not: that needs p below 0.01.

• p-value. How surprising the gap would be if A and B were really identical. It says nothing about how big the effect is. A tiny, useless lift can have a tiny p-value on enough traffic.
• Lift. The relative change from A to B: (6.00% − 5.00%) ÷ 5.00% = +20% in the example. It is your best single estimate, but it is noisy.
• Confidence interval. The range of true differences that fit the data, in percentage points. For the example at 95% it runs from about +0.1 to +1.9 points: B is better, but the real gain could be anywhere from almost nothing to nearly double what you measured. Use it to decide whether the change is worth shipping, not only whether it is real.

If the interval crosses zero, the result is not significant at that confidence level. At 99% the example’s interval is −0.2 to +2.2 points, which is exactly why it fails there.

The same 5% → 6% result (a 20% lift), with different numbers of visitors in each variant:

A “not significant” result at 1,000 visitors does not mean B made no difference. It means the test was too small to tell. Work out the traffic you need before launching with the sample size calculator: a 20% lift on a 5% baseline needs 8,156 visitors per variant at 95% confidence and 80% power.

• You reached the planned sample. Stopping the first time p dips under 0.05 inflates false positives well beyond 5%. Read the result once, at the sample size you chose up front.
• The split is what you set. On a 50/50 test, 5,000 vs 4,700 visitors is too uneven to be chance (chi-square p ≈ 0.002). A sample ratio mismatch usually means a redirect, a bot filter or a tracking bug is dropping one variant’s visitors, and the conversion rates cannot be trusted.
• It ran whole weeks. Weekday and weekend visitors convert differently; a test that ends mid-week over-weights some days.
• The lift is worth having. Look at the low end of the interval. If the smallest plausible gain would not pay for the change, a significant result is not a reason to ship it.
• You did not go looking for it. Slicing a flat test by device, country and source until one segment turns significant is a reliable way to find a false positive. Test a segment only if you planned to.

• Check whether the test could have found the effect. If you had fewer visitors than the sample size calculator asks for, the test was underpowered and the honest verdict is “unknown”, not “no difference”.
• Read the interval. An interval of −0.3 to +2.3 points rules out a large loss but leaves a worthwhile gain possible. That can still justify shipping a low-risk change, as a judgment call rather than a test result.
• Do not keep extending the test. Adding traffic until it turns significant is peeking with extra steps. Plan a new test with a bigger change or a busier page instead.
• Find out why nothing moved. Watch session replays of visitors in each variant and check where the funnel loses them. A variant nobody noticed is a different problem from one people noticed and ignored.