A/B Test Sample Size Calculator

The calculator uses the standard sample size formula for comparing two conversion rates. For each variant:

n = (zα/2 + zβ)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)²

• p₁ is your baseline conversion rate, as a fraction.
• p₂ is the rate you want to be able to detect: the baseline raised by the minimum detectable effect, so p₂ = p₁ × (1 + MDE).
• z_α/2 comes from the confidence level, two-sided: 1.645 at 90%, 1.96 at 95%, 2.576 at 99%.
• z_β comes from the power: 0.842 at 80%, 1.282 at 90%, 1.645 at 95%.

Worked example: the default inputs

A 5% baseline, a 10% relative lift, 95% confidence and 80% power:

• p₁ = 0.05 and p₂ = 0.05 × 1.10 = 0.055, so the difference to detect is 0.005.
• (1.96 + 0.8416)² = 2.8016² ≈ 7.849.
• p₁(1 − p₁) + p₂(1 − p₂) = 0.0475 + 0.051975 = 0.099475.
• n = 7.849 × 0.099475 ÷ 0.005² ≈ 31,231.0, rounded up to 31,232 visitors per variant.

Two variants need 62,464 visitors in total. That is the figure the calculator shows before you change anything.

Visitors needed per variant at 95% confidence and 80% power. Read across for the relative lift you want to detect.

Two patterns stand out. Halving the lift you want to detect roughly quadruples the sample. And a low-converting step, such as a 1% purchase rate, needs several times the traffic of a 10% step, such as a click through to pricing.

Turn the sample size into a duration before you launch:

days = visitors per variant × number of variants ÷ daily visitors who enter the test

Then round up to whole weeks. Weekday and weekend visitors behave differently, so a test that ends mid-week over-weights some days.

• With the defaults (62,464 visitors in total) and 2,000 eligible visitors a day: 31.2 days, so run for 5 weeks.
• With 5,000 eligible visitors a day: 12.5 days, so run for 2 weeks.
• With 800 eligible visitors a day: 78 days. That is a sign to test a bigger change or a busier page (see below).

Count only visitors who actually reach the page or step being tested, not total site traffic. A checkout test only enrolls people who reach checkout.

Even with a lot of traffic, run for at least one full week. Decide the end date up front and do not stop early because the result looks significant on day three. Checking repeatedly and stopping at the first significant reading inflates the false-positive rate well above the confidence level you chose.

Confidence protects against false positives: calling a winner when there is no real difference. At 95%, a test of two identical variants shows a “significant” result about 5% of the time.

Power protects against false negatives: missing a real effect. At 80% power, if the true lift is exactly your minimum detectable effect, the test detects it 8 times out of 10.

Raising either one costs traffic. With a 5% baseline and a 10% lift:

One-sided vs two-sided tests

The confidence level here is two-sided: the test can find the variant better or worse than control. A one-sided test at 95% uses z = 1.645, the same value as two-sided 90%, so it needs the same 24,601 visitors per variant. That saving is only honest if you would treat “worse” and “no different” the same way. If a losing variant would change what you do, keep the test two-sided.

• Relative vs absolute effect. The minimum detectable effect here is relative. Moving a 5% rate to 6% is a 20% relative lift (8,156 per variant). Entering 1 would mean 5% to 5.05%, which needs about 3 million per variant.
• Counting the wrong unit. The sample size counts the unit you randomize and convert. If you assign variants per visitor, count visitors, not sessions or pageviews. Repeat sessions from one person are not independent samples.
• More than two variants. The figure is per variant, so an A/B/C test needs three times it. Each extra comparison against control also raises the chance of a false winner; a common fix is to split the error rate across comparisons, for example testing each of two comparisons at 97.5% instead of 95%.
• Diluted traffic. If only some of the enrolled visitors ever see the change, the real effect is watered down and the test is underpowered. Enroll people at the point where the variants actually differ.
• Changing the test mid-run. Editing a variant, changing the traffic split or adding a variant halfway through restarts the clock.

• Test a bigger change. A redesigned pricing page can plausibly move conversion by 20–30%; a button color cannot. Larger effects need far smaller samples.
• Measure a higher-converting step. Test on the click through to signup rather than on paid conversion, then confirm the downstream effect over a longer period.
• Accept 90% confidence knowingly. It saves about a fifth of the traffic in exchange for more false positives. Write the trade-off down before the test starts.
• Do not A/B test at all. For small sites, watching session replays of people who drop off, and checking where the funnel leaks, usually finds the problem faster than a test that cannot reach significance.

When the test has run its full length, check the result with the A/B test significance calculator.