- 1 Measure the baseline conversion rate
- 2 Pick the smallest lift worth finding
- 3 Calculate sample size per variant
- 4 Round the duration up to whole weeks
- 5 Run without stopping early
- 6 Judge once at the planned sample
The most common A/B testing mistake is not a bad hypothesis. It is watching the dashboard and stopping the moment one variant pulls ahead. Results jump around a lot early in a test, so stopping at the first significant-looking moment produces “winners” that disappear after you ship them. Fixing the sample size in advance removes that trap.
Why it works
Conversion data is noisy. With small samples, random swings make one variant look ahead, and those swings cross the significance line again and again during a test. If you check daily and stop at the first good-looking moment, you give chance many tries to fool you, so the real false-positive rate ends up well above the one your tool reports.
A fixed sample size gives the test a single decision point. The confidence level then means what it says, and the team can trust a win enough to build on it.
How to do it
- Find your baseline conversion rate for the exact step you are testing, for example, visitor-to-signup on the pricing page over the last four weeks.
- Choose the smallest lift worth detecting. Be honest: if a 5% relative lift would not change any decision, do not design the test to find one. Smaller lifts need dramatically more traffic.
- Calculate the sample size per variant at 95% confidence and 80% power. Those are sensible defaults for most product and marketing tests.
- Estimate the duration by dividing the total sample by your daily traffic to that page. Round up to whole weeks.
- Write the plan down before launch: metric, sample size, end date. Then don’t peek-and-stop.
- Evaluate once at the planned sample size and record the result, including losers and flat tests.
Example
A landing page converts 4% of visitors to signups. The team only cares about lifts of 20% or more (4% → 4.8%). At 95% confidence and 80% power, that needs roughly 10,000 visitors per variant. With 1,500 visitors a day split across two variants, the test needs about two weeks, which also neatly covers two full weekly cycles.
Metrics to track
- Share of tests that reach their planned sample. It should rise as the team stops ending tests early.
- Primary conversion rate per variant at the planned end date, read with its confidence interval.
- Conversion rate of shipped winners after rollout. If winners hold their lift, the process is working. If they fade, look for early stopping or a novelty effect.
- Actual test duration against the plan. Tests that keep overrunning usually mean traffic estimates were too optimistic.
When it backfires
- Low-traffic pages. If the required sample takes months, the test will be polluted by seasonality and other changes. Test bolder ideas or don’t test.
- Testing many variants at once. Each extra variant raises the chance of a false winner and splits your traffic further.
- Wrong metric. A test that lifts clicks but not revenue is not a win. Pick the metric closest to the business outcome your traffic can support.
Questions people ask
How long should an A/B test run?
Until each variant reaches the sample size you calculated in advance, and ideally for at least one full week so weekday and weekend behaviour are both included.
What if I do not have enough traffic for the sample size?
Test bigger changes (larger expected effects need far fewer visitors), test higher up the funnel where there is more traffic, or skip the test and make the change based on research instead.
Can I look at results while the test is running?
Yes, to catch bugs or broken tracking, such as a variant that records no conversions at all. Just do not stop the test or call a winner until the planned sample is reached.