A/B Testing Statistics That Hold Up

Why this matters

An A/B test is a measuring instrument. If the instrument is miscalibrated, every decision built on it is wrong, and you will not know. A team shipping “wins” that are really noise will see its dashboard full of green results while the business metric stays flat. That gap destroys trust in experimentation faster than any losing test.

Most statistical failures in growth teams come from a handful of causes: too little sample, stopping early, broken randomisation, too many comparisons and novelty effects. Each one is easy to prevent once you know what to look for. This lesson gives you the arithmetic and the checks. For the button-pressing part, use the A/B test calculator.

The framework

Every test balances four numbers. Fix three and the fourth is determined.

A p-value is the probability of seeing a difference at least this large if the change truly had no effect. It is not the probability that your variant is better. A p-value of 0.03 means “this would be surprising if nothing were going on”, nothing more.

The sample you need grows with the square of how precise you want to be. Halving the MDE needs four times the traffic. That single fact explains why low-traffic sites cannot test small tweaks.

Where you put the primary metric matters just as much. Each funnel step shrinks the volume you can test on. Drag the sliders to see how few orders a 100,000-visitor month really produces.

Step-by-step playbook

1. Get the baseline right. Pull the last four full weeks of the primary metric for exactly the audience the test will include. Exclude internal traffic and bots. Use users, not sessions, as the unit if you randomise by user.
2. Choose the MDE from value, then check feasibility. Ask what lift would justify the cost of shipping and maintaining the change. Then compute sample size. If the duration is over six to eight weeks, change the test, not the statistics.
3. Round duration up to whole weeks. Weekday and weekend behaviour differ, especially for B2B. Minimum one week; two is safer.
4. Decide your stopping rule before launch. Either fixed horizon (look once at the end) or a sequential design with pre-set looks and boundaries. Write it in the test brief.
5. Run an A/A or pilot first on new set-ups. When you change tools or tracking, run control versus control for a week. It should show no significant difference and no SRM.
6. Check SRM on day one and at the end. Compare observed assignment counts with the configured split using a chi-square test.
7. Read the primary metric once, then guardrails, then segments you pre-registered. Report the confidence interval, not just the p-value. “Plus 4%, interval plus 1% to plus 7%” says more than “significant”.
8. Check for novelty. Plot the daily effect. If it starts high and decays, extend the test or analyse only users who were new during the test.

Worked example

Sample size

Say a D2C apparel brand converts 4.0% of checkout-page visitors and wants to detect a 10% relative lift, to 4.4%. It gets 3,000 eligible visitors a day.

• p₁ = 0.040, p₂ = 0.044, so δ = 0.004.
• Variance terms: 0.04 × 0.96 = 0.0384 and 0.044 × 0.956 = 0.042064. Sum = 0.080464.
• Multiply by 7.84: 0.080464 × 7.84 = 0.63084.
• Divide by δ² = 0.000016: 0.63084 ÷ 0.000016 ≈ 39,428 visitors per variant.
• Total for two variants: 78,856. At 3,000 a day: 78,856 ÷ 3,000 ≈ 26.3 days.
• Round up to whole weeks: 28 days.

The shortcut gives 16 × 0.0384 ÷ 0.000016 = 38,400 per variant, within 3% of the exact figure. Good enough for planning.

If the brand wanted to detect a 5% lift instead, δ halves to 0.002 and the sample quadruples to roughly 155,000 per variant, more than three months of traffic. That test should not run.

SRM check

At the end, the 50/50 test shows 50,600 users in control and 49,400 in treatment, out of 100,000.

A 1.2% imbalance sounds trivial, but it is far too large to be chance at this sample size. Something removed users unevenly: a redirect that loses some visitors, a variant that breaks on one browser, or bot filtering that reacts to the change, as in the MSN carousel case below.

Peeking and sequential testing

Fixed-horizon statistics assume you look once. Each extra look is another chance for noise to cross the line. Statistician Evan Miller showed that if you stop a no-difference test at the first significant reading, the real false-positive rate in his worst-case example climbs to 26.1%, more than five times the nominal 5%. He also calculated that with 10 interim looks, you would need a per-look threshold of roughly 1% to keep the overall rate near 5%.

You have three honest options:

• Do not peek at significance. Watch only SRM and guardrails for breakage.
• Group sequential design. Pre-plan, say, three looks with stricter boundaries early (O’Brien-Fleming style) and the normal threshold at the end.
• Always-valid inference. Some platforms use methods such as the mixture sequential probability ratio test that stay valid at any look. Read your tool’s documentation to know which you have.

Multiple comparisons, novelty and when not to test

Multiple comparisons. With four variants each compared to control at 5%, the chance of at least one false winner is 1 − 0.95⁴ ≈ 18.5%. Track ten metrics and it is 1 − 0.95¹⁰ ≈ 40%. A Bonferroni correction for ten metrics uses 0.05 ÷ 10 = 0.005 per metric. It is conservative but simple. Guardrails do not need correction if they can only veto, not approve, a launch.

Novelty and primacy. Returning users react to anything new, and some resist anything different. Both fade. A new recommendation widget may get clicks for a week because it is new. A redesigned navigation may dip for a week while people relearn it. Check the daily effect and segment by new versus returning users.

When not to A/B test:

• Traffic cannot reach your MDE in about six to eight weeks.
• The change is a bug fix, legal requirement or clear accessibility improvement.
• The decision is irreversible or strategic, such as a rebrand or a price change for existing customers, where network or press effects contaminate groups.
• Users can see each other’s variants, as in marketplaces or social products. Use cluster or geo randomisation instead.
• You would ship it regardless of the result. Then just ship and monitor.

B2B SaaS vs D2C: what changes

Avoid testing during major sale events such as Diwali or Black Friday unless the test is about the sale itself. Behaviour is unusual and results rarely generalise.

How to measure it

• Sample size per variant: users assigned to each group at analysis time, counted once each, by the randomisation unit.
• Conversion rate per variant: users with at least one target event within the attribution window, divided by users assigned. Use an intention-to-treat view: everyone assigned counts, whether or not they saw the change.
• Relative lift with 95% interval: (treatment rate − control rate) ÷ control rate, reported with its confidence interval.
• SRM rate: tests failing the SRM check divided by tests run, per quarter. If yours is far above the roughly 6% Microsoft reported, audit your set-up.
• Post-launch holdback: for important wins, keep 5 to 10% of users on control for a few weeks after launch to confirm the effect persists.