Why this matters
Most experiment programmes fail long before the statistics. They fail at the sentence that starts the test. “Let’s try a new hero” gives you a result but no learning. If it wins, you do not know why. If it loses, you do not know what to try next.
A good hypothesis does three jobs. It forces you to show evidence for the idea, which kills weak ideas cheaply. It fixes the metric and the decision rule before you see any data, which protects you from fooling yourself. And it makes every result, win or loss, feed the next idea. Teams that write hypotheses well get more value per test slot, and test slots are the scarcest resource in growth. You only have so much traffic.
The Bing case study below shows the other side. Teams are bad at guessing which ideas matter. That is the reason to test at all, and it is why the framing of the test has to be clear enough that the data, not the loudest opinion, settles the question.
The framework
Every hypothesis has five parts. Write them in this order.
| Part | Question it answers | Example (D2C skincare) |
|---|---|---|
| Observation | What did you see in the data or research? | 61% of mobile visitors to the serum page never scroll past the first screen. |
| Change | What exactly will you change, and for whom? | Move ingredients and the skin-type match quiz into the first screen on mobile. |
| Expected effect | Which direction, and roughly how big? | Add-to-cart rate rises by at least 10% relative. |
| Metric | How will you measure it, primary and guardrails? | Primary: mobile add-to-cart rate. Guardrails: revenue per visitor, return rate. |
| Why | What evidence or mechanism makes you believe it? | Five of eight customer interviews said they buy only after checking ingredients. |
Primary metric versus guardrails
The primary metric decides the test. It should be the metric closest to customer value that still has enough volume to reach significance. For a signup page, completed signups beat CTA clicks. For a product page, add-to-cart or orders beat scroll depth.
Guardrail metrics do not decide the test. They can only veto it. Typical guardrails are revenue per visitor, average order value, refund or return rate, unsubscribe rate, page load time and support contacts. A test that lifts the primary metric while breaking a guardrail is not a win.
Diagnostic metrics explain why something happened. Clicks, scroll depth and time on page live here. Never let a diagnostic metric decide a test.
Minimum detectable effect
The minimum detectable effect (MDE) is the smallest real change your test can reliably detect, given your traffic, baseline rate and test duration. If the true effect is smaller than your MDE, the test will usually come back inconclusive.
This shortcut comes from statistician Evan Miller’s widely used guidance. The next-but-one lesson derives the full formula. For now, use it to sanity-check ideas before you build them.
Step-by-step playbook
- Start from evidence, not a brainstorm. Pull your funnel report and find the step with the largest absolute drop. Then gather qualitative signal on that step: five to ten session recordings, the last 50 support tickets, sales-call notes or a one-question on-site poll. Every hypothesis should cite at least one quantitative and one qualitative source.
- Name the mechanism. Write one line on why the change should work. Common mechanisms are clarity (users do not understand the offer), motivation (they do not want it enough), friction (it is too hard) and anxiety (they do not trust it). If you cannot name the mechanism, you are guessing.
- Choose the primary metric. Pick the event closest to value with at least a few hundred conversions per variant over the test window. Define numerator, denominator and attribution window in writing.
- List two to four guardrails. Always include one money metric (revenue per visitor or AOV) and one quality metric (return rate, refunds, early churn or support tickets).
- Compute the MDE before you build. Use your weekly traffic and baseline rate. If the detectable lift is 25% and your idea might plausibly move things 3%, either make the change bolder, test higher-traffic pages, or ship it without a test.
- Fix duration in whole weeks. Run at least one full week, preferably two, so every weekday is represented. Never stop on a lucky day.
- Write the decision rule. For example: ship if the primary metric wins at 95% confidence and no guardrail drops more than 2%. Iterate if it is directionally positive but inconclusive. Kill and log the learning if it is flat or negative.
- Log it. Record the hypothesis, screenshots, dates, sample sizes, result and the learning in a shared experiment log. The log is how your hit rate improves over time.
Weak vs strong hypotheses
Weak: Shorter signup form will increase signups.
Strong: Because 47% of visitors who start the signup form abandon at the company-size and phone fields, and 6 of 10 recorded sessions show hesitation there, we believe removing both fields and asking for them after activation will raise completed signups by at least 12% relative. Primary: completed signups per form view. Guardrails: sales-qualified lead rate and 14-day activation rate.
Weak: Adding reviews will boost sales.
Strong: Because 52% of product-page visitors open the reviews tab before adding to cart, and returns cite fit as the top reason, we believe showing a fit summary from reviews next to the size picker will raise add-to-cart rate by at least 8%. Primary: add-to-cart per product-page session. Guardrails: return rate and revenue per visitor.
Weak: Let's test annual pricing.
Strong: Because 70% of paying customers stay beyond 12 months but only 15% choose annual billing, we believe defaulting the toggle to annual with the monthly-equivalent price shown will lift the share of annual plans by 30% without lowering trial-to-paid conversion. Primary: annual share of new paid plans. Guardrails: trial-to-paid rate and refund requests in 30 days.
Weak: New checkout design.
Strong: Because shipping cost is first revealed at step three and 40% of abandoners in our exit survey cite surprise costs, we believe showing a delivery estimate and the free-shipping threshold on the cart page will raise checkout completion by at least 6%. Primary: orders per cart session. Guardrails: AOV and revenue per visitor.
Worked example
Say a B2B SaaS company has a demo-request page with 2,000 visitors a week and a 5% conversion rate to a booked demo. The team wants a four-week test with two variants.
- Visitors per variant over four weeks:
2,000 × 4 ÷ 2 = 4,000. - Variance term:
p × (1 − p) = 0.05 × 0.95 = 0.0475. - Plug into the MDE form:
δ ≈ √(16 × 0.0475 ÷ 4,000) = √(0.76 ÷ 4,000) = √0.00019 ≈ 0.0138. - So the test can detect an absolute change of about 1.4 percentage points, from 5.0% to 6.4%. In relative terms that is
1.38 ÷ 5 ≈ 28%.
What this means in practice: a headline tweak that might move conversion 5% will almost certainly be inconclusive. The team has three honest choices.
- Make the change bolder, for example replace the form with an instant calendar booking and a 90-second product video.
- Move the primary metric up to a higher-volume step, such as form starts, and keep booked demos as a guardrail.
- Ship small, low-risk improvements without a test and save the test slot for a bigger bet.
Now flip it. If the same page had 20,000 visitors a week, n would be 40,000 per variant and δ ≈ √(0.76 ÷ 40,000) ≈ 0.0044, a detectable relative lift of about 9%. More traffic buys you the right to test subtler ideas.
B2B SaaS vs D2C: what changes
| Factor | B2B SaaS | D2C / e-commerce |
|---|---|---|
| Traffic | Usually low, so MDEs are large | Often high, so smaller lifts are detectable |
| Best primary metric | Signup, activation or qualified demo, not paid conversion (too slow) | Orders per visitor or revenue per visitor |
| Key guardrails | Lead quality, activation, sales acceptance rate | AOV, return rate, discount usage |
| Evidence sources | Sales calls, onboarding analytics, churn interviews | Reviews, returns reasons, exit surveys, session replays |
| Typical bold change | Packaging, onboarding path, sales-assist routing | Offer framing, shipping threshold, bundle, payment options like COD or UPI |
For Indian D2C brands, include payment mix as a guardrail. A change that lifts orders but shifts more volume to cash on delivery can raise return-to-origin rates and hurt contribution margin.
How to measure it
- Primary conversion rate: unique visitors who complete the target event in the session or within a fixed window (for example 7 days) divided by unique visitors assigned to the variant. Count by user, not by pageview.
- Revenue per visitor: total revenue from assigned users in the window divided by assigned users. This catches AOV and conversion trade-offs in one number.
- Experiment win rate: tests that shipped as clear wins divided by tests concluded, per quarter. Mature programmes often see only a minority of tests win. A very high win rate usually means weak statistics or timid ideas.
- Learning rate: tests that produced a documented learning (win or loss) divided by tests concluded. Aim for this to be close to 100%. A strong hypothesis makes it possible.
Segment results only by dimensions you named in advance, such as device or new versus returning. Slicing after the fact will always find a “winning segment” by chance.