Most B2B A/B tests are too small to mean anything

Someone on your team ran an A/B test on the pricing page. Three weeks in, the new version is "winning" at 95% significance, and a slide says demo requests are up 40%. Before anyone ships it, ask how many visitors that test needed to find a real effect. On a page with a few thousand visitors a month, that can top a year's traffic.

That's no reason to stop experimenting. It's a reason to size every test first.

How many visitors does an A/B test need?

Four inputs decide A/B testing sample size: your baseline conversion rate, the smallest lift worth detecting (the minimum detectable effect), the significance level, and power, the chance of catching a real effect that size. Industry defaults are 5% significance and 80% power.

With those defaults, Ron Kohavi, Alex Deng and Lukas Vermeer give a rule of thumb in "A/B Testing Intuition Busters," their KDD 2022 paper. Each variant needs about 16 * p * (1 - p) / d^2 users, where p is the baseline rate and d is the absolute change you want to detect. Their worked example: a 3.7% conversion rate and a 10% relative lift (3.7% to about 4.07%) need 41,642 users per variant. The test behind the example ran about 80 users per variant, had roughly 3% power for that lift, and reported a 337% lift.

Note the square: halve the lift you want to detect and you need four times the traffic.

What does that look like on a B2B page?

Now a page like yours, as arithmetic, not anyone's result. Say your pricing page gets 3,000 visitors a month and 2% request a demo. Split two ways, that's 1,500 per variant a month.

A 10% lift (2.0% to 2.2%) needs about 78,000 visitors per variant: more than four years. Stop at six months and you'd have roughly a one-in-six chance of catching a real one.
A 30% lift (2.0% to 2.6%) needs about 8,700 per variant: roughly six months.
A metric higher up the funnel is different. If 20% of visitors click through to the demo form, a 20% relative lift on that click (20% to 24%) needs about 1,600 per variant: about five weeks.

All three use the same rule of thumb; a stricter two-proportion calculation lands up to 12% higher. The KDD paper's guidance: A/B tests are useful when you have at least thousands of active users, preferably tens of thousands.

What goes wrong when you run the test anyway?

Underpowered winners are often false or inflated. The KDD paper lists published success rates, the share of ideas that improved the target metric, from about a third at Microsoft to 8% at Airbnb Search. The authors call these ballpark estimates, from companies with far more traffic than yours. By their math, at 20% power and a 10% success rate like Booking.com's or Netflix's, more than half of statistically significant results would be false positives, and real effects that reach significance in low-power tests tend to come out exaggerated. A 40% lift on 3,000 visitors a month should make you more suspicious, not less.

Peeking breaks the math. A standard p-value assumes you fixed the sample size in advance. Stop the first time the dashboard says significant and false positives climb. In "How Not To Run an A/B Test," a 2010 essay, Evan Miller works through a worst case: a change that does nothing, tested after every visitor up to 150. It declares a winner 26.1% of the time, not 5%. Ramesh Johari, Leo Pekelis and David Walsh, in an arXiv paper (2015, revised 2019), call standard p-values "wholly unreliable" under continuous monitoring. Their always-valid p-values allow checking as you go and ran on a large commercial platform across hundreds of thousands of experiments.

How do you run experiments with low traffic?

Five moves, in the order we'd try them.

Test bigger changes. If you can only detect a 30% lift, only test ideas that could produce one: a shorter demo form, a new pricing structure, a rebuilt onboarding flow, a real redesign of the page. Button colors won't. For calibration, the KDD paper calls 10% from one experiment large and says Bing's monthly revenue gains from many experiments were usually in the low single digits.
Move the metric up the funnel. Clicks to the demo form, onboarding steps completed, a key feature used in week one. Higher base rates need far smaller samples. The catch: a proxy only helps if it moves with the outcome you care about, so check your history first. A demo request isn't pipeline either, as our demand generation post argues.
Randomize by account in product tests. Users inside one customer account share screens, settings and opinions. Split a team between two workflows and the groups can contaminate each other. Assigning whole accounts avoids that. It's design reasoning, the subject of chapters 14 ("Choosing a Randomization Unit") and 22 ("Leakage and Interference between Variants") of Kohavi, Tang and Xu's 2020 book Trustworthy Online Controlled Experiments. The cost: your sample is now accounts, not users.
Cut variance with pre-experiment data. CUPED, which Microsoft researchers published at WSDM 2013, adjusts each user's result using pre-experiment data, typically the same metric from before the test. At Bing it cut variance by about 50%: the same power with half the users or half the duration. The gain depends on how well the before-period predicts the test, so it's strong for logged-in product usage and has nothing to work with for first-time visitors. Never use a covariate the change can affect; the paper shows one reversing a known result.
Write the stopping rule before launch. A fixed sample size you wait for, or a sequential method designed for peeking. Never "we'll see."

When should you skip the A/B test?

When the arithmetic says years, the test is theater. Make the change on judgment and user research, write down what you expect, and watch the metric, knowing a before-and-after comparison can't separate your change from seasonality. An honest "we decided" beats a fake "the data proved."

What goes in a test plan?

One page, signed off before the feature flag flips:

Primary metric and its event, defined once and landing in your warehouse the same way for both variants.
Baseline rate from at least a month of history.
Minimum detectable effect, significance and power, and why that effect matters to the business.
Sample size and duration, rounded up to whole weeks so weekdays and weekends both count.
Randomization unit and stopping rule.

A plan that comes back with "two years" did its job.

Run fewer, bigger tests

Most B2B test ideas belong to Marketing and Studio: landing pages, the pricing page, onboarding. Build ships the feature flags that assign variants, and Data sizes the test before launch and reads it after. Running fewer, bigger tests is a planning decision across all four, not a tool setting. And if your tools disagree on conversion counts, fix the numbers first.