What Is Statistical Significance? Test Wins vs. Luck

What a P-Value Actually Measures

Every A/B test starts by assuming your two versions sell equally well. That's the null hypothesis. The p-value asks: if that's true, how often would luck alone produce a gap this big or bigger?

If the p-value is small, luck is a poor explanation. At the usual 5% cutoff, you accept getting fooled about 1 time in 20 when there's no real difference. A tool's "95% confidence" is usually that same cutoff, flipped.

Same Gap, Two Different Verdicts

Traffic matters as much as the gap. Say you test a new product page layout. The old page (A) converts at 2.0%, the new one (B) at 2.5%, so B looks 25% better.

Visitors per version	A orders	B orders	p-value	Significant at 5%?
1,000	20 (2.0%)	25 (2.5%)	About 45%	No
10,000	200 (2.0%)	250 (2.5%)	Under 2%	Yes

At 1,000 visitors each, B leads by just 5 orders. If A and B really sold the same, you'd see a gap that size about 45% of the time. At 10,000 each, B leads by 50. Luck alone does that less than 2% of the time. Same rates, just more evidence.

For sample size and test duration, see statistical significance in our Shopify A/B testing guide.

Significant Doesn't Mean Worth It

Passing the cutoff tells you luck is an unlikely explanation, not how big the difference is. With 200,000 visitors per version, 2.0% vs. 2.1% clears the 5% cutoff. That's one extra order per 1,000 visitors. It may not pay for the change.

Then there's the metric. A deeper discount can win on conversion rate and still lose on profit, so for offer tests, pick revenue per visitor or profit before the test starts.

Why Weekly Shopify Numbers Aren't a Test

Say your Shopify reports show 1.8% conversion last week and 2.3% this week. Significant? A calculator will give you a number, but it can't tell your change apart from an email send, a holiday or a new ad that week.

Significance is meant for a real A/B test: both versions live at the same time, visitors split at random. Testing apps show it as a p-value, a confidence percentage or a simple "significant" label.

What 95% Confidence Doesn't Mean

A p-value of 3% doesn't mean a 97% chance B truly wins. It means that if A and B were really equal, you'd see a gap this big about 3% of the time. And missing the cutoff doesn't prove they're equal. Your test may just be too small, like the 1,000-visitor one above.