How long do I need to run an A/B test to get reliable results?

Question

As an e-commerce manager, I'm struggling to determine the optimal duration for A/B testing my promotional strategies. I want to ensure my test results are statistically significant and provide meaningful insights, but I'm unsure about the right sample size, duration, and metrics to track. How long should I run my A/B tests to get reliable, actionable data that can genuinely improve my conversion rates and marketing performance?

Muhammed Tüfekyapan · Accepted Answer

How Long Do I Need to Run an A/B Test to Get Reliable Results?
Most A/B tests are stopped too early. Early results are almost always misleading because early visitors are not representative of your typical traffic - weekday vs. weekend visitors behave differently, and small sample sizes produce high variance results that appear decisive but aren't.

A/B Test Duration Requirements

RequirementMinimumWhy It Matters

Calendar duration2 full weeksCaptures full weekly traffic cycle; Mon-Sun patterns differ significantly
Conversions per variant100 conversions minimumBelow 100, statistical variance makes results unreliable
Statistical confidence95% (p-value 0.05)At 95%, only 5% chance results are random; industry standard
Sample size per variantDepends on current CVR and expected liftUse a sample size calculator before starting

Sample Size Calculator Guide
Before running a test, calculate how many visitors you need per variant:

Quick estimate formula:
If current CVR = 2%, expected improvement = 20% (new CVR = 2.4%), you need ~3,800 visitors per variant for 80% power at 95% confidence.
At 10,000 monthly sessions, 50/50 split = 5,000 per variant per month. This test would take about 2-3 weeks.
Use Optimizely's free sample size calculator (search "Optimizely sample size calculator") for exact numbers.

The Most Common Testing Mistakes

Stopping after 3 days: Day 3 results have no statistical validity regardless of how dramatic the difference looks
Peeking daily: Checking results daily and stopping when you see significance is called "the peeking problem" - it inflates false positive rates from 5% to 30%+
Running tests during atypical periods: A/B tests during major sales events produce results that don't apply to normal traffic
Declaring a loser too early: Many tests show initial improvement that fades as novelty effects wear off - or initial losses that recover

How long do I need to run an A/B test to get reliable results?

TL;DR - Quick Answer

Complete Expert Analysis

How Long Do I Need to Run an A/B Test to Get Reliable Results?

A/B Test Duration Requirements

Sample Size Calculator Guide

The Most Common Testing Mistakes

Turn This Knowledge Into Real Revenue Growth

+32% Conversion Rate

60-Second Setup

14-Day Free Trial

Muhammed Tüfekyapan

Continue Learning

How do I write a Mother's Day cart abandonment recovery email?

What is the best timing for a Mother's Day cart recovery email?

Should I offer an extra discount in my Mother's Day recovery email?

Use Cases

Requirement	Minimum	Why It Matters
Calendar duration	2 full weeks	Captures full weekly traffic cycle; Mon-Sun patterns differ significantly
Conversions per variant	100 conversions minimum	Below 100, statistical variance makes results unreliable
Statistical confidence	95% (p-value 0.05)	At 95%, only 5% chance results are random; industry standard
Sample size per variant	Depends on current CVR and expected lift	Use a sample size calculator before starting