What sample size calculator should I use?

Question

As an e-commerce manager, I'm struggling to determine the right sample size for my A/B tests and conversion rate optimization experiments. I need a reliable method to calculate statistical significance that doesn't require advanced statistical knowledge. My goal is to understand how many visitors or customers I need to test a new design, discount strategy, or marketing campaign with confidence. What are the best sample size calculators and methods for ensuring my test results are meaningful and actionable?

Muhammed Tüfekyapan · Accepted Answer

What Sample Size Calculator Should I Use?
Running an A/B test without calculating sample size first is the single most common testing mistake in e-commerce. Most store owners stop tests when they see a promising result, not when they've reached statistical validity. This "peeking" produces false positives that lead to wrong decisions far more often than most realize.

Recommended Sample Size Calculators

ToolBest ForKey Feature
  
    Evan Miller (evanmiller.org)Standard frequentist A/B testsSimple, accurate, widely used in industry
    AB Testguide.comE-commerce conversion testsIncludes duration estimates and revenue impact
    VWO Sample Size CalculatorVWO usersIntegrated with VWO platform settings
    Optimizely Sample SizeEnterprise testingBayesian and frequentist options
    CXL Sample Size ToolLearning the conceptsGood explanations alongside calculations

How to Use a Sample Size Calculator

Input 1: Baseline Conversion Rate
    Your current conversion rate for the page/element you're testing. Check Shopify Analytics or Google Analytics. Example: product page add-to-cart rate = 4.2%.

Input 2: Minimum Detectable Effect (MDE)
    The smallest lift you care about detecting. For most tests, 10-20% relative improvement is meaningful. If your baseline is 4%, you want to detect lifts to 4.4% (10% relative) or higher. Smaller MDEs require much larger samples.

Input 3: Statistical Power
    Default 80% power is standard. This means if there's a real effect, you'll detect it 80% of the time. 90% power requires ~50% more traffic but reduces missed winners.

Input 4: Significance Level
    Standard is 95% (p=0.05). This means 5% chance of a false positive. For high-stakes decisions (major site changes), use 99% (p=0.01).

Sample Size Reference Table

Baseline CVRDetecting 10% LiftDetecting 20% LiftApprox. Duration (1K daily visitors)
  
    1%~40,000/variant~12,000/variant80 days / 24 days
    3%~14,000/variant~4,000/variant28 days / 8 days
    5%~8,500/variant~2,500/variant17 days / 5 days
    10%~4,000/variant~1,200/variant8 days / 2-3 days

Built-In Statistical Testing
  Growth Suite's A/B Testing Module handles sample size and statistical significance automatically within Trigger Campaigns - you set the test variants, and the system calculates when sufficient data exists to declare a winner. This eliminates the "peeking" problem and ensures you're making decisions based on valid data, not premature patterns. The system also accounts for novelty effects that can inflate early test results.

What sample size calculator should I use?

TL;DR - Quick Answer

Complete Expert Analysis