What Is A/B Testing? How Split Tests Work

How a Split Test Works

Start by picking one thing to change. It might be a product photo, the button text or the size of a discount. The current version is called the control (A). The new one is the variant (B).

Your testing tool sends each visitor to A or B at random. A good tool also keeps people on the same version when they come back. That random split is what makes the test fair. Both groups get roughly the same mix of shoppers: new and returning, on phones and on laptops, from ads and from emails. So if B does better, the change is probably why, not the crowd.

Testing three or more versions at once is called A/B/n testing. Same idea, more groups. Each group needs its own share of traffic, though.

Why Comparing Stores or Months Isn't a Test

Without a random split, a cause and a coincidence look the same. In our data from 250 Shopify stores, offers in the cart drawer were accepted 12% of the time, versus 7% on product pages. That wasn't a controlled test. Different stores chose different spots, so the gap may come from the stores, not the spot.

Before-and-after checks on your own store fall into the same trap. Sales can shift from March to April because of ads, a holiday or a competitor's sale. An A/B test gets around that by running both versions at the same time, to the same kind of visitors.

A Discount Test, Worked Out

Say you test a time-limited offer. Version A gives 10% off. Version B gives 20% off. Each goes to 5,000 visitors. Your average order is $60 before the discount, and the products in it cost you $24.

	A: 10% off	B: 20% off
Visitors	5,000	5,000
Orders	150	175
Conversion rate	3.0%	3.5%
Revenue per order	$54	$48
Revenue	$8,100	$8,400
Profit after product cost	$4,500	$4,200

B comes out ahead on conversion rate and on revenue. But the bigger discount gives away $6 more on every order. That's why B ends up with $300 less profit. Which version "won" depends on the goal you set before the test started.

The gap is also small. If both offers truly performed the same, a gap this size would still show up about 1 time in 6 by pure chance. So don't crown B yet. Let the test run to the sample size you set at the start, then check whether the result reached statistical significance, meaning chance alone is an unlikely explanation.

To decide which discount levels are worth testing in the first place, see how to choose the right discount percentage.

When a Test Is Worth Running

There's enough traffic. A store with a few orders a week may wait many weeks for a clear answer.
Only one thing changes. Swap the photo, price and headline at once, and you won't know which one mattered.
You've picked the goal first. Conversion rate, AOV and revenue can point to different winners, as the example shows.
It's a normal stretch of time. A test run during Black Friday tells you about Black Friday shoppers, and they may not act like your usual visitors.

For the setup steps, how long to run a test and the mistakes to avoid, see our guide to A/B testing on Shopify.

How Growth Suite Handles Offer Tests

Growth Suite, our app, has an A/B testing module built in. You pick the goal: conversion rate, average order value or total revenue. Then you set the discount depth, the urgency and the share of traffic each version gets. As visitors come in, the results update in real time.

If you're weighing other tools too, see our picks for the best Shopify conversion rate apps.

Myth: If B Is Ahead, B Has Won

It's tempting to check a test every day and stop it the first time B pulls ahead. Early numbers swing a lot, though. In a small sample, a few lucky orders can move things. Each time you peek and stop on a good day, you raise the odds of picking a "winner" that was only luck. Set your sample size or run time before you start. Then let the test finish.