Why A/B test instead of just changing the page
Every Shopify store gets advice: add trust badges, add a countdown, shorten the page. Some of it works on your store, some of it does nothing, and some of it costs you orders. On one store, a review block that “should” have helped cut orders by more than half in its first week. Without a test, nobody would have known.
An A/B test removes the guessing. Same ads, same prices, same week, two versions of the page. The one that makes more money stays.
Real Shopify A/B tests and their numbers
| Test | Result |
|---|---|
| One five-star review under Add to Cart | +8% revenue per visitor, +$67,365 |
| Video instead of a photo as the main product image | +16% revenue per visitor, +$87,665 |
| Price raised by 7.4% | Conversion slightly down, +$37,219 revenue |
| Honest low-stock counter across the journey | +5.84% cart-to-purchase, 98% confidence, ~$68,000 |
| Customer videos on the mobile homepage | +33% revenue per visitor, +$58,094 |
| A “Selling fast” tag based on real sales | +17.1% revenue per visitor, +$18,534 |
| Trust box under Add to Cart (high-ticket store) | +6%, two orders: not proven, still running |
The last line matters as much as the first. Most tests don’t win, and a programme that hides its losers isn’t testing honestly. (Six tests, winners and losers.)
What I test on Shopify stores
- Product pages (product page optimization): main image or video, reviews near the button, description placement, size guides, delivery promises, bundles. Usually the biggest wins. (Description above the button: ~$760K a year.)
- Collection pages: sort order, filters, product card information, badges.
- Homepage: hero message, social proof, brand and customer videos, navigation to bestsellers.
- Cart: free-shipping progress, upsells, payment logos, express checkout buttons.
- Checkout (Shopify Plus, checkout optimization): trust, delivery dates, upsells. A rebuilt checkout took one brand from $400K to $530K a month. (Checkout customization.)
- Offers and prices: price points, bundles, free-shipping thresholds, discount types. (The price test.)
How the programme works
- Research. AI reads your analytics, thousands of session recordings and every review and support message, and groups the reasons people almost didn’t buy.
- Plan by money and traffic. Each idea is ranked by the revenue it could move, and each test gets a sample size and a planned length before it starts.
- Build. AI builds the variant in your store’s style in hours; I review every one on mobile and desktop.
- Run 50/50. No peeking and no early calls. A test that is clearly losing is stopped early to protect your sales.
- Decide on revenue. Revenue per visitor on most stores; conversion rate on high-ticket stores, where one big order can distort revenue.
- Roll out and stack. Winners go live for everyone and the next test builds on them.
Shopify Rollouts vs a testing app vs done for you
| Shopify Rollouts (built in) | A/B testing app | Done for you (me) | |
|---|---|---|---|
| Cost | Included in some plans | From a small monthly fee, rising with traffic | From $4,000 a month (app cost separate) |
| What you can test | Theme versions | Pages, prices, offers, often checkout | Everything an app can test |
| Statistical confidence | Not shown at the time of writing | Usually shown | Planned before the test, reported after |
| Who builds and decides | You | You | AI builds, I decide and check |
| Best for | Simple theme tests on stores with a team | Stores with an in-house CRO person | Brands doing $100K+ a month without a CRO team |
How much traffic you need
Visitors needed per side, at 95% confidence and 80% power:
| Your conversion rate | To prove +20% | To prove +10% |
|---|---|---|
| 1% | ~43,000 | ~163,000 |
| 2% | ~21,000 | ~81,000 |
| 3% | ~14,000 | ~53,000 |
Less traffic? Test bigger changes on your busiest pages, run one test per page, and plan longer tests. More in how AI A/B testing works.
Who this is for
A good fit if you: make at least $100,000 a month, get 10,000+ visitors a month, and can commit to at least six months of testing.
Not a fit yet if you: have little traffic, want results in a week, or don’t want to change anything on your site. Start with the Shopify CRO audit checklist and the free calculators instead.
Done-for-you Shopify A/B testing starts at $4,000 a month: CRO pricing. How AI makes it faster: AI CRO agency.
