AI A/B Testing: How It Really Works, What It Gets Wrong, and 5 Real Tests
AI A/B testing explained by someone who runs it: what AI builds, when bandits beat 50/50 splits, how many visitors you need, the mistakes, and 5 real tests.
“AI A/B testing” gets used for three very different things, and most articles mix them up. Some mean AI that writes and designs the test. Some mean AI that moves traffic to the winning version while the test runs. Some mean AI that explains the result. Only one of them changes how much money a store makes from testing.
I run AI-built A/B tests on e-commerce stores every week. For one brand that’s been 60+ tests in 12 months, a 91% higher conversion rate and $7M in extra revenue. Below: how AI A/B testing really works, step by step, what it can’t do, how many visitors you need, and five real tests with their numbers.
The three kinds of AI in A/B testing
| Kind | What AI does | How much it matters |
|---|---|---|
| AI that builds the test | Reads your data and reviews, then designs, writes and codes the variant | The biggest change. Building used to take 2–4 weeks per test; now it takes hours |
| AI that allocates traffic | Multi-armed bandits send more visitors to the version that’s ahead | Useful for short promotions; risky for long-term decisions |
| AI that analyses results | Summarises the result, splits it by device, source, new vs returning | Saves time; still needs a person to decide what it means |
The reason the first one matters most: in classic CRO, the bottleneck was never ideas or traffic. It was production. A strategist, a designer, a developer and a copywriter each had to touch every test, so most agencies shipped one or two tests a month. When AI builds the test, the limit becomes your traffic, which is where it should be.
How AI A/B testing works, step by step
This is the process I use on every store.
1. AI reads the evidence
AI goes through your analytics and thousands of session recordings in hours: which pages get visitors but few buyers, where people stop scrolling, which step of the funnel leaks. (What to look for at each step.)
2. AI groups what customers are telling you
Reviews, surveys and support messages are full of reasons people almost didn’t buy: will it fit, can I trust this store, when will it arrive, is it worth the price. AI sorts thousands of them by doubt, so you can see which doubt is biggest.
3. A person picks the test
This is where AI stops. A strategist chooses which doubt to answer first, ranked by how much revenue it could move, and checks that the page has enough traffic for a clear result.
4. AI builds the variant
AI writes the copy from customers’ own words, designs the section in the store’s style and writes the code. What took a team weeks takes hours.
5. A person checks it
Every variant is checked on mobile and desktop, for brand, for accuracy (no invented claims, no fake numbers) and for speed.
6. The test runs 50/50
Half the visitors see the original, half see the new version. Same ads, same prices, same traffic. The test has a planned length and a minimum sample before anyone looks at the winner.
7. AI pulls the numbers, a person decides
Revenue per visitor, conversion rate, order value, by device and by new vs returning visitors. Then the decision: keep it, drop it, or test a stronger version.
Bandits vs classic A/B tests
A multi-armed bandit is the “AI” in many AI testing tools. Instead of keeping a 50/50 split, it sends more visitors to whichever version is ahead.
| Classic 50/50 A/B test | Bandit (AI traffic allocation) | |
|---|---|---|
| Goal | Learn what works | Earn the most during the test |
| Answer at the end | Clear, with a confidence level | Blurry: the losing side got little traffic |
| Best for | Product pages, checkout, offers, anything you keep for months | Short promotions, holiday banners, headlines |
| Risk | You show a weaker version to half your visitors for a while | Early luck can crown the wrong winner |
My rule: if you’ll live with the change for months, use a 50/50 test. Bandits are fine for a Black Friday banner that disappears in a week.
How many visitors does an AI A/B test need?
AI makes tests faster to build. It doesn’t make them faster to prove. The math is the same as always. Visitors needed per side, at 95% confidence and 80% power:
| Your conversion rate | To prove +20% | To prove +10% | To prove +5% |
|---|---|---|---|
| 0.5% | ~86,000 | ~328,000 | ~1,281,000 |
| 1% | ~43,000 | ~163,000 | ~637,000 |
| 2% | ~21,000 | ~81,000 | ~315,000 |
| 3% | ~14,000 | ~53,000 | ~208,000 |
Two things follow from this table:
- With less traffic, test bigger changes. A whole product page rebuilt around customers’ doubts can move 10–20%. A button colour won’t move anything you can measure.
- Run fewer tests at once on the same page. Two tests on one page mix their effects, and you can’t tell which one did the work.
On a high-ticket store with a 0.2% conversion rate, proving a 20% lift takes about 200,000 visitors per side. (How I test on a store like that.)
What AI gets wrong in A/B testing
- Generic copy. AI’s first draft sounds like every other store. Copy built from your customers’ own review language wins more often.
- Invented claims. AI will happily write “loved by 50,000 customers” or “ships in 24 hours”. Every claim has to be checked against reality.
- Fake urgency. Countdown timers that reset and “only 1 left” forever can lift a test and then raise refunds. An honest low-stock counter did win: +5.84% cart-to-purchase at 98% confidence. (That test.)
- Calling tests too early. AI summaries can declare a winner after a few days. Small samples swing wildly; plan the length first and stick to it.
- Judging on the wrong number. Clicks and add-to-carts are easy to lift. On one test, fewer people added to cart and revenue per visitor still rose 33%. Judge on revenue per visitor, or conversion rate on high-ticket stores.
- Ignoring losers. Most tests don’t win. A review block that “should” have helped cut orders by more than half on one store, and I stopped it on day seven. Losing tests teach you what your customers don’t care about.
Five real AI-built A/B tests
| Test | Result |
|---|---|
| One five-star review under Add to Cart | +8% revenue per visitor, +$67,365 |
| Customer videos and a brand story on the mobile homepage | +12% conversion, +33% revenue per visitor, +$58,094 |
| Video instead of a photo as the main product image | +9% conversion, +11% revenue per visitor, +$57,644 |
| Product page rebuilt around customers’ doubts | +13% conversion rate |
| Honest low-stock counter across the journey | +5.84% cart-to-purchase, 98% confidence, ~$68,000 |
Full write-ups: the review test, the homepage test, the stock counter.
How to start AI A/B testing on your store
- Check your traffic against the table above. Under ~10,000 visitors a month, fix the obvious leaks first and save testing for later.
- Pick one number to judge on: revenue per visitor, or conversion rate if your orders are high-ticket.
- Collect the evidence: your last 500–1,000 reviews, support messages, and 30–90 days of analytics.
- Let AI group the doubts, then choose the biggest one yourself.
- Build one strong variant that answers that doubt, check it, and run it 50/50 for a planned length.
- Write down every result, win or lose, and let the next test build on it.
If you’d rather have it done for you, that’s what my Shopify A/B testing and AI CRO agency services do, from $4,000 a month. For the tools by category, see AI CRO tools.
Frequently asked questions
What is AI A/B testing?
A/B testing where AI does part of the work: building the variants (design, copy, code), allocating traffic between them, or analysing the results. The test itself is the same: two versions, real visitors, and the numbers decide.
Can AI run A/B tests on its own?
It can generate variants and shift traffic automatically, but it can't tell which idea is worth testing for your brand, whether a claim on the page is true, or why a test really won. Fully automatic programmes tend to produce many small, noisy tests and few lasting wins.
Is a multi-armed bandit better than an A/B test?
Not for learning. A bandit sends more traffic to the version that's ahead, which earns more during the test but makes the final answer less clear. Use bandits for short promotions and headlines; use 50/50 A/B tests for changes you'll keep for months.
How many visitors do I need for AI A/B testing?
The same as for any A/B test. At a 2% conversion rate, proving a 20% lift takes about 21,000 visitors per side; a 10% lift takes about 81,000. AI makes tests faster to build, not faster to prove.
What are the best AI A/B testing tools?
It depends on what you need AI to do: build variants, analyse sessions and reviews, or allocate traffic. Most stores need a solid testing app plus AI for research and building. See the AI CRO tools guide for each category.
Alexander Skibinskiy
17 years in digital marketing and e-commerce. Former head of growth at Karma and Luck, a nine-figure brand, where conversion rate grew 50% in one year of A/B testing. Founder of a seven-figure e-commerce brand; 100+ brands helped since 2015.
Watch the free training →