The test was clear. Variant B beat the original by 18%, the report said "winner", and you shipped it. A month later you look at revenue and it's... the same. Maybe a little lower.
This is one of the most common and least talked-about experiences in A/B testing. It makes people stop testing altogether, because if winners don't pay, what's the point?
The good news: there are only a handful of reasons it happens, and every one of them can be fixed before your next test starts.
Key takeaways
Stopping a test the first time it looks good turns a change that does nothing into a "winner" about 1 in 4 times.
Winners look bigger than they are. In our simulation, a real +10% improvement that came out as a winner showed a measured lift of about +31%. After launch, it shrinks back to +10%.
Conversion rate is not revenue. A page can win on sales and lose on money if it attracts cheaper buyers or fewer upsells.
Lost purchases skew results. If some sales can't be traced back to the version the buyer saw, the comparison is built on partial data.
Novelty, seasonality and audience mismatch can make a test result true for the test and false for everyone after.
The fix is a short checklist: revenue as the goal, a fixed length, two versions, full attribution, and a sanity check after launch.
1. It was never a winner
Every A/B test result includes some luck. The standard 95% confidence level means that if your new page makes no difference at all, there's still a 5% chance the test calls it a winner. That's the price of testing, and it's fine, as long as you play by the rules that keep it at 5%.
Most funnel owners don't, for a very human reason: they look at the results every day.
We simulated a test where both pages were identical, on a funnel with about 10,000 visitors a month, run for four weeks:
Checked once, at the end: a false winner appeared 5% of the time, exactly as promised.
Checked every day, stopped the first time it looked significant: a false winner appeared 24% of the time.

Early in a test the numbers swing wildly, and if you check daily, one of those swings will eventually cross the line. Stop there and you've crowned a page that does nothing.
Testing many versions at once has the same effect. With one challenger, a change that does nothing shows a significant win about 2.5% of the time. With four challengers, none of them better, at least one shows a significant win about 8% of the time. More lottery tickets, more fake winners.
Fix it: decide the test length before you start, in full weeks, and only make the call at the end. Test two versions, not five.
2. The win was smaller than it looked
Even when your winner really is better, the number in the report is almost always too high. Statisticians call this the winner's curse.
Here’s why. Results bounce around the true value. A test only declares a winner when the result bounces high enough to cross the significance line. So the tests that produce winners are disproportionately the ones that got lucky upward.
In our simulation, a page that was truly 10% better, tested on 10,000 visitors a month for four weeks, came out significant only about 12% of the time. And when it did, the median measured lift was +31%: three times the real effect. Ship that page and you'll get +10%, not +31%, which on a busy month can look like nothing at all.
Fix it: treat the measured lift as an upper bound, not a forecast. The smaller your traffic and the closer the call, the more it will shrink. Report the range, not just the headline number.
3. You optimized the wrong number
This is the one that costs funnel owners the most money, because the test isn’t wrong. It’s answering the wrong question.
Say you sell a $97 course with a $197 upsell, and you test a new sales page headline. It's more urgent and promises a faster result. Over 1,000 visitors each:
Control | Variant B | |
|---|---|---|
Conversion rate | 3.5% | 4.0% |
Buyers who take the $197 upsell | 30% | 15% |
Revenue per visitor | $5.46 | $5.06 |
Variant B wins on conversion rate by 14%. It also attracts buyers who were looking for a quick fix, half as many of them take the upsell, and the page earns 7% less per visitor. If the test was judged on conversion rate, you just shipped a revenue cut and called it a win.
The same thing happens with discounts that pull in bargain hunters, bigger promises that raise refund rates, and opt-in pages that fill your list with people who never buy.
Fix it: make revenue per visitor the primary metric for anything that sits on the path to a purchase, and count the whole funnel: order bumps, upsells and downsells included. Add refund rate as a guardrail so a winner that sells more but keeps less gets caught.
4. Some of the sales never reached the test
A funnel's sales often happen somewhere other than the page you tested. The buyer lands on your sales page, clicks through to a checkout on another domain, maybe switches from the Facebook app's built-in browser to Safari, maybe comes back two days later on their laptop.
Every one of those hops is a chance for the purchase to lose track of which version the buyer saw. When that happens, the sale still lands in your bank account, but not in your test.
That would be harmless if it hit both versions equally. It often doesn't. If Variant B sends more people through a mobile checkout, or through an in-app browser, it can lose more of its sales to tracking gaps than the control does. The test then compares a complete count against a partial one.
Fix it: before you trust a revenue result, check how much of your revenue was actually matched to a variation. A test where 95% of purchases are matched is trustworthy. One where 60% are matched is a guess. Use tracking that follows the buyer through the checkout with the checkout platform’s own purchase data, such as a webhook from your funnel builder, rather than relying on a pixel on the thank-you page. Our ClickFunnels 2.0 guide shows what that looks like. In Splitlab, every conversion is marked Attributed, Partial or Unattributed, so you can see the gap before you act on a result.
5. The test ran at the wrong time, or to the wrong people
Some winners are real, but only for the people and the week they were tested on.
Novelty effect. Returning visitors notice anything new and click on it out of curiosity. The lift fades once the change isn't new anymore.
Seasonality. A test run during a launch, a holiday sale or a big ad push measures how people behave that week. Normal weeks behave differently.
Audience mismatch. A test that only ran on mobile, or only on paid Facebook traffic, tells you about that audience. Ship it to everyone and the effect gets diluted, or reverses.
Fix it: run tests for at least one or two full weeks of normal traffic. Avoid starting or ending a test during a promotion. Ship the winner to the same audience the test ran on, or retest before widening it.
6. It won on a step that doesn't move the money
Not every step in a funnel is worth the same. A 20% lift on a page that sits far from the purchase, or that very few buyers pass through, can be completely real and still be invisible in your revenue.
A typical example: a new thank-you page that gets 20% more people to join your Facebook group. Real result, worth doing, but it won't show up in this month's revenue.
Fix it: before you test, estimate what the step is worth: how many buyers pass through it and how much each one spends. Put your testing time on the steps closest to the money: the sales page, the order form, the bump and the upsell.
A checklist for winners that pay
Before the test:
Primary metric is revenue per visitor for any page on the path to a purchase.
Refund rate or another guardrail is set, so a win that sells more but keeps less is caught.
Two versions, with a difference big enough to matter.
The length is fixed in advance, in full weeks, away from promotions.
Purchases are tracked through checkout by your funnel builder's purchase data, not just a pixel.
At the end:
Most revenue is matched to a variation. If a big share isn't, fix tracking before trusting the result.
The lift is read as a range, and the bottom of the range still justifies the change.
The winner ships to the same audience that was tested.
After launch:
Compare revenue per visitor for the four weeks before and after, same days of the week, outside promotions. If the gain has vanished, one of the six reasons above is usually why.
FAQ
Why did my A/B test winner not increase revenue?
Usually one of six reasons: the winner was a false positive from stopping early or testing many versions; the measured lift was inflated by luck; the test measured conversion rate instead of revenue; some purchases weren't tracked back to the test; the test ran at an unusual time or on a narrower audience; or the winning step sits too far from the purchase to move revenue.
What is peeking in A/B testing?
Peeking is checking a test's results repeatedly and stopping as soon as it looks significant. It feels harmless, but it multiplies the chance of a false winner. In our simulation, checking daily and stopping at the first significant result produced a false winner 24% of the time instead of 5%.
What is the winner's curse in A/B testing?
It's the tendency of winning tests to overstate the size of the win. Tests only declare a winner when the result lands high, so the winners you see are biased upward. Expect the real lift after launch to be smaller than the number in the report, especially on low traffic.
Should I use conversion rate or revenue to pick a winner?
For any page on the path to a purchase, use revenue per visitor. Conversion rate ignores order value, upsells and refunds, so a version can win on conversions and still earn less.
How do I check whether a test winner really worked?
Compare revenue per visitor for the four weeks before and after you shipped it, using the same days of the week and avoiding promotions. If you have enough traffic, keep a small share of visitors on the old version for a few weeks as a holdout.
How long should an A/B test run?
Long enough to reach the sample size you planned for, and at least one or two full weeks so every day of the week is included. Decide the length before you start and make the call only at the end.
My final thoughts
Every reason on this list comes back to one thing: a test is only as honest as the number it’s judged on and the time it’s given.
If you fix just two habits, make them these. Decide the sample size before you launch and don’t stop early. And judge the winner on revenue per visitor, with every sale credited to the version the buyer saw, not on clicks or opt-ins.
Do that and most “winners that didn’t pay” never get shipped in the first place. Book a demo if you want to see how Splitlab tracks revenue to every variant.
A real winner shows up in your bank account, not just in your test report.
Built for revenue
Test for revenue, not just clicks.
Run A/B, multivariate, and multi-armed bandit experiments — with real revenue tied to every variation.
We’ll increase your conversion rate by 30% or you don’t pay. Terms apply.
✓
Revenue per variation
✓
A/B & MVT
✓
Multi-armed bandit
Test for revenue, not just clicks.
A/B, multivariate and multi-armed bandit experiments, with real revenue tied to every variation.


