Search for articles…
How is a winner decided?
Splitlab uses Bayesian statistics, in the style of VWO’s SmartStats, to decide when a variation has genuinely beaten your control. Rather than stopping after one good day, it waits until the evidence clears a set of thresholds.
1. What is measured
The decision is made on the campaign’s primary metric. For each variation, Splitlab simulates thousands of plausible outcomes from the data so far and works out its chance to be best and its potential loss — how much you’d stand to lose, in percentage points, if you picked it and it turned out not to be better.
Pick a primary metric that matches what you care about. If revenue matters most, the reports also show Revenue per Visitor (RPV), which tells you which variation actually earns more, not just which one converts more. See Revenue per visitor vs conversion rate.
2. The thresholds a winner must clear
A variation is declared the winner only when all of these are true at once:
it has at least a 95% chance to be best
its potential loss is below the smallest improvement worth acting on (worked out from your control’s conversion rate)
it has at least 25 conversions of its own
the test has at least 1,500 visitors in total
the test has run for at least 7 days
The visitor, conversion and duration thresholds are there to stop an early lucky streak being called a winner. Only variations can win; if none beats the control, no winner is declared.
3. When the check runs
Running campaigns are checked once a night. Until the first check, the report says the campaign hasn’t been judged yet. While there’s no winner, the report lists what’s still missing — for example “The test needs more visitors overall” or “The test has not yet run a full week”.
When a winner is declared, the report says You have a winner and the campaign owner gets an email.
4. A winner can be withdrawn
The test keeps running after a winner is declared, and the check keeps looking. If later traffic pulls the variation back below the bar, the call is withdrawn and the report says so. It can win again once the evidence recovers.
5. What to do with a winner
In the campaign report you choose what happens next:
Roll out winner creates a draft rollout campaign for a share of visitors you choose. Review it, then start it when you’re ready.
Serve to everyone sends all traffic to the winning variation. This ends the comparison: the control stops collecting data and the decision can no longer be withdrawn.
Under When a winner is declared you can choose Tell me, and I will decide (the default) or Serve it to everyone automatically. The automatic option ends the comparison without asking, so use it with care.
6. Good habits
Decide your primary metric before you start, and avoid changing it mid-test.
Let the test run at least a full week so weekday and weekend behaviour are both included.
Avoid changing the traffic split while the test is live.
7. Still stuck?
Contact us or email team@splitlab.io with the campaign name if a result doesn’t look right.
Related articles
Still stuck?
Email team@splitlab.io or send us a message.
Contact us