Search for articles…
How the multi-armed bandit moves traffic
With a multi-armed bandit, Splitlab moves traffic for you while the test runs, sending more visitors to the variations that are doing better. This article explains when it moves traffic, how much, and the trade-offs.
1. Turn the bandit on
When you set up a campaign, go to the variations step and find Traffic Split. Choose Multi Arm Bandit instead of Equal Distribution or Custom Distribution. You can then pick an Algorithm; new campaigns start on Splitlab Adaptive, which is the recommended choice if you’re unsure.
2. When the split changes
Splitlab checks bandit campaigns every hour. A campaign’s split is recalculated once at least 20 new conversions have arrived since its last update, and the first update runs within the hour of the campaign starting. Busy funnels can move within the hour; quieter ones wait until there is enough new data to say something new.
Each update looks at all of the campaign’s data so far, not just the latest conversions. The bandit settings show when the split was last updated.
3. How the split is decided
Every algorithm works out each variation’s chance of being the best on your primary metric, then turns that into a share of traffic. They differ in how boldly they act on it:
Splitlab Adaptive keeps traffic close to even until the results are convincing, then shifts it gradually, by at most 15 points per update.
Thompson Sampling gives each variation traffic in proportion to its chance of being the best. It earns the most conversions during the test but commits quickly on low traffic.
Top-Two Thompson splits traffic between the leader and its closest challenger, to settle the winner with fewer visitors.
Epsilon-Greedy sends 90% of traffic to the current leader and spreads 10% evenly across the rest.
Recency-weighted Thompson counts recent days more: a day’s results weigh half as much every 7 days, so the split follows seasonality.
Apart from Splitlab Adaptive, the algorithms keep the split even while they collect data, until each variation has enough visitors.
4. The safety rails
The control always keeps at least 10% of traffic, so there is always a fair baseline to compare against.
A variation that is clearly doing far worse than the control is taken out of the running, so it stops receiving bandit traffic.
Only enabled variations take part. If only one is enabled, it gets all the traffic.
5. Returning visitors keep their variant
A new split decides where new visitors go. Anyone already assigned keeps seeing the same variant, so the overall mix shifts more gradually than the percentages alone suggest. See Do visitors always see the same variant?.
6. Trade-offs to know about
A bandit aims to earn more while the test runs, and that weakens the statistical confidence of the head-to-head comparison in your report. If what you need is the cleanest possible answer, use an even split instead.
If you set percentages by hand on a live bandit campaign, the split is pinned and the bandit stops reallocating traffic. The dashboard asks you to confirm before it does this.
For more background, read How a multi-armed bandit works.
7. Still stuck?
Contact us or email team@splitlab.io with the campaign name and what you expected the split to do.
Related articles
Still stuck?
Email team@splitlab.io or send us a message.
Contact us