Splitlab home

Book a Demo

Splitlab home

Book a Demo

Splitlab home
A fixed A/B test keeps a 50/50 split for the whole test, while a multi-armed bandit shifts traffic to the variant earning more, adding $1,680 from the same 10,000 visitors

November 6, 2026

By

Bishal Katuwal

·

12 min

read

Multi-Armed Bandit vs A/B Testing: Which One Should a Funnel Business Use?

November 6, 2026

By

Bishal Katuwal

·

12 min

read

Multi-Armed Bandit vs A/B Testing: Which One Should a Funnel Business Use?

In the ever-evolving business environment, achieving a competitive edge demands innovative solutions beyond conventional methods. Organizations require smart systems that can optimize workflows, enhance customer interactions, and reveal new avenues for growth. Enter Biotix, the cutting-edge AI-powered CRM management tool.

Every visitor you send to the losing variant is money spent on a lesson you've already learned. Bandits fix that, at a cost. Here's when the trade is worth it, and which bandit to pick.

Bishal Katuwal

Bishal Katuwal

Splitlab Founder

A fixed A/B test keeps a 50/50 split for the whole test, while a multi-armed bandit shifts traffic to the variant earning more, adding $1,680 from the same 10,000 visitors
A fixed A/B test keeps a 50/50 split for the whole test, while a multi-armed bandit shifts traffic to the variant earning more, adding $1,680 from the same 10,000 visitors

A classic A/B test sends half your traffic to a page you suspect is worse, for weeks, on purpose. That's the price of a clean answer. For a research team, fair enough. For a business paying for every click, it's a cost that adds up quickly and quietly.

Multi-armed bandits are the alternative. They start even, watch the results, and move traffic toward the variant that’s earning more while the test is still running. They are not “better” than A/B tests. They answer a different question, and knowing which question you're asking is the entire decision. And once you’ve picked a bandit, there’s a second decision most guides skip: which bandit.

Key takeaways

  • A/B testing asks "which is better, and by how much?". A bandit asks "how do I earn the most from this traffic while I'm still learning?".

  • Bandits earn more during the test. Fixed A/B tests give a more precise estimate of the difference, because they keep feeding the loser.

  • Not all bandits behave the same. Thompson sampling commits fast, top-two Thompson finds the winner sooner, epsilon-greedy is simple, recency-weighted follows change, and Splitlab Adaptive moves cautiously enough for any traffic level.

  • Point the bandit at revenue per visitor, not conversion rate, or it will happily chase the discount variant.

What is a multi-armed bandit?

A multi-armed bandit is a traffic allocation method that continuously shifts visitors toward the best-performing variant based on the results so far, instead of holding a fixed split until the test ends. Each variant is treated like a slot machine arm: pull each one a little, notice which pays out more, pull that one more often, but keep trying the others occasionally in case you were wrong.

The name comes from a gambler facing a row of slot machines (“one-armed bandits”) and trying to work out which one to play without blowing the whole bankroll finding out. Statisticians call it the explore/exploit trade-off: every visitor spent learning is a visitor not spent earning.

The two questions

A/B testing asks: which variant is better, and by how much? Traffic is split evenly and held there. You wait until you have enough data, then decide. The even split is what gives you a clean comparison, and the price of that cleanliness is the traffic you sent to the loser.

A bandit asks: how do I make the most money from the traffic I have, while I'm still learning? Over the life of the test, most visitors end up on the best variant. You get less certainty about exactly how bad the losers were, because you stopped sending them people.

What a bandit is worth: a worked example

Say a five-day promotion sends 10,000 visitors to a sales page with two variants. The control earns $2.00 revenue per visitor. Variant B earns $2.60.

Fixed 50/50 test. 5,000 visitors each: 5,000 × $2.00 + 5,000 × $2.60 = $23,000.

Bandit. Even split for the first 2,000 visitors while it learns ($4,600), then roughly 85/15 toward Variant B for the remaining 8,000: 6,800 × $2.60 + 1,200 × $2.00 = $20,080. Total $24,680.

That's $1,680 more, about 7%, from the same traffic and the same two pages. The gap grows with more variants, a bigger difference between them, or a longer test. It shrinks when the variants are close, because there’s less to gain from picking a side early. (The numbers are illustrative, not a customer result.)

The fixed test still has something the bandit doesn't: a tight estimate of how much better B is. If you'll reuse that number to plan ten other funnels, the $1,680 might be worth paying.

The trade-off, honestly

Fixed A/B test

Multi-armed bandit

Goal

Learn which is better

Earn the most during the test

Traffic to the loser

Half, for the whole test

Shrinks as evidence grows

Time to a confident answer

Shorter for the same traffic

Longer, because the loser gets less data

Precision of the lift estimate

Good

Weaker for losing variants

Many variants

Slow

Drops losers fast

Changing conditions

Averaged away

Tracked continuously

Clean before/after story

Yes

Harder to tell

Risk from a lagging metric

Low

Medium, it may chase early noise

The line that matters most is the third one. A bandit is slower at proving the loser lost, because it stops sending it traffic. If you need a precise estimate of how much worse variant C was, run an A/B test. If you just need to stop paying for variant C, run a bandit.

When a bandit is the right call

Promotions and time-limited offers. A Black Friday page runs for five days. There's no "after the test" to apply the learning to. Every visitor sent to the weaker offer is gone.

Paid traffic on evergreen pages. You pay per visitor and the page will run for months. Continuous optimization is worth more than a tidy lift number.

Many variants. Five headlines, four price presentations, three guarantees. A fixed test with twelve arms takes forever. A bandit finds the obvious losers in days and spends the rest of the traffic on the contenders.

Always-on optimization. Some pages are never "done". Leave a bandit running, add new variants when you have ideas, and let it find its own balance.

Seasonality inside the test window. If weekdays behave differently from weekends, a bandit that learns from recent days follows the shift. A fixed test averages it into mush.

When a fixed A/B test is the right call

Big redesigns you need to learn from. If the point is to understand whether the new sales page structure works so you can roll it out to ten other funnels, you want a precise, unbiased estimate.

Lagging metrics. If the revenue event lands weeks after the visit, like a won deal in a CRM, a bandit is optimizing on stale information and can chase early noise. Run a fixed test, or point the bandit at a faster signal such as booked appointments and read the final result on revenue.

When you have to present the result. "Variant B lifted revenue per visitor 11%, range 4% to 18%" is a sentence stakeholders trust. Bandit results are harder to explain because the traffic shares moved.

Very low traffic, with a classic bandit. With a few hundred visitors a week, an aggressive bandit moves traffic on noise. Either hold an even split, or use an algorithm built to stay cautious until the data is convincing (more on that below).

Splitlab's five bandit algorithms

Most testing tools give you one bandit and no say in how it behaves. Splitlab lets you choose per test, because the right allocator depends on your traffic, how long the test runs, and whether you care more about earning during the test or finding the winner fast.

Splitlab algorithm picker, enlarged: Splitlab Adaptive (recommended, safe at any traffic level), Thompson Sampling (earns the most conversions during the test), Top-Two Thompson (finds the winner with fewer visitors), Epsilon-Greedy (simple and predictable, 90% to the leader), Recency-weighted Thompson (follows recent performance, for long tests)

Splitlab Adaptive (the default)

Our own algorithm. It keeps traffic close to even until the results are convincing, then shifts it gradually, by at most 15 percentage points per update. Low-data variations are pulled toward the test average, so one lucky early conversion can’t hijack the split. It works from the first visitor to millions.

Choose it when: you're not sure, your traffic is modest, or you want a bandit you can leave alone.

Watch out for: on low traffic the split stays near even for a while. That’s intended: it won’t bet on a winner the data can’t back up yet.

Thompson Sampling

The classic. At every update, each variation gets traffic in proportion to its chance of being the best.

Choose it when: you have healthy traffic and want to earn the most conversions or revenue during the test.

Watch out for: it commits quickly. Under about 1,000 visitors per variation, early luck can push traffic to the wrong variation, and the changing split makes results harder to read as a classic A/B test.

Top-Two Thompson

Splits traffic between the current leader and its closest challenger, so the comparison that decides the winner gets settled with the fewest visitors.

Choose it when: the goal is to identify the winner quickly and confidently, not to squeeze the most out of the test period.

Watch out for: it earns less during the test than Thompson sampling, because about half the traffic keeps testing the runner-up.

Epsilon-Greedy

Sends 90% of traffic to the variation currently most likely to be the best, and spreads the other 10% evenly so every variation keeps being tested.

Choose it when: you want behaviour that's simple to explain to a client or a boss.

Watch out for: the leader can change from one update to the next, so on close results the split may swing between variations. It doesn't get more cautious on low traffic.

Recency-weighted Thompson

Thompson sampling where recent days count more: a day's results weigh half as much every 7 days. The split follows seasonality and variations that improve or fade over time.

Choose it when: the test runs for weeks or months, traffic is steady, and performance shifts with the season, the offer or the ad mix.

Watch out for: it learns from less effective data than plain Thompson, so it needs steady traffic and reacts more to a few unusual days.

Here is how differently they behave on the same data. After a week, Variant B probably leads and Variant C is the runner-up. One update later, each algorithm lands somewhere else:

Illustrative split after one update on the same data, starting from 34% · 33% · 33%, shown as Control / Variant B / Variant C: Splitlab Adaptive (default, moves at most 15 points per update) 22/48/30, Thompson Sampling 10/66/24, Top-Two Thompson 10/52/38, Epsilon-Greedy 10/85/5, Recency-weighted Thompson 10/36/54 if C improved over the last few days; every algorithm keeps at least 10% on the control

Which bandit algorithm should you pick?

Your situation

Pick

Not sure, or under ~1,000 visitors per variation

Splitlab Adaptive

High traffic, want the most revenue during the test

Thompson Sampling

Need to name a winner quickly and confidently

Top-Two Thompson

Need behaviour you can explain in one sentence

Epsilon-Greedy

Long-running page, seasonal or shifting performance

Recency-weighted Thompson

The settings that control a Splitlab bandit

Choose Multi Arm Bandit as the traffic split on a test, next to Equal and Custom Distribution, and pick an algorithm. That’s the only setting.

Splitlab app Variations step, enlarged: Traffic Split set to Multi Arm Bandit and the Algorithm set to Splitlab Adaptive, the default

When the split updates is decided for you. Splitlab checks every bandit test each hour and recomputes the split once 20 new conversions have arrived since the last update. Busy funnels move within the hour; quiet ones wait until there’s enough new data to say something new, so the split never moves on noise.

What it learns from is all of the test’s data. Recency-weighted Thompson is the exception: it counts a day’s results half as much for every 7 days of age.

The first update runs within the hour of switching the bandit on. After that, the panel shows when the split was last updated.

Whatever algorithm you choose, the bandit optimizes the test’s primary metric. Set that to revenue per visitor and it sends traffic to the variation that makes more money, upsells and order bumps included, not just the one that converts more often.

The safety rails that come with every algorithm

A bandit that's free to move traffic can also move it to the wrong place. These limits apply whichever algorithm you pick.

Six safety rails on every Splitlab bandit: a 10% minimum control floor, a 5% exploration share, a −40% stop-loss (after 100 visitors, 95% sure), a 50-visitor warm-up per variation, Splitlab Adaptive moving at most 15 points per update, and updates only after 20 new conversions, checked hourly
  • Control floor. The control always keeps at least 10% of traffic, so you can still say how much the winner beat it.

  • Exploration share. 5% of traffic is spread across every variation, so none is starved of new data.

  • Stop-loss. Once a variation has 100 visitors, if it's doing 40% worse than the control and Splitlab is 95% sure of it, the variation loses its traffic immediately instead of being phased out.

  • Warm-up. The classic algorithms hold an even split until every variation has 50 visitors. Splitlab Adaptive doesn't need one, because it moves cautiously from the start.

  • Gentle steps. Splitlab Adaptive never moves any variation more than 15 points in one update, so one noisy day can't swing the split.

The report still shows probability to be best, expected loss and lift for every variation, so you can end the test and lock in a winner, or leave it running as an always-on optimizer.

The hybrid most teams settle on

  1. Start with Splitlab Adaptive. It behaves like an even split while the data is thin and shifts only as evidence builds, so you get the warm-up for free.

  2. Switch algorithms if the situation changes. Moving to Thompson sampling once traffic is healthy, or to recency-weighted for a long-running page, keeps your other settings.

  3. Read the final report on probability to be best and expected loss. They still work with unequal sample sizes.

  4. Prune and add. Drop clear losers, add new ideas, and let it keep running.

Pitfalls specific to bandits

Novelty effects. A new variant often gets a short bump from curiosity. A bandit will chase it, then correct. Give it time before you read anything into early traffic shares.

A metric that can be gamed. A conversion-rate bandit on a page with a discount variant will send everyone to the discount. Optimize on revenue.

Expecting fast moves on thin data. A page with a few sales a day gives the bandit little to learn from, so the split moves slowly. That’s deliberate: moving on a handful of sales would be moving on noise.

Two allocators on one page. Running a platform's native split test and a bandit on the same step means two systems fighting over the same visitors. Pick one.

Reading traffic share as the result. "Variant B is getting 80% of traffic" is not a finding. Read the probability of being best and the expected loss, not the allocation.

Frequently asked questions

Is a multi-armed bandit more accurate than an A/B test?

No. For the same traffic, a fixed A/B test gives a more precise estimate of the difference between variants, because it keeps feeding the losers. A bandit is more profitable during the test. Which one is "better" depends on whether you want the estimate or the money.

What is Thompson sampling in A/B testing?

Thompson sampling is a bandit method that gives each variant a share of traffic equal to its probability of being the best, based on the data so far. Variants that are probably better get more visitors, but no variant drops to zero while there's still a chance it's the winner.

What's the difference between Thompson sampling and epsilon-greedy?

Thompson sampling sizes each variation's share by how likely it is to be the best, so the split tightens smoothly as evidence builds. Epsilon-greedy gives a fixed 90% to whichever variation currently leads and spreads 10% across the rest, so it's simpler but can swing between variations when results are close.

Which bandit algorithm is best for low traffic?

One that refuses to bet early. Under about 1,000 visitors per variation, Thompson sampling and epsilon-greedy can lock onto early luck. Splitlab Adaptive keeps the split near even until the results are convincing, then moves at most 15 points per update.

Can a bandit pick the wrong winner?

Temporarily, yes. A well-built bandit never sends zero traffic to a contender, so an early unlucky streak gets corrected as more data arrives. The risk is highest with tiny traffic, aggressive algorithms and lagging metrics.

How often should a bandit update the traffic split?

Often enough to react, rarely enough that each update sees real data. That’s why Splitlab ties updates to new evidence rather than the clock: it re-splits once 20 new conversions have arrived, which is within the hour on a busy funnel and slower on a quiet one.

Can a bandit optimize for revenue instead of conversions?

If the tool supports revenue metrics, yes. In Splitlab the bandit optimizes the test's primary metric, so with revenue per visitor as the primary metric, a variant that converts less but sells more upsells is correctly favoured.

Should I use a bandit for a multivariate test?

They pair well. Multivariate testing produces many combinations, and a bandit drops the weak ones quickly instead of splitting traffic evenly across all of them for the whole run.

My final thoughts

For a funnel business paying for every click, sending half your traffic to a weaker page for weeks has a real cost. That’s why I like bandits for paid traffic and launches.

But a fixed A/B test still earns its place when you need a clean answer, like a pricing change you’ll roll out across every funnel. Most teams I work with end up using both: a bandit to earn, an A/B test to learn.

Pick the method that matches the question you’re asking.

Stop paying for traffic to the losing variant. Turn on Multi Arm Bandit in Splitlab, pick the algorithm that fits your traffic, and optimize on revenue while the test runs. Connect your checkout or CRM from integrations so the bandit sees real revenue, and see the features overview for the rest. Start testing or book a demo.

Built for revenue

Test for revenue, not just clicks.

Run A/B, multivariate, and multi-armed bandit experiments — with real revenue tied to every variation.

We’ll increase your conversion rate by 30% or you don’t pay. Terms apply.

✓

Revenue per variation

✓

A/B & MVT

✓

Multi-armed bandit

Test for revenue, not just clicks.

A/B, multivariate and multi-armed bandit experiments, with real revenue tied to every variation.


Start optimizing for revenue today not just conversion rates.

Run experiments, track revenue per variation, and scale what actually makes you money.

We’ll increase your conversion rate by 30% or you don’t pay. Terms apply.


Start optimizing for revenue today not just conversion rates.

Run experiments, track revenue per variation, and scale what actually makes you money.

We’ll increase your conversion rate by 30% or you don’t pay. Terms apply.


Start optimizing for revenue today not just conversion rates.

Run experiments, track revenue per variation, and scale what actually makes you money.

We’ll increase your conversion rate by 30% or you don’t pay. Terms apply.