Personalisation lift: why you must run a controlled test

Personalisation A/B testing is the only honest way to answer the question every buyer of a recommendation engine eventually asks: did it work? A before-and-after chart cannot answer it. Revenue rose in the month after launch, but so did paid traffic, a festival fell in that month and the merchandising team ran a sale. A held-out control group, chosen at random and shown the old experience, removes every one of those confounders at once. This article explains how to run that test, what to measure, what goes wrong, and how to present the result so it is believed.

Why before-and-after comparisons fail

Every business has trends it does not control. Seasonality moves conversion by itself. Marketing spend changes the mix of new and returning visitors, and returning visitors always convert better. A competitor's stock-out sends you customers who were going to buy anyway. When you compare the month after a personalisation launch with the month before, you are measuring all of those things plus your new system, and you cannot separate them. Vendors know this, which is why their case studies are so often before-and-after. Treat any personalisation ROI figure without a control as unproven.

Personalisation A/B testing compared with the alternatives

How to design a recommendation A/B test

Choose one primary metric before you start

Pick the metric that the business case was written on and commit to it in writing. For an e-commerce recommender that is usually revenue per visitor or conversion rate; for content it is sessions per user or retention; for a subscription product it is activation. Clicks on the recommendation widget are not the metric. A widget can attract clicks while the visitor buys exactly what they would have bought anyway. Secondary metrics are fine, but they are for diagnosis, not for declaring victory.

Randomise on the user, not the session

Assign each user to treatment or control once, using a hash of a stable identifier, and keep them there. Randomising by session puts the same person in both groups on different days, which dilutes the effect and makes retention metrics meaningless. If your identity across devices is weak, this is a reason to fix the event pipeline first, not a reason to skip the test.

Size the test in advance

Decide the smallest lift you would act on, look at the variance of your metric, and calculate how many users each arm needs to detect that lift. Then run for at least one full business cycle, usually two weeks, so weekday and weekend behaviour are both included. Stopping the moment the dashboard turns green is the most common way teams fool themselves; the number will drift back as the sample grows.

Keep the control genuinely untouched

The control group must see the experience they would have seen without the project: the old "bestsellers" carousel, the manual merchandising, the rules engine. Do not show them an empty widget, because that measures the value of any widget rather than the value of personalisation. And keep the marketing team informed, so a campaign is not sent only to the treatment group by accident.

Measuring personalisation ROI from the result

Once the test has run, the lift is the difference between arms on the primary metric, expressed with a confidence interval. Multiply the point estimate by annual traffic to get a gross value, then subtract running costs: inference, the feature store, the event pipeline, and the care plan that keeps it healthy. Present the low end of the interval as well as the point estimate. Finance trusts a range they can see the edges of far more than a single large number. Where the interval crosses zero, say so; a null result on one placement is useful information and often points at a better placement.

Uplift testing: who benefits, not just whether

A single average lift hides the fact that personalisation helps some segments a great deal and others not at all. Uplift testing splits the result by new versus returning, by device, by category and by traffic source. Returning users with rich histories usually show the largest lift; brand-new visitors show little, because the system has nothing to go on until the cold-start fallbacks kick in. Knowing this changes the roadmap: it tells you where to invest next and which segments to leave on simpler rules.

Common mistakes that invalidate the test

• Peeking daily and stopping early on the first significant reading
• Randomising by session or by page load instead of by user
• Letting the control group receive a partial version of the new experience through a shared cache
• Changing the recommender mid-test and treating the whole period as one experiment
• Reporting widget click-through as if it were revenue lift
• Running the test over a festival week and generalising the result to the rest of the year
• Forgetting that a lift in one placement can cannibalise another

After launch: keep a holdback

The test does not end at launch. Keep a small random holdback on the old experience so you can report lift every quarter as the model, the catalogue and the traffic change. A recommender that showed strong lift at launch can decay quietly as the catalogue turns over or user behaviour shifts, and a holdback is the only way to notice. It also lets you evaluate every model change against a control rather than against last month, which is the same discipline we apply to every AI system we ship: evaluations over demos.

A worked example

A mid-sized D2C brand had a vendor-supplied recommendation widget and a slide in the board deck claiming it drove a large share of revenue. When we were asked to replace it with a personalisation engine, the first thing we did was set up a proper test rather than a demo. Users were hashed into three arms: the vendor widget, the new engine, and a plain bestsellers carousel as control. The test ran for two full weeks across the storefront and the WhatsApp channel, with revenue per visitor as the single primary metric. The result was instructive. The vendor widget was statistically indistinguishable from bestsellers, which explained why a huge share of "attributed" revenue had never shown up in the bank. The new engine showed a clear lift for returning customers and almost none for first-time visitors. That segmented result shaped the next phase: better cold-start handling and a messaging programme for returning customers, both measured against the same holdback. The engagement is described in the D2C personalisation case study.

Team and timeline

Designing and running a controlled test needs a data engineer to build the assignment and the metric pipeline, an analyst or data scientist to size the test and read the result, and a product owner who can keep marketing and merchandising from contaminating the control. For a client with a storefront and reasonable event data, setup takes about two weeks and the test itself two to four weeks. If you have no recommender yet, the experiment framework is built as part of the personalisation engines service, which starts at $21,000 / ₹13.6L. If you want to validate the approach first, the AI POC Sprint at $6,250–10,500 runs one placement against a control in three weeks. Ongoing holdback reporting sits under a Care Plan; see the pricing page for the tiers.

Before you start: a checklist

• Write down the single primary metric and the lift you would act on
• Confirm you can assign users to arms with a stable identifier across devices
• Calculate the sample size and the minimum run length before launch
• Define what the control group sees and confirm nothing new leaks to it
• Tell marketing and merchandising the dates so campaigns hit both arms equally
• Agree who reads the result and that the low end of the interval will be reported
• Plan the permanent holdback that continues after launch

Glossary

• Control group: users randomly held out and shown the old experience during the test
• Lift: the difference in the primary metric between treatment and control
• Confidence interval: the range within which the true lift plausibly sits
• Holdback: a small permanent control kept after launch for ongoing measurement
• Interleaving: mixing two rankers' results in one list to compare them with less traffic
• Uplift testing: splitting the lift by segment to see who benefits
• Cannibalisation: revenue gained in one placement at the expense of another

Related reading

Start with recommendation engines explained for e-commerce leaders, then real-time ranking within a session for what a well-tested system can do. Google's Cloud Architecture guidance on recommendation systems covers experiment design in a production context. Build and running costs are on the pricing page.

If a personalisation number was not produced against a control, it is a story, not a result; run the test and the story becomes evidence.