Is a 200% Conversion Lift From AI Product Recommendations Real? What Controlled Research Shows

Short answer: a 200%+ lift is usually real as a click-through or engagement number and seldom real as a site-wide revenue number. Randomized field experiments on large retailers find that recommendations lift sales, but by a figure closer to 10% than to 200%. McKinsey’s research puts the typical revenue lift from personalization at 10 to 15 percent, with company-specific results ranging from 5 to 25 percent.

Neither side of that gap is lying. They measure different things. This article lays out the best verifiable evidence, explains why headline numbers and controlled results diverge, and gives you a way to measure what a recommendation engine does for your store.

In this article

  1. What the evidence says, at a glance
  2. Why headline numbers and controlled results diverge
  3. Attributed revenue vs. incremental revenue
  4. What randomized experiments actually found
  5. A sanity check you can run on any vendor claim
  6. Why smaller stores often can’t see the lift
  7. When recommendations underperform
  8. How to measure your own lift
  9. What the evidence can’t tell us yet
  10. FAQ

What the evidence says, at a glance

The table separates three kinds of evidence, because they answer three different questions.

SourceType of evidenceWhat was measuredResultMain caveat
McKinsey, Next in Personalization 2021Cross-company researchRevenue lift from personalizationMost often 10–15%; company-specific 5–25%Covers personalization broadly, not recommendation widgets alone
McKinsey, 2013Industry analysisShare of purchases from recommendations35% of Amazon purchases; 75% of Netflix viewingAttribution, not causal lift. Over a decade old
Gomez-Uribe & Hunt (Netflix), 2015Company-authored paperShare of hours streamed influenced by recommendationsAbout 80%“Influences choice” is not “would not have happened otherwise.” Over a decade old
Sharma, Hofman & Watts, 2018Natural-experiment analysis of Amazon dataShare of recommendation-attributed traffic that is causalDepending on category, 50–80% would have happened anywayMeasures page traffic, not purchases
Kumar & Hosanagar (ISR)Randomized experiment, major department storeNet effect on product salesOverall increase of about 11% (as reported)One retailer, one recommender type
Lee & Hosanagar (Management Science, 2020)Randomized experiment, 184,375 usersEffect on views and on conversionBoth increase; size varies by product type and ratingsOne retailer, one recommender type
Lee, Gopal & Park (ISR, 2020)Randomized experiment, South Korean retailerViews, sales, click-through, conversionAll improve; larger effect on mobileSingle retailer

Two patterns stand out. The biggest-sounding figures (35%, 75%, 80%) are attribution numbers: they count what happened after someone interacted with a recommendation. The smaller figures come from experiments, which compare shoppers who saw recommendations with shoppers who did not.


Why headline numbers and controlled results diverge

A shopper’s journey has several measurable stages: seeing a recommendation, clicking it, adding the item to the cart, completing the purchase, and the size of the final order. A recommendation engine tends to change the early stages by a lot in percentage terms, because it creates clicks that barely existed before. By the time you reach revenue for the whole store, those changes are diluted by every visitor who never touched a recommendation and every purchase that would have happened anyway.

So a vendor can report a very large lift at the click stage, and a modest lift at the revenue stage, and both can be accurate. The problem appears when the click-stage number is quoted as if it were the revenue-stage number.

Two further points change how you should read any “conversion lift” claim:

  • Revenue per session is conversion rate multiplied by average order value. A recommendation engine can leave conversion nearly flat and still raise revenue if shoppers add more items. Judging it on conversion rate alone can make a working engine look like a failure.
  • “Conversion rate” has several definitions. It can mean sessions that end in a purchase, product views that end in a purchase (the definition used in the Lee & Hosanagar experiments), or purchases among only those shoppers who clicked a recommendation. These produce very different numbers from the same data.

Attributed revenue vs. incremental revenue

This is the single most important distinction in the whole topic.

  • Attributed revenue is revenue from sessions where the shopper interacted with a recommendation.
  • Incremental revenue is the extra revenue that exists because the recommendation was there.

They differ because shoppers who click recommendations are often already interested in what is being recommended. Recommendation algorithms typically suggest items that are frequently viewed or bought together, so many of those items are ones shoppers would have found anyway through search or browsing.

Research on Amazon’s own data quantifies this. Sharma, Hofman and Watts found that the widely used click-through metric overestimates the causal impact of recommender systems: depending on product category, they estimate that 50–80% of the traffic attributed to recommendations would have happened even without them. That finding concerns product-page traffic rather than sales, so it should not be applied directly to revenue. It does show that an attributed figure can overstate causal impact by a large margin.

Anuj Kumar of the University of Florida reached a similar skepticism from the sales side. In the university’s summary of the study he co-authored with Kartik Hosanagar, he called the 35%-of-Amazon-sales estimate “hard to believe,” pointing to two inflating factors: shoppers can find what they want through search and filters, and recommendations are built from items that are already commonly viewed or bought together.

The attributed numbers are still informative. They show that recommendations sit in the path of a large share of activity at companies like Amazon and Netflix. They just don’t show how much of that activity the system created.


What randomized experiments actually found

Randomized experiments are the gold standard here, because random assignment means any difference between groups can be attributed to the recommender. Four are worth knowing.

Kumar & Hosanagar: a major department store. The researchers analyzed more than two million product-page views across nearly a million visitor sessions, comparing shoppers who saw recommendations with a control group who did not. According to the University of Florida’s summary, recommendations raised sales of the recommended products by 9 percent but reduced sales of the original product the shopper was viewing by 1.9 percent, for an overall product-sales increase of about 11 percent. The summary reports these figures as stated; consult the paper for exactly how they are aggregated. The effect was uneven: popular products linked to many other items gained more, while some less-popular items sold worse with recommendations.

Lee & Hosanagar: effects by product type. In an experiment with 184,375 users on a top North American retailer’s site, a purchase-based collaborative-filtering recommender increased both product views and the conversion rate from view to purchase. The size of each effect depended on the product. The impact on views was greater for utilitarian products (a hammer, in the authors’ example) than for hedonic ones (perfume). The impact on conversion was greater for hedonic products, and greater for products with lower average review ratings, which suggests a recommender can partly substitute for strong reviews.

Lee & Hosanagar: effects on sales diversity. In a separate analysis of a randomized experiment covering 82,290 products and 1,138,238 users, they found that niche items gained in absolute terms but lost market share as recommendations concentrated attention on popular items. They also found that recommenders based on purchase data had a larger effect than those based on product views.

Lee, Gopal & Park: mobile vs. desktop. A randomized experiment with a South Korean online retailer found that recommendation systems improved views and sales of recommended products, click-through, and conversion, and that the effects were significantly larger for mobile users, which the authors attribute to the higher search costs of mobile browsing.

Taken together, controlled evidence supports the direction of the vendor claim (recommendations help) while pointing to a much smaller magnitude than the headline numbers, and it shows the effect depends on product mix, device, and algorithm design.


A sanity check you can run on any vendor claim

You can pressure-test a claim in a few minutes with two questions and one piece of arithmetic.

Question 1: Which stage does the number describe? Click-through, add-to-cart, conversion among exposed shoppers, conversion store-wide, or revenue? If the vendor can’t say, the number is not usable for forecasting.

Question 2: Was there a randomized control group? If the comparison is “sessions with recommendation clicks vs. sessions without,” it is an attribution figure. Shoppers who click recommendations differ from shoppers who don’t.

The arithmetic. If a dashboard attributes a share of revenue to recommendations, discount it for activity that would have happened anyway. A purely hypothetical example: a dashboard attributes 20% of revenue to recommendations. If 50–80% of that would have happened anyway, the incremental share is 20% × (1 − 0.5) = 10% at the optimistic end and 20% × (1 − 0.8) = 4% at the pessimistic end. That 4–10% range sits in the same neighborhood as the McKinsey and experimental figures above.

Treat this as a plausibility check, not a forecast. The 50–80% figure comes from Amazon page traffic, and your catalog, traffic mix, and algorithm will differ.

How the lift can hide in the details. Because revenue per session is conversion rate multiplied by average order value, the two can move very differently. As a hypothetical, a 5% rise in conversion combined with a 20% rise in average order value gives 1.05 × 1.20 = 1.26, a 26% rise in revenue per session. Watching only conversion would show 5%.


Why smaller stores often can’t see the lift

Even if a recommender works, a small store may be unable to prove it. Detecting a small difference between two groups takes a lot of visitors.

A standard rule of thumb for an A/B test (Lehr’s rule, at roughly 80% power and a 5% significance level) is that you need about 16 × p(1 − p) ÷ δ² visitors per group, where p is the baseline conversion rate, and δ is the absolute difference you want to detect.

With a 2% baseline conversion rate:

Relative lift you want to detectAbsolute difference (δ)Approx. visitors needed per group
20% (2.0% → 2.4%)0.4 pointsabout 19,600
10% (2.0% → 2.2%)0.2 pointsabout 78,400

These are approximations, and testing revenue per session needs more traffic because order values vary widely. The takeaway is practical: if realistic lifts are around 10%, a store with a few thousand sessions a month cannot measure its effect in a reasonable time. Such a store is better off with simple, inspectable rules, such as “frequently bought together” lists reviewed against its own sales data, than with a paid engine whose benefit it can never confirm.

Note that this is a statement about measurability. It does not claim a universal minimum traffic level for a recommender to work. Widely repeated thresholds for “enough data” usually come from single vendor blog posts, and the experiments reviewed here do not establish one.


When recommendations underperform

The research points to several recurring causes.

  • Cannibalization. In the department-store experiment, recommendations reduced sales of the product the shopper was originally viewing. Some of what a recommender “wins” is simply moved from another item.
  • Concentration on popular items. Recommendations can shift attention toward bestsellers, which can hurt niche products’ market share even when their absolute sales grow.
  • Product mix. The effect on conversion is larger for hedonic products and for lower-rated products, and the effect on views is larger for utilitarian ones. A catalog skewed the wrong way may see less benefit.
  • The wrong scoreboard. Judging a recommender by site-wide conversion rate hides changes in order value, and judging it by attributed revenue hides what would have happened anyway.
  • Poor placement and timing. The Kumar study’s authors suggest withholding recommendations when shoppers are close to buying, to avoid distracting them. This is their recommendation from one retailer’s data, so test it on your own site.

How to measure your own lift

A holdout test is the most reliable method, and it needs no special research expertise.

  1. Randomize at the visitor level. Assign each visitor to see recommendations or to see a plain default (no widget, or a simple bestseller list), and keep them in the same group across visits.
  2. Run both at the same time. Comparing before and after launch will pick up seasonality, promotions, and traffic changes and credit them to the recommender.
  3. Pick the primary metric before you start. Revenue per visitor is a strong default because it captures both conversion and order value.
  4. Size the test first. Use the rule of thumb above to estimate how long you need to run, and don’t stop early because the numbers look good.
  5. Track the whole store, not just the widget. Include sales of the products people were originally viewing, so cannibalization shows up.
  6. Keep a small permanent holdout if you can. It gives you a continuing estimate as your catalog and algorithm change.

Report the result as incremental revenue (the revenue difference between groups, scaled to the traffic that saw recommendations) rather than as a percentage of a metric that no one agrees on.


What the evidence can’t tell us yet

Being honest about gaps is part of reading this research well.

  • Most-cited figures are old. The Amazon 35% figure dates to McKinsey’s 2013 analysis, and the Netflix 80% figure to a 2015 paper. They are useful as evidence of scale, not as current benchmarks.
  • Netflix figures differ by source. McKinsey’s 2013 article says 75% of what people watch comes from recommendations, while the 2015 Netflix paper says recommendations influence about 80% of hours streamed. These are different measures from different sources.
  • The experiments cover specific retailers and recommender types. The randomized studies above involve a few retailers and collaborative-filtering-style systems.
  • I found no public randomized evidence for newer LLM-based recommenders. Claims about them should be treated as vendor claims until a holdout-tested result is published.
  • Vendor case studies are mostly unverifiable. Many carry anonymized client names, undisclosed methods, or unspecified metrics, so this article does not rely on them.

FAQ

Is a 200% conversion lift from AI recommendations realistic?

Not as a store-wide result. A 200%+ figure usually describes click-through or engagement on recommended items. Randomized experiments point to overall sales gains around 10%, and McKinsey reports a typical personalization revenue lift of 10–15%, with a 5–25% range by company.

How much do product recommendations increase sales?

In a large randomized experiment at a department store, the authors reported an overall product-sales increase of about 11%. McKinsey’s cross-company research finds personalization most often lifts revenue 10–15%. Your result will depend on your catalog, traffic, and algorithm.

What is the difference between attributed and incremental revenue?

Attributed revenue is sales from sessions where a shopper interacted with a recommendation. Incremental revenue is the extra revenue that exists because recommendations were shown. Research on Amazon data estimates that 50–80% of recommendation-attributed traffic would have happened anyway, depending on category.

How do I measure whether a recommendation engine works?

Run a randomized holdout test: show recommendations to one group of visitors and a plain default to another, at the same time, and compare revenue per visitor. Decide the metric in advance and run the test long enough to reach the sample size you need.

What should I track instead of conversion rate alone?

Revenue per visitor, average order value, and sales of the original products people viewed, so cannibalization is visible. Conversion rate alone can miss an order-value effect and can include visitors who never saw a recommendation.

Do recommendations work better on mobile?

One randomized experiment with a South Korean online retailer found significantly larger effects of a recommendation system for mobile users than for PC users. That is one retailer’s result, so test it on your own audience.


Sources

Figures are quoted from the sources above as described in their abstracts, official pages, or institutional summaries. Hypothetical examples are labeled as such. Last fact-checked November 2026.


Related reading

Leave a Reply

Your email address will not be published. Required fields are marked *