What sells with what
Every bill your business prints is a small confession. It tells you what a customer needed together on one trip, in one cart, on one order.
Multiply that by a few lakh bills and a pattern appears. Market Basket Analysis (MBA) is how you read those patterns at scale, and turn them into bundles, schemes, shelf layouts and cross-sell lists.
What MBA is, and what it isn’t
Almost everyone has heard the beer-and-diapers story, the poster child of market basket analysis. Fewer know the store never actually moved the shelves: a pattern, with no “why” and no action. Let’s not retell it.
MBA thinks in baskets, not in people. A basket is one transaction: one bill, one order, one cart. MBA doesn’t know who you are, what you bought last month or what you like. A one-time walk-in and a loyal weekly customer look exactly the same to it, basket for basket.
That makes it different from personalised recommendation:
- MBA: “people who buy A also buy B”. Everyone who adds A sees the same suggestion, like a “frequently bought together” strip.
- Recommender systems: “you, specifically, will like B”. That’s for later.
If you want to see how “customers like you” is measured, Part 3 of Notions of Distance covers Jaccard and Hamming similarity.
One side effect of “every basket is equal”: a weekly customer contributes 52 baskets a year and a one-timer contributes one. If your regulars are mostly contractors and your one-timers are homeowners, the rules will quietly reflect contractor behaviour.
Under the hood: association rules
The backbone of MBA is association rule mining. It scans thousands of baskets, finds the item combinations that keep turning up together, and states them as rules: “if a basket has X, it often has Y too.” The method comes from Agrawal, Imieliński and Swami at IBM (SIGMOD 1993), built on supermarket data.
Three terms carry the rest of this piece:
- Itemset: any group of one or more items. {WC} is a 1-itemset; {WC, seat cover} is a 2-itemset. The ones that show up often enough are called frequent itemsets, and only those become rules.
- Rule: written {X} → {Y}, e.g. {phone} → {power bank}. It means baskets with a phone often also contain a power bank. Not always, and not because of the phone.
- Antecedent and consequent: X (the phone) is the antecedent, the “if”. Y (the power bank) is the consequent, the “then”. Direction matters: most power-bank buyers don’t buy a phone.
But we needed a clever algorithm
With n unique items there are 2ⁿ − 1 possible combinations.
example: Two products, Phone and Watch, give 3: both, phone only, watch only. The count explodes from there:
No computer checks 10³⁰ combinations one by one. That’s the problem Apriori solves.
Apriori and support, step by step
Apriori grows combinations one size at a time, and only builds bigger ones from smaller ones that have already proven popular. The yardstick for “popular” is support.
Support, defined
Support is the share of all baskets in the data that contain an itemset.
N is the total number of baskets or transactions. N_XY is the number of baskets containing both X and Y; other items may be in the basket too. For a single item, Support({X}) = N_X / N.
Three properties to keep in mind:
- It runs from 0 to 1 (0% to 100%).
- It has no direction: {Phone, Case} is the same as {Case, Phone}.
- It can only fall as you add items. A pair can never be more common than either of its items. That one fact is what lets Apriori skip work.
The running example: 8 baskets
A small electronics shop. We follow three products (Phone, Case, Power bank) across 8 bills, two of which hold other things.
Step 0: you set the bar
You decide the minimum support. Say an item or combination must appear in at least 2 of the 8 baskets, which is 25%. This is a business choice, not something the algorithm decides.
Apriori is only as smart as the bar you give it. Set it too high and you’ll rediscover your best-sellers, while niche but high-margin combinations never surface. Set it too low and you’ll drown in coincidences.
Start with two items: Phone and Case
Two items give 3 possible combinations: {Phone}, {Case}, {Phone, Case}.
- Count single items. Phone is in baskets 1, 2, 3, 4, 7: support 5/8 = 62.5%. Case is in 1, 2, 4, 5: 4/8 = 50%.
- Drop the rare. Both clear 25%, so both survive.
- Build pairs only from survivors. {Phone, Case} becomes a candidate: worth counting, not yet frequent.
- Count the pair. Baskets 1, 2, 4: support 3/8 = 37.5%. It survives.
- Try to go bigger. No third item, so stop.
Where’s the skipping? Suppose Case had appeared in only 1 basket. Step 2 drops it, and {Phone, Case} is never counted, because a pair can’t appear in more baskets than either of its items. That saves one count here. At scale, the same logic saves billions.
Add a third item: Power bank
Three items give 2³ − 1 = 7 possible combinations. Same bar of 25%.
The triple is skipped without being counted. A triple becomes a candidate only if all three of its pairs survived, and Case + Power bank didn’t. Result: 6 of 7 combinations counted.
The loop, in plain words: count the combinations of the current size, drop the rare ones, build the next size only from survivors whose every part also survived. Stop when nothing new survives.
Step 0, revisited
Raise the bar to 4 of 8 baskets (50%), same data. Power bank (3 baskets) drops out at size 1. The only pair left to check, Phone + Case, sits in 3 baskets and fails too. The output shrinks to {Phone} and {Case}: no pairs, no rules.
Same baskets, different bar, different business answer. Step 0 decides what you find, not just how fast you find it.
Confidence: given X, how likely is Y?
Apriori has used our first metric without naming it. But a frequent pair isn’t a rule yet. To turn {Phone, Case} into “phone buyers also take a case”, we need two more numbers. The first is confidence.
Confidence is a conditional probability: of the baskets that contain X, the share that also contain Y.
The shift from support is the denominator. Support divides by all baskets; confidence divides only by the baskets that already have X in them.
Each frequent pair can be read in both directions, so our two surviving pairs give four rules:
What to watch for:
- Direction matters. The same pair gives 60% one way and 75% the other, because the denominators differ.
- It’s always at least the pair’s support, because N_X can’t exceed N.
- Minimum confidence is a second business choice, a Step 0 for rules rather than itemsets. At 50%, Phone → Power bank drops out.
- It’s blind to popularity. Phone is in 62.5% of all baskets, so any rule ending in “→ Phone” looks strong almost by default. That blind spot is what lift fixes.
Lift: more often than chance?
Lift asks the honest question: do X and Y turn up together more than they would by pure chance?
The denominator is how often the pair would appear together if the two items had nothing to do with each other. So lift is actual ÷ expected by chance.
Lift, intuitively
Lift compares two numbers: how likely a random shopper is to buy Y, and how likely a shopper who already has X in the basket is to buy Y.
In our shop, Case is in 4 of 8 baskets, so a random shopper takes a case 50% of the time. Among the 5 phone buyers, 3 took a case: 60%. And 60% ÷ 50% = 1.2. Phone buyers are 1.2 times as likely to take a case as an average shopper.
The milk trap. Suppose milk is in 90% of all baskets, and 90% of bread buyers also take milk. Confidence for bread → milk is an impressive 90%. Lift is 90% ÷ 90% = 1. Bread buyers take milk exactly as often as everyone else, so bread tells you nothing about milk.
The party picture. Two friends each go to half of all parties. If they choose independently, chance puts them at the same party about 25% of the time (½ × ½). If they actually turn up together 40% of the time, lift is 1.6: something links them. If only 10%, lift is 0.4: something keeps them apart. Maybe they avoid each other; maybe they just move in different circles. The number won’t tell you which.
Reading lift
On our 8 baskets:
Two things stand out. Power bank → Phone looked strong at 67% confidence, but its lift of 1.07 says popularity explains most of it. And Case + Power bank’s 0.67 rests on a single basket; Apriori had already pruned it. Lift on low-support pairs is noise until proven otherwise.
Lift is symmetric: Phone → Case and Case → Phone share the same 1.2. Which way to act on it is a business call.
Putting it together: one decision flow
Three gates and two business choices turn 8 baskets into one decision. Support finds what’s common, confidence finds what follows, and lift checks whether it’s more than chance.
Only Phone ↔ Case passes all three: common enough (37.5%), reliable enough (60–75%) and more than chance (1.2×). The decision that follows: show cases on the phone page and at the phone counter, and test a phone + case bundle against no bundle.
Setting the baseline for support, confidence, and lift
There’s no magic number. The right bar depends on how many baskets you have, how full they are, how sales spread across SKUs, and what decision you’re making.
Catalogue size still matters. If a typical basket holds about 4 items, each SKU’s average support is roughly 4 ÷ the number of SKUs: about 8% at 50 SKUs, 4% at 100, 0.8% at 500, and 0.4% or less beyond 1,000. Pairs are rarer still, so a bar that works for 50 SKUs finds nothing at 1,000.
Indicative starting points. However, these are not industry standards, so tune them on your data:
Confidence bars fall as the catalogue grows, because any specific Y is rarer. Lift bars rise, because rare items throw up noisy, inflated lifts. For reference, Pradhan and Dinesh Kumar use a minimum support of 2% on the classic Groceries dataset: about 9,800 baskets across 169 product categories.
What practitioners lean on instead of a fixed number:
- A minimum count, not just a percentage: e.g. a pair must appear in at least 20–30 baskets.
- Work backwards from the 20–100 rules a manager can actually review.
- Sensitivity checks: run two or three bars and keep the rules that survive all of them.
- Roll up when sparse: at 1,000+ SKUs, mine at category level first, then drill down.
- Set bars per channel: a smaller D2C channel needs a lower bar than retail.
Where MBA works, and where it struggles
MBA works best where baskets are frequent, carry many items, and hold natural complements. That’s not only frequent purchases: a well-planned project can be just as rich.
It struggles in two places:
- Rare or single-item purchases: cars, mattresses, a single large appliance. Few baskets, and nothing co-occurs.
- Influenced purchases: promo bundles, combo offers, a strong sales push. MBA will find the rule, but it reflects your scheme, not the customer’s need.
It needs scale, and it’s worth splitting by channel
With a handful of SKUs, the rules just confirm what any sales manager already knows. MBA earns its keep when the catalogue is too big for a human to hold in their head, sold across many channels.
Run it per channel too: retail, e-commerce and D2C. The same customer may build a very different basket in a showroom than in an app. Pooled data can hide this: a strong e-commerce-only rule gets diluted below the bar, or a pooled rule looks universal when it really comes from one channel. The cost of splitting is fewer baskets per channel, so smaller channels need lower bars.
Too many SKUs creates the opposite problem: baskets get sparse and pairs become rare. Mine at category or sub-category level first, then drill down. A rule like faucets → health faucets can be invisible at SKU level.
What MBA won’t do for you
It shows what, not why
MBA gives you correlation, not causation. A rule like {tile adhesive} → {grout} is a co-occurrence pattern. It doesn’t say why. Maybe customers need both for one job. Maybe the two sit next to each other on the shelf. Maybe a contractor’s checklist bundles them.
The “why” is the actual customer insight, and it still needs research, store observation and domain knowledge. Even a lift well above 1 says the items go together, not why. Treat every rule as a hypothesis, and test it (a bundle offer, a shelf placement, an A/B test) before you scale it.
It needs history: the cold-start problem
No transactions, no rules. A new business, a new store or a newly launched product starts with nothing to mine. Cold start hits SKUs too: an established brand still can’t find rules for a range launched last week.
Until your own data builds up, borrow patterns from outside:
- Competitor or industry data, where you can get it; raw competitor baskets are rare, but syndicated panels exist in some categories.
- Analogous stores or markets: transfer rules from a similar city or format.
- Product logic: items used in the same job get bundled. A basin goes with a faucet and a waste coupling.
- Expert-seeded bundles, then let the data confirm or kill them once transactions arrive.
The payoff: it arms the expert
MBA doesn’t replace the knowledge and experience of domain experts. It strengthens them. Experienced sales and category people already know the obvious pairings. What MBA adds is the non-obvious ones, the combinations nobody thought to look for.
That gives a short- to medium-term edge in two ways:
- Sharper schemes. Customer and trade schemes built on what people actually buy together are more relevant and earn a higher ROI. A dealer scheme on the highest-lift pair beats a blanket volume target. A dealer whose orders lack the usual companion item is a ready cross-sell list.
- Better products. If a third-party accessory keeps showing up beside your product, add it to the feature set or the range, and charge a premium that lifts value and margin per basket.
The partnership runs both ways. Obvious rules are a sanity check: if the algorithm can’t find the pairs experts expect, check the data. And experts are the filter: a run can throw up thousands of rules, and domain knowledge picks the 20 worth acting on.
Why only short to medium term? Buying patterns drift, competitors run the same analysis, and a successful scheme changes the very behaviour it was built on. So run MBA as a cadence, by season and by channel, not as a one-off project.
Action plan: from code to conclusions
The code implementation is there to draw conclusions and learnings from the data, not to show off syntax. Every step below ends in a business learning, and most of them test a claim made above on real transactions.
Dataset: UCI Online Retail II, about 1.07 million rows from a UK online gift-ware retailer, December 2009 to December 2011, with many wholesale customers across several countries. Licence CC BY 4.0. Built in Google Colab with mlxtend.
What the data showed
On about a million order lines from a UK gift-ware wholesaler, we followed the same steps as the action plan above.
Step 0 & Stage 1: Apriori algorithm with support
The apriori algorithm was run with a support of 2% (0.02). I wanted to go lower than 2% however the algorithm didn’t run for the need of more compute power.
At 2% support level, I managed to isolate 130 itemsets, highest support level was around ~10%.
Out of these 130 itemsets, 4 itemsets have more than one product.
Stage 2: Confidence
On these isolated 130 itemsets, I ran association rule with a confidence of 5% threshold. The confidence filter generated 8 combinations. All these combinations had exactly one SKU in antecedent and one SKU in consequent.
Stage 3: Analysing lift
The lift for the 8 combinations varies between 6 and 15, these are very assertive numbers.
Scatterplot is the best visualisation to observe the interplay among Support, confidence, and lift.
The best combinations to pick are Top-right, the most uncertain are bottom-left.
Directed graph is another visualisation that suggests directional association between antecedent and consequent. It also suggests that an antecedent in one graph can be a subsequent in another graph.
Reading the directed graph:
Rule R3 (Bottom-Center Node) – Antecedent (If): 21733-Red Hanging Heart T-Light Holder Consequent (Then): 85123A-White Hanging Heart T-Light Holder
And then there are several other plots to visually direct the user towards powerful itemsets.
Top product associations by Support
Top product associations by lift
Now run it on your own baskets
The best way to understand market basket analysis is to point it at your own bills. The complete notebook is free to copy and run: Market Basket Analysis on GitHub.
All you need is a file with one row per line item, holding at least an invoice or order number and a product code or name. Then:
- Swap in your file and map its invoice and product columns.
- Clean first: drop cancellations, returns, fees and postage, and keep one name per product.
- Set Step 0 from the indicative starting points for your catalogue size, then try two or three other bars.
- Shortlist the rules that pass support, confidence and lift, and ask your sales team why those items travel together.
I’d love to hear the most surprising pair you find. Share it in the comments or tag me on LinkedIn.
Coming up
This is the first post in a series on business problems solved with AI and ML. Market basket analysis answers “what goes together in one basket”. The natural next question is “what will this customer want next”, which is the job of recommender systems, and the BigBasket case is where we’ll start.
The annotated notebook is on GitHub.
Acknowledgements and references
This piece owes a debt to Manaranjan Pradhan, whose treatment of association rules in Machine Learning using Python (with U Dinesh Kumar) and whose BigBasket case at IIM Bangalore shaped how I think about baskets and recommendations.
- Manaranjan Pradhan and U Dinesh Kumar, Machine Learning using Python, Wiley India, 2019. Chapter 9.2, Association Rules.
- Paul Abraham, Manaranjan Pradhan, Lakshminarayanan S, Ganesh Iyer and Dinesh Kumar Unnikrishnan, Customer Analytics at Bigbasket: Product Recommendations, IIM Bangalore case IMB573, 2016.
- R. Agrawal, T. Imieliński and A. Swami, “Mining Association Rules between Sets of Items in Large Databases”, SIGMOD 1993.
- R. Agrawal and R. Srikant, “Fast Algorithms for Mining Association Rules”, VLDB 1994.
- D. Chen, Online Retail II, UCI Machine Learning Repository, 2012.
- Beer and diapers: the true story, The Register, 2006.
Leave a Reply