Simpson’s Paradox

It was supposed to be simple. A neat little A/B test. Two landing pages, two weeks of simultaneous traffic. Same channel. Same spend. Same audience (or as close as any marketing segment gets).

Every couple of days, the team gathered around the dashboard with the optimism of a pub-quiz team who’d just Googled the first answer. Almost every check-in told the same story: B was flying. A was crawling through the mud.

By the end of Week 1, B looked unstoppable. By mid-Week 2, it was practically victory laps. Someone made a GIF of Landing Page B as Conor McGregor, pacing and shouting at its opponent. Someone else dropped a screenshot of Homer Simpson on his walk-out to fight Drederick Tatum (a bit of foreshadowing that, in retrospect, was almost too satisfying to ignore).

By Friday of Week 2, the tone in the room was pure coronation. The final report? A formality. Just a polished deck before showing the client. Then something strange happened.

When the full data landed, A, the underdog, the slow starter, the embarrassment of Week 1 and Week 2, was the clear winner. The room went quiet. Someone refreshed the dashboard. It didn’t change.

That was the day we met Simpson’s Paradox, the moment two separate truths combine to create one big lie.

Step by step

Week 1

• Page A: 63 conversions from 90 visits (70%)
• Page B: 8 conversions from 10 visits (80%)

Result: B wins Week 1.

Week 2

• Page A: 4 conversions from 10 visits (40%)
• Page B: 45 conversions from 90 visits (50%)

Result: B wins Week 2.

So far, B is winning both weeks. Simple enough. Then we combine the data:

• Page A total: 67 conversions from 100 visits → 67%
• Page B total: 53 conversions from 100 visits → 53%

B wins each week. A wins overall. That’s Simpson’s Paradox, the reversal that happens when you add valid subgroup truths together and get a misleading whole. Every individual truth pointing one way, the whole pointing another.

Why this happens

No one mis-calculated. The maths is fine. The problem is weighting (the hidden villain of tidy dashboards). B looked better in each week because it received more visits in an easier context. A did better where it mattered more, but with smaller sample sizes.

Each slice told the truth. The total forgot the context.

This is what statisticians call a confounding variable, a hidden factor (here, uneven sample size) that changes the weighting of results.

Where you’ll see it

• A/B tests where traffic splits shift week to week
• Campaigns that merge paid and organic results
• Funnels where geography or device mix changes mid-stream
• Quarter-on-quarter comparisons covering non-identical periods
• Any dashboard that trumpets an “overall result” without the segment view beside it

From inside each segment, the story feels consistent. From above, it turns upside down.

How to catch yourself

Before you roll out a “winner,” ask:

• Has the traffic mix changed since the test began?
• Are segments weighted fairly, or just summed?
• Would the trend hold if I controlled for audience or channel?
• Does this still make sense when I zoom in?

If the graph flips when you aggregate, you’ve found the paradox, not a breakthrough.

Guardrails

1. Control your confounders. If your test variants aren’t facing the same conditions (channel, device, geography, or timing) you’re not comparing like with like. Simpson’s Paradox loves an uneven fight.
2. Weight before you merge. Totals lie when one side carries more traffic than the other. Check proportions before you celebrate numbers. A heavyweight and a featherweight can’t share the same scorecard.
3. Keep the segments visible. The paradox only shocks people who never look under the hood. If your dashboard hides the subgroup breakdown, it’s not analysis, it’s wishful thinking.
4. Rerun the result at a constant mix. Once you’ve crowned a winner, replay the test under an even 50/50 split. If the reversal disappears, you’ve found your culprit: weighting, not wonder.
5. When in doubt, fight the same fight twice. If the result still flips, it’s not noise, it’s narrative. That’s the data trying to tell you what you missed.

Tactful pushbacks

Slack / email: “Before we call B the winner, can we double-check the weighting? Simpson’s Paradox loves a lazy total.”

Stakeholder-friendly: “Each week looks fine, but the combined result reverses. The data’s true, the conclusion isn’t.”

Close but not quite

• Ecological fallacy: assuming what’s true for a group applies to each person.
• Base-rate neglect: ignoring the overall probabilities.

Later that night I replayed the data, week by week. Each round, B landed clean shots. Each graph showed momentum. And yet, when the bell rang, A was still standing.

It reminded me of Homer’s fight with Drederick Tatum, “Why Can’t We Be Friends” blaring over the speakers as Homer took hit after hit, too stubborn (or too oblivious) to fall down. That’s Simpson’s Paradox in marketing form: every micro looks like a knockout, until the total steps back and asks who’s actually upright.

The team thought it was a B-shaped triumph. Slack threads were lit, dashboards gleaming green, everything’s coming up Milhouse. And then the ever-cromulent data put its arm around us and quietly said, “Don’t.”