Data Dredging

Did you know that spelling bees predict spider deaths? Or that black coffee preference correlates with sociopathy?

Unfortunately, espressos don’t come with a diagnosis. And spiders couldn’t care less whether your eight-year-old can spell Pneumonoultramicroscopicsilicovolcanoconiosis.

The real world doesn’t work like that. But a spreadsheet can. And in 2011, psychology proved just how far you could take it.

That year, a team of psychologists published a peer-reviewed paper showing that, using the standard statistical methods of the time, you could “prove” that listening to When I’m Sixty-Four by The Beatles literally makes you younger. Not metaphorically. Biologically.

They didn’t falsify their data. They didn’t break any rules. They simply used the standard statistical methods of the time.

That paper became a flashpoint for the “Replication Crisis”. A moment when science was forced to confront an uncomfortable truth: many “significant” findings were actually statistical ghosts, patterns conjured by data dredging and p-hacking that vanished the moment anyone tried to repeat them.

Using these same “acceptable” methods, people have “proven” that chocolate accelerates weight loss, that pricing changes how pizza tastes, and that all manner of unrelated variables move together for no causal reason at all.

When large-scale replication efforts followed, the damage became visible. Only around 30 to 50% of prominent findings in social psychology could be reproduced. Theoretically, downstream of social psychology, marketing should have got the memo. Instead, it appears we’ve operationalised the mistake.

The green arrow

Monday morning. Quarterly review. The numbers are flat. Spend is up. ROI is sulking somewhere below one. The room has that fluorescent quiet that usually precedes a budget cut.

Then someone clears their throat. “We dug a little deeper.” Next slide. A glorious, upward-pointing green arrow. +400% lift among iOS users in suburban London between 2pm and 4pm on Tuesdays.

Relief floods the table. The CMO leans back. The phrase “double down” is spoken with conviction. It feels like insight. It feels like survival. It is very often neither.

What data dredging actually is

Data dredging happens when you search through a dataset long enough to find something that looks significant, and then treat that discovery as if it were the plan all along.

It’s the statistical equivalent of shooting at a barn, walking up to the tightest cluster of bullet holes, and painting a bullseye around them.

The maths makes it worse. A p-value threshold of 0.05 doesn’t mean “95% chance this is true.” It means: if nothing is happening, you’ll still see a result this dramatic about 1 in 20 times. That’s fine, if you only ask one question.

But if you ask twenty questions, the odds change. Run 20 independent tests (and in marketing, they’re rarely truly independent), and even if none of your hypotheses is real, there’s about a two-thirds chance at least one will come back “significant” just by luck. Not because the universe whispered the answer. Because you rolled the dice twenty times.

The more segments you slice, the more dashboards you open, the more filters you apply, the more likely you are to mistake noise for narrative.

The marketing version

We call it “digging for insight.” We slice by: age, gender, device, time of day, geo, creative variant, audience overlay, retargeting depth. Somewhere in the cross-section of all that slicing, something will pop. It always does.

And because marketing isn’t a maximally truth-seeking discipline, it’s a persuasion-and-decision discipline, that “something” becomes the story. At best, it misleads customers. At best, it misleads management. At worst, it misleads us: a strategy built on statistical theatre, dressed up as “data-led”, sold as optimisation.

We aren’t usually lying. We’re relieving pressure. Pressure to justify the budget. Pressure to explain performance. Pressure to avoid the phrase “inconclusive”. In academia, they call it publish or perish. In marketing, it’s perform or disappear.

Where you’ll see it

• A/B tests stopped early because the dashboard turned green.
• Campaign decks leading with a niche segment that “over-performed”.
• MMM models tweaked until the right channel wins.
• PR surveys with 40 questions that only publish the one dramatic stat.
• Attribution narratives assembled backwards from the sale.

Every one of them is plausibly defensible. None of them is necessarily true.

Why it works (until it doesn’t)

Once you know the trick, you start seeing its fingerprints everywhere. A journalist famously ran a deliberately weak diet study, measured enough outcomes, and “proved” chocolate accelerates weight loss, then watched it explode across headlines. A personality study about bitter taste preferences (black coffee included) gets compressed into: black coffee drinkers are psychopaths. A website of spurious correlations can show you a high correlation between the Spelling Bee winning word length and deaths by venomous spiders.

None of this requires forged data. It requires two things:

1. enough variables
2. enough willingness to stop searching the moment you find a win

Data dredging isn’t fraud. It’s intolerance of uncertainty. We would rather have a wrong answer with confidence than admit we don’t know yet. And that false confidence is expensive. Because when you scale a fluke, you pay for it twice.

How to catch yourself

Before you celebrate a segment, ask:

• Was this hypothesis defined before we looked?
• How many other cuts did we test?
• Would this hold up in a fresh dataset?
• Are we explaining variance, or just describing it?

If the result only exists after five layers of filtering, it might be a lead. It is not yet a conclusion.

Guardrails

1. Call your shot. Write down the primary metric before you launch. No moving goalposts.
2. Label exploration honestly. If you’re fishing, call it fishing. Exploration is allowed. Pretending it was the plan is the sin.
3. Raise the bar if you slice. The more segments you test, the less impressed you should be by 0.05.
4. Retest surprises. Treat unexpected wins as new hypotheses, not finished truths.
5. Split your data. Find patterns in one half. Prove them in the other.

The napkin test

If you only found the insight after the fifth filter, you didn’t discover it. You might have selected it.

Close but not quite

• Texas Sharpshooter Fallacy: drawing the target after the shot.
• Publication Bias: only the winners make it to print.
• Overfitting: building a model that explains yesterday perfectly and tomorrow not at all.

When the replication crisis hit psychology, parts of science did something rare. It admitted the system was flawed. It made replication fashionable. It built tools like pre-registration and registered reports to separate “we found something” from “we proved something.”

Marketing didn’t have a crisis. We had better dashboards. Which means the responsibility sits with us.

The next time a miraculous green arrow appears in a tiny corner of your data, pause. Not because it’s impossible.

If you torture the data long enough, it will always confess.

The question is whether you were interrogating it, or just asking it what you needed to hear.