Sampling Bias

Dover High Street. Thursday. 11 a.m. Drizzle so fine it felt like static. Two seagulls were fighting over a cigarette end.

I was nineteen, wearing formal shoes and a bright yellow rain jacket so thin it seemed to somehow double the volume of water, the sort of outfit that made me look like a man who’d lost both a job and a bet.

That morning’s assignment was public research. Clipboard. Smile. Stop strangers. Ask them questions about a product I barely understood.

Dover High Street wasn’t in its optimal state. Half the shutters were down. Every face looked like it was waiting for something better to happen, and had been for a while. Most people gave me the same expression: averted eyes, forced politeness, or a muttered insult about my height as they passed. I smiled through it, doing mental arithmetic on how many interviews stood between me and lunch.

After an hour, I looked down at my notes. Every respondent was either a woman in her seventies or a man in his fifties who smelled faintly of Strongbow and rain. The only people willing to stop were the ones with nowhere else to be on a Thursday morning in Dover, and who didn’t mind being spoken to by someone dressed like a banana in leather shoes.

That was when it hit me: I wasn’t surveying the public. I was surveying the subset of the public willing to be surveyed by me.

100% of marketing survey participants are the type of people who answer marketing surveys at least once.

What sampling bias actually is

Sampling bias happens when the people you measure aren’t representative of the people you’re trying to understand. Not because they lied. Because you only reached the kind of people who were easy to reach, or willing to talk.

The result? A dataset that looks complete but is quietly lopsided. The truth is hiding in the people who never showed up.

The marketing version

We do it all the time:

• Poll your LinkedIn followers and call it “market research.”
• Survey your newsletter readers and assume it reflects the entire audience.
• Run a focus group with your most loyal customers and call it brand insight.
• Collect NPS scores from people who stuck around long enough to answer.
• Test messaging on colleagues, then wonder why it flopped in the wild.

The method becomes the distortion. Your “insight” is just the shape of whoever showed up.

Where you’ll see it

• Social polls that mirror your own network back at you.
• UX research full of power users and early adopters.
• Brand trackers that oversample loyalists.
• Post-campaign surveys that only reach people who saw the ad.
• B2B research run on LinkedIn that mostly measures marketers.

From inside the sample, the story looks consistent. From outside, it’s a cul-de-sac.

How to catch yourself

Before you put a number in a deck, ask:

• Who didn’t we hear from, and why?
• Does this sample actually look like our market?
• Would this result hold if we weighted it to reality?
• Did our method invite a certain kind of person?
• If I renamed this dataset “People Like Us,” would the insight still sound smart?

If the honest answers make the graph less impressive, you’re getting closer to the truth.

Guardrails

1. Define the population first. “UK pet owners who buy food monthly” is a population. “My followers” is a coincidence.
2. Quota to reality, not convenience. Match the proportions that exist in the real world, not the ones that turn up in your inbox.
3. Recruit outside your channels. If everyone came through your owned media, you’ve measured familiarity, not behaviour.
4. Triangulate methods. Combine what people say with what they do. Agreement means confidence; disagreement means learning.
5. Audit non-response. The ones who ignore you are a dataset too. Follow up, change channel, or shorten the ask.
6. Kill the total number. A single average hides the skew. Always show the breakdown next to it.

The napkin test

If the way you found people explains the result better than the question you asked, you’ve got sampling bias, not insight.

Close but not quite

• Survivorship bias: studying only those who made it back. Sampling bias decides who you meet; survivorship bias decides who’s left to study.
• McNamara Fallacy: measuring what’s easy to reach instead of what matters. Sampling bias is its fieldwork cousin.

I still think about that day sometimes. The clipboard’s probably long gone. The drizzle probably hasn’t stopped. But somewhere in Dover, a retired woman and a man with eight pints of courage are still statistically significant.

The data looked convincing then, too. It just forgot to include the people who had somewhere better to be.