Surveys
Date: 2026-08-17
The easiest research method to run and the easiest to run badly. Question wording changes answers more than opinion does, and the people who respond are never the people who didn’t.
A survey collects structured responses from many people at once. Closed questions produce quantitative data; open text produces qualitative data needing coding — Qualitative Coding.
Non-response bias is the big one
The people who answer are systematically different from the people who don’t.
WHO RESPONDS
the delighted
the furious
the bored
people with time
people who like surveys
WHO DOESN'T
the mildly dissatisfied ← the largest
and most
actionable group
the busy
people who've already left
A 4% response rate is not a sample of your customers. It’s a sample of people willing to complete surveys, and that population differs from yours in ways that correlate with exactly what you’re asking about.
This cannot be fixed by asking more people. Doubling the invitations doubles the same biased group — Populations and Samples.
Question wording changes the answer
More than most people believe, and reliably:
LEADING
"How helpful was our new checkout?"
→ assumes helpful
DOUBLE-BARRELLED
"Was the site fast and easy to use?"
→ one answer, two questions
ABSOLUTES
"Do you always check delivery costs?"
→ "always" pushes people to no
LOADED VOCABULARY
"Do you support charging for delivery?"
vs "Do you support free delivery
funded by higher prices?"
→ same policy, opposite answers
VAGUE QUANTIFIERS
"Do you shop online often?"
→ "often" means different things to
everyone. Ask for a number
Ask about behaviour with a timeframe. “How many times in the last month” beats “how often” in every case, because it’s answerable from memory rather than from self-image.
Scales
BINARY yes / no. Loses nuance,
easy to analyse
LIKERT 5 or 7 strongly disagree →
strongly agree
the workhorse
NPS 0–10 one specific question,
heavily criticised
— Net Promoter Score
SEMANTIC cheap ←→ expensive
DIFFERENTIAL
See: Net Promoter Score
Odd-numbered scales allow a neutral midpoint; even-numbered force a side. Neither is correct — forcing a side manufactures opinions from people who don’t have one, and allowing neutrality lets people avoid answering. Choose deliberately, and don’t change it between waves.
Label every point, not just the ends. Unlabelled middle points are interpreted inconsistently.
Where surveys are strong
- Attitudes and self-reported context — things behaviour can’t reveal, like why someone was shopping
- Sizing something already identified qualitatively — the honest use of “how many”
- Tracking a consistent measure over time, where absolute accuracy matters less than the trend
- Reaching people you cannot observe — non-converters, lapsed customers
Where they’re weak
- Predicting behaviour. Stated intention and action diverge routinely — Willingness to Pay
- Anything about a specific interface. Watch instead — Usability Testing
- Recall of detail. People misremember what they did and when
- Explaining a number. An open-text box gets short, unrepresentative answers
Practical rules
- Fewer questions than you want. Completion falls with length, and drop-off is not random — the people who abandon differ from those who finish
- Ask the important question first, while attention is highest
- Pilot with five people, out loud. Ambiguity is invisible to the author
- Randomise answer order where order could bias — people pick early options disproportionately
- Include “don’t know” and “not applicable”, or you force invention
- Never ask what you can measure. “How long did checkout take?” is in your analytics, more accurately — Funnel Analysis
- Report the response rate and the sample size, every time. A finding without them is uninterpretable
On-site surveys specifically
Intercept surveys — the pop-up asking why you’re here — have their own trade: they reach people mid-behaviour, which is valuable, and they interrupt the behaviour, which is a cost.
Trigger on exit intent or after a delay, sample a small percentage, and never on the checkout. A survey that costs conversions to explain conversions is a bad trade, and it’s measurable — Voice of Customer Data.