Getting valid results from surveys: meet the Survey Octopus.
Surveys are a powerful research method, but not easy to get right. The Survey Octopus is a way of thinking through the issues that will ensure that you'll get solid results from your survey that you can use to make decisions. Presentation from the UX in the City conference, Oxford, March 2016
5. Caroline Jarrett @cjforms (CC) BY SA-4.0
Response relies on
effort, reward, and trust
People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram from Jarrett, C, and Gaffney, G (2008)
“Forms that work: Designing web forms for usability”
inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
5
6. Caroline Jarrett @cjforms (CC) BY SA-4.0
I’m a forms specialist
Image credit: Flickr, taxrebate.org.uk
6
7. Caroline Jarrett @cjforms (CC) BY SA-4.0
Why do people answer questions?
Caroline Jarrett @cjforms (CC) BY SA-4.0
7
9. Caroline Jarrett @cjforms
The survey is a
systematic method
for gathering information from
(a sample of) entities
for the purpose of
constructing quantitative descriptors
of the attributes of the larger population
of which the entities are members.
Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor &
Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
10. Caroline Jarrett @cjforms
The survey is a
process
for gathering information from
(a sample of) entities
for the purpose of
constructing quantitative descriptors
of the attributes of the larger population
of which the entities are members.
11. Caroline Jarrett @cjforms
The survey is a
process
for getting answers to questions from
(a sample of) entities
for the purpose of
constructing quantitative descriptors
of the attributes of the larger population
of which the entities are members.
12. Caroline Jarrett @cjforms
The survey is a
process
for getting answers to questions from
(a sample of) people
for the purpose of
constructing quantitative descriptors
of the attributes of the larger population
of which the entities are members.
13. Caroline Jarrett @cjforms
The survey is a
process
for getting answers to questions from
(a sample of) people
for the purpose of
getting numbers
of the attributes of the larger population
of which the entities are members.
14. Caroline Jarrett @cjforms
The survey is a
process
for getting answers to questions from
(a sample of) people
for the purpose of
getting numbers
that you can use to make decisions
15. Caroline Jarrett @cjforms
To make decisions People
The survey is a
process for getting
answers to questions
getting numbers
16. Caroline Jarrett @cjforms (CC) BY SA-4.0
The aim of a survey is to get a number
that helps you to make a decision
16
18. Caroline Jarrett @cjforms (CC) BY SA-4.0
Fieldwork used to be expensive
so a survey was a rare event.
Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/
18
19. Caroline Jarrett @cjforms (CC) BY SA-4.0
1950s mindset: “Ask Everything”
Survey =
Big Honkin’ Survey
19
20. Caroline Jarrett @cjforms (CC) BY SA-4.0
2016 mindset: the Light Touch survey
• Choose ONE question
• Find ONE person
• Ask the question, face-to-face
• See if you can make ONE decision
• Improve, iterate, increase
20
28. Caroline Jarrett @cjforms (CC) BY SA-4.0
Asking the right people is better
than asking lots of people Sample:
the list you
sample from
Caroline Jarrett @cjforms (CC) BY SA-4.0
28
31. Caroline Jarrett @cjforms (CC) BY SA-4.0
Feeling special is rewarding
People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram from Jarrett, C, and Gaffney, G (2008)
“Forms that work: Designing web forms for usability”
inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
31
32. Caroline Jarrett @cjforms (CC) BY SA-4.0
Ask fewer people to get better response
Sample:
the number of
people to ask
Caroline Jarrett @cjforms (CC) BY SA-4.0
32
33. Caroline Jarrett @cjforms (CC) BY SA-4.0
Try mail, phone or face-to-face for
better response rates
Fieldwork:
Who answers?
Caroline Jarrett @cjforms (CC) BY SA-4.0
33
34. Did we get answers from
the right people?
Is this sample representative?
Image credit: Caroline Jarrett / CorelDraw
34
35. Population of assorted birds
Is this sample representative?
Image credit: Caroline Jarrett / CorelDraw
35
36. Caroline Jarrett @cjforms (CC) BY SA-4.0
We don’t just want answers,
we want representative answers
Response
Caroline Jarrett @cjforms (CC) BY SA-4.0
36
40. Caroline Jarrett @cjforms
In your last five days at work, what
percentage of your work time do you
estimate that you spent using publicly-
available online services (not including
email, instant messaging, and search) to
do your work using a work computer or
other device?
%
Caroline Jarrett @cjforms (CC) BY SA-4.0
40
41. Caroline Jarrett @cjforms (CC) BY SA-4.0
Helps a lot if you ask good questions
Questions:
What are you
asking about?
How many
questions?
Caroline Jarrett @cjforms (CC) BY SA-4.0
41
42. Caroline Jarrett @cjforms (CC) BY SA-4.0
Response relies on
effort, reward and trust
People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Diagram from Jarrett, C, and Gaffney, G (2008)
“Forms that work: Designing web forms for usability”
inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
42
43. Caroline Jarrett @cjforms (CC) BY SA-4.0
A good question works in three ways
Obvious Interesting
Appropriate
43
44. Caroline Jarrett @cjforms (CC) BY SA-4.0
Why did you visit our website today?
Obvious Interesting
Appropriate
44
45. Caroline Jarrett @cjforms (CC) BY SA-4.0
Would you recommend us to
a friend or family member?
In a shop,
buying a
baby buggy
In a hospital,
having a
miscarriage
Obvious Yes
Interesting Yes
Appropriate Yes Cruelly
inappropriate
45
46. Caroline Jarrett @cjforms (CC) BY SA-4.0
Would you recommend us to
a friend or family member?
In a shop,
buying a
baby buggy
In a hospital,
having a
miscarriage
Obvious Yes
Interesting Yes
Appropriate Yes Cruelly
inappropriate
46
47. Tip
Test your questions by
interviewing in context
Caroline Jarrett @cjforms (CC) BY SA-4.0
47
48. Caroline Jarrett @cjforms (CC) BY SA-4.0
We don’t want any old answers.
We want useful answers.
Response
Caroline Jarrett @cjforms (CC) BY SA-4.0
48
49. Caroline Jarrett @cjforms
In your last five days at work, what
percentage of your work time do you
estimate that you spent using publicly-
available online services (not including
email, instant messaging, and search) to
do your work using a work computer or
other device?
%
Caroline Jarrett @cjforms (CC) BY SA-4.0
49
50. Caroline Jarrett @cjforms
On your most recent working day,
what percentage of time did you spend
using the Internet?
%
Caroline Jarrett @cjforms (CC) BY SA-4.0
50
51. Caroline Jarrett @cjforms (CC) BY SA-4.0
The approximate curve of forgetting
Major life event
Noticeable, occasional
Unremarkable,
repetitive
Perfect
recall
Day
Gone
Week Month YearHour
51
52. Caroline Jarrett @cjforms (CC) BY SA-4.0
Can’t remember = can’t answer
Response
Caroline Jarrett @cjforms (CC) BY SA-4.0
52
53. Caroline Jarrett @cjforms (CC) BY SA-4.0
Insights are the answers that you use
to make the decisions
Insights Caroline Jarrett @cjforms (CC) BY SA-4.0
53
55. Caroline Jarrett @cjforms
"Phone photography" by Petar Milošević -
Own work. Licensed under CC BY-SA 3.0 via
Commons -
https://commons.wikimedia.org/wiki/File:Pho
ne_photography.jpg#/media/File:Phone_phot
ography.jpg
Modified by Caroline Jarrett
55
56. Tip
Always allow for ‘other’
Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk
56
57. Caroline Jarrett @cjforms
Can we make the decision yet?
Can we make the decision yet?
Can we make the decision yet?
Caroline Jarrett @cjforms (CC) BY SA4.0
57
This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
A process starting with one person face to face, continues through 10 people by phone, gets to 100 people by email or pop-up.
It’s best to check that your question works with one person before you hassle 10 people with it. Then check it works with 10 people before you send it to 100. Once you’ve tried it on 100 people, you might be more interested in a new question than getting more answers on this question
The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
Prank leaves Justin Bieber facing tour of North Korea
By Daniel Emery Technology reporter, BBC News
5 July 2010
Image caption It is highly unlikely Bieber would be given permission to enter North Korea Canadian singer Justin Bieber's has become the target of a viral campaign to send him to North Korea.
A website polled users as to which country he should tour next, with no restrictions on the nations that could be voted on.
There are now almost half a million votes to send the singer to the secretive communist nation.
The contest, which ends at 0600 on 7 July, saw North Korea move from 24th to 1st place in less than two days.
Many of the votes are thought to originate from imageboard website 4chan, which has built a reputation for triggering online viral campaigns.
The octopus, with focus on 'The list you sample from'
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The octopus, with focus on "the ones whose answers you use"
The octopus again. This time we're looking at 'the questions we ask'.
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
Obvious questions require no effort to answer. Interesting questions are rewarding to answer. Appropriate questions are ones that inspire trust in the respondent.
People come to the web with their own questions, so they’re likely to know why they are on your website. It’s a relatively interesting question to answer, and it’s appropriate to ask visitors why they’re visiting.
In your last five days at work, what percentage of your work time do you estimate that you spent using publicly-available online services (not including email, instant messaging, and search) to do your work using a work computer or other device?
Two questions from a survey:
'24: Do you use a Windows or Mac computer'?
'25. What is your gender'?
Photo of a Samsung (android) mobile with the same questions as previous slide. If you only have an Android mobile, how do you answer ‘do you use a Windows or Mac computer’ when the answer options are ‘Windows’, ‘Mac’ and ‘Both’?
A model wears a t-shirt with Gender: 'Male' (crossed out), 'Female' (crossed out) and 'Other' (added and ticked'.
A process starting with one person face to face, continues through 10 people by phone, gets to 100 people by email or pop-up.
It’s best to check that your question works with one person before you hassle 10 people with it. Then check it works with 10 people before you send it to 100. Once you’ve tried it on 100 people, you might be more interested in a new question than getting more answers on this question
The 8 tentacles of the Survey Octopus are:
Left side:
Goals: the resources you have
Questions: the questions you ask
Response: the answers you get
Insights: the answers you use
Right side:
The list you sample from
The sample you ask
The ones who answer
The ones whose answers you use
The 8 tentacles of the Survey Octopus are:
Left side:
Goals: the resources you have
Questions: the questions you ask
Response: the answers you get
Insights: the answers you use
Right side:
The list you sample from
The sample you ask
The ones who answer
The ones whose answers you use