How to gather data to made design decisions on almost anywhere almost any time for very little money.

1. 1
The quick, the cheap,
and the insightful
Usability testing in the wild
Dana Chisnell
San Francisco STC
November 2008

2. 2
What is a usability test?

3. 3
http://www.sigchi.org/chi97/proceedings/overview/tst.htm

4. 4

5. 5
Quick, cheap, insightful
✤ Minimal steps to focus on
✤ Where the value comes from in doing usability tests
✤ Where it may be risky to go minimalist
✤ How to think about the trade-offs
✤ What’s essential
✤ What might be nice to have

6. 6
Minimal steps

7. 7
Classic, with everything
✤ Chapter 5. Develop a test plan
✤ Chapter 6. Choose a testing environment
✤ Chapter 7. Find and select participants
✤ Chapter 8. Prepare test materials
✤ Chapter 9. Conduct the sessions
✤ Chapter 10. Debrief with participants and observers
✤ Chapter 11. Analyze data and observations
✤ Chapter 12. Create findings and recommendations

8. 8
Photo courtesy of Tom Tullis

9. 9
Classic, with everything
Can take a lot of time: weeks or months.

10. 10
What if you have to do something
quick? and cheap?

11. 11
You don’t have to do it
by the book.

12. 12
What’s essential?
Develop a test plan
Choose a testing environment
Find and select participants
Prepare test materials
Conduct the sessions
Debrief with participants and observers
Analyze data and observations
Create findings and recommendations

13. 13
What’s essential?
Develop a test plan
Choose a testing environment
Find and select participants
Prepare test materials
Conduct the sessions
Debrief with participants and observers
Analyze data and observations
Create findings and recommendations

14. 14
What’s essential?
Develop a test plan
Find participants
Conduct the sessions
Debrief with observers

15. 15
What’s essential?
Develop a test plan
Find participants
Conduct the sessions
Debrief with observers and decide together

16. 16
Sit next to someone.
Watch them do stuff.

17. 17
Story:

18. 18
California.

19. 19
1 day

20. 20
a ballot and instructions

21. 21
a ballot and instructions

22. 22
random participants

23. 23
no helpers

24. 24
results due: today

25. 25
What to do?

26. 26
Improvise: Ground rules
✤ Say “yes.”
✤ Trust your team and your participants
✤ Don’t let the team or stakeholders block
✤ Work to the top of your collective intelligence

27. 27
How we got value
✤ Used the ballots and instructions at hand
✤ Focused on one thing at a time
✤ Had participants generate and collect data
✤ Drafted observers and debriefed

28. 28
More examples

29. 29
Cafe

30. 30
Company
lobby

31. 31
Trade
show
http://www.flickr.com/photos/dans180/72408664/

32. 32
Listen in
on
customer
service

33. 33
Transit
seat
mates

34. 34
Remote
testing

35. 35
Browser history
Gives a recent snapshot of activity
Bookmarks & favorites
Shows deliberate returns to sites
Email trails
Builds a story
Shows organizing strategies

36. 36
Customer service logs
View on persistent, difficult problems
Sales reps
Proxies for buyers, end users
User forums & wikis
Frames the conversations users are having

37. 37
Value

38. 38
Each phase includes input
from users
Multidisciplinary teams
Enlightened management
Willingness to learn as you go
Defined usability goals and
objectives
Supporting usable design

39. 39
Each phase includes input
from users
Multidisciplinary teams
Enlightened management
Willingness to learn as you go
Defined usability goals and
objectives
Supporting usable design

40. 40
Where’s the value in testing?
70% watching someone use
the design
20% working with the team
to prepare to test
8% discussing what
happened

41. 41
Getting value in the wild
✤ Use what’s at hand
✤ Narrow the scope of the test
✤ Focus on
✤ what can make the most difference to the
most users
✤ what can be implemented easily with the
resources available

42. 42
✤ If you need
✤ summative
data
When is testing in the ✤ benchmarks
✤ answers to
wild not valuable? hard
problems

43. 43
Risks

44. 44
Risks of testing in the wild
✤ Participant sample may be too small,
biased
✤ Inconsistent approach may net
inconsistent data

45. 45
What do you lose?
✤ Quantitative data
✤ Rigor
✤ Relatively unbiased sample, maybe
http://drb.lifestreamcenter.net/Lessons/process_maps/

46. 46
Trade offs

47. 47
What do you need?
✤ To inform a design:
Testing in the wild Classic usability testing
Qualitative data Quantitative data
Opportunity Planning
Fits the schedule Don’t know how to fit UT into
the schedule
Just in time After the fact
$ $$$$
Something Maybe nothing

48. 48
Where’s the ROI?
✤ Qualitative data
✤ Opportunity
✤ Fitting into a schedule
✤ Timeliness

49. 49
So far.
You don’t have to do it Value of usability Tradeoff: having
by the book testing some data over
having no data
+ observing users
+ working with the
team

50. 50
Coming up.
What’s the bare Steps for testing in What to add if you
minimum the wild have resources

51. 51
Essentials

52. 52
What you need
Somebody
(Human)
✤ Someone who will try the design
Something Some place
✤ Somewhere to test (Activity) (Context)
✤ Something to study

53. 53
1. Plan, minimally
2. Get the team on
board
3. Design the test,
minimally
4. Recruit
participants
5. Conduct
sessions
Follow these steps: 6. Debrief and
decide

54. 54
Plan, minimally
✤ What
✤ Why
✤ Who
✤ When
✤ Where

55. 55
Plan, minimally: Example
✤ What: near-final design
✤ Why: inform user training and support
✤ Who: inexperienced customers
✤ When: the end of the week
✤ Where: trade show at user group meeting

56. 56
Minimalist plan
Find out whether information about admission is
Goals and objectives
easy to find and use
Participant
People who have college-bound kids
characteristics
Description of Sit-by parents attending a high school basketball game, each
method trying three university sites with the same scavenger hunt task
Find out how and when applications are due
List of tasks Determine whether there are fees for applying
Learn when acceptances will be sent

57. 57
Get the team
on board
✤ Visualize the desired user
experience
✤ Share the intellectual property of
observing users

58. 58
Design the
test, minimally
✤ Why are you testing?
✤ What questions are you trying to
answer?
✤ What constrains the design?
✤ What are you going to do with
what you find out?

59. 59
Recruit
participants
✤ Convenience sample
✤ Requirements, not
demographics

60. 60
✤ Staff not on the
design project
✤ Friends and family
✤ Personal,
professional
networks
✤ Online social
networks
✤ Community
organizations
✤ Online classifieds
✤ Association,
society, user
group, union
Sources of participants contacts
✤ Temp agencies

61. 61
Conduct
sessions
✤ Rehearse (P1)
✤ Interview-based tasks (or based
on previous field work)
✤ Explore in short, focused
sessions
✤ Iterate test design and product
design

62. 62
Greet the
participant
Explain the study,
your role, and their
role
Interview (maybe)
Do tasks from
interview
Debrief with
participant
Session outline Debrief with
observers

63. 63
✤ Impartial,
unbiased
observing
✤ No teaching!
✤ Listen and
watch
✤ Ask open-
ended
questions:
Moderating, not training Why? How?
What?

64. 64
Debrief and
decide
✤ Write up issues on sticky notes
and sort them into priority lists
✤ Ask for top 10 items -- base on
data and observations, not
opinion
✤ Take a vote for priorities

65. 65
Nice to have

66. 66
✤ Screened,
scheduled
participants
✤ Official
paperwork
Add these ingredients
✤ Recordings
✤ More observers

67. 67
Participants
What Why
appropriate experience greater confidence in data
scheduled ahead of time easier to get data

68. 68
Paperwork
What Why
script ensure consistent instructions and moderating
consent forms official acknowledgment for taking part
recording waivers permission for recordings
pre-test questionnaire experience, knowledge, value
post-test questionnaire feedback on tasks, UI

69. 69
Recordings
What Why
audio transcripts, verbal protocol analysis
video or photos stories, double-check, highlights
Morae, Camtasia, other digital data, automated collection

70. 70
More observers
✤ Better-informed design recommendations
What Why
developers technology boundaries
management, execs business priorities

71. 71
Nutshell
http://www.flickr.com/photos/stoan/145497333/

72. 72
✤ Value:
observing,
planning
✤ Plan: 4 steps
✤ Recruit:
behavior
✤ Moderating:
Take away: listen, observe

73. 73
Quick. Cheap. Insightful.
You can compensate Low risk, little money Value comes from
for some getting as close as
shortcomings, and just possible to what real
test more people are doing
with your design

74. 74
Where to learn more
Dana’s blog: http://
usabilitytestinghowto.blogspot.com/
Download templates, examples, and
links to other resources from
www.wiley.com/go/usabilitytesting

75. 75
Me
Dana Chisnell
dana@usabilityworks.net
www.usabilityworks.net
415.519.1148