1. Testing Your Baby
“How do I get Janice and the
luxr mafia off my back?”
Rick Boardman
@uxrick
luxr Jun/17/2011

2. Cast
Product/Biz UX
User
Eng
Thanks to Braden Kowitz for smiley graphics

3. Think Usability?
Intro to Intro to
(Speaker/ How to Lean UX Task
generative Personas Intersect What are methods? Analysis?
Lecture) interviewing Distinguish
& POVs
Brainstorm
you building
btw needs w/ Biz & Analytics
Distro in next week
& insights Agile?
& next 3 mo?
User Customer/ Solution
Needs Uses Features Prototype Story/ Test
Empathy Problem Ideation Release
How do we test our Definition
Make hypotheses? Make persona
(Method/ Practice with of target user(s) Framing Wireframes
Mock
Activity) partner Brainstorm dashboard
POV madlib
Practice Ecosystem & Storyboard
Sketches
synthesizing Distribution Flows
empathy map map How to test
conversion
Refine survey Compare personas Prioritize How to events?
questions created by diff based on tech/ test? Test with real
team members biz/distro users &
Check Product
Design Guild
What are How to
(Method/ riskiest How to
test?
Activity) Show partner test?
hypotheses?
map
Interview
real users?
Thurs Fri Sat

4. http://www.all-tribes.com/eng/catalog/rtfm-vol2-girls-mastic-p-51.html

5. How do I do Unix?
http://www.all-tribes.com/eng/catalog/rtfm-vol2-girls-mastic-p-51.html

6. RTFM!
How do I do Unix?
Read The F****** Manual
http://www.all-tribes.com/eng/catalog/rtfm-vol2-girls-mastic-p-51.html

7. Pic: http://www.flickr.com/photos/victoriapeckham/164175205

8. How do we
test our
hypotheses?
Pic: http://www.flickr.com/photos/victoriapeckham/164175205

9. How do we LTYFU!
test our
hypotheses?
Listen To Your F****** Users
Pic: http://www.flickr.com/photos/victoriapeckham/164175205

10. Head in sand Guess
Ask Janice
Ask Jason
Which method?
Demo at “Launch and cognitive walkthrough
SXSW see”
Analytics
expert review Usability testing Friends n’ Family
A/B testing
Dogfooding Funnel analysis
remote testing Contextual interviews
Beta
click/5sec tests
Focus groups Surveys Show to
David Pogue

11. Head in sand Guess
Ask Dave
Ask Enrique
Which method?
Demo at
SXSW
H ere’s a dirty
“Launch and cognitive walkthrough
see”
secUsability
expert review
...
ret .testing Friends n’ Family
Analytics
A/B testing
s shhhh Funnel analysis
sssDogfooding
remote testing Contextual interviews
Beta
click/5sec tests
Focus groups Surveys Show to
David Pogue

12. Head in sand Guess
Ask Dave
Ask Enrique
Which method?
Demo at
H ere’s a dirty “Launch and cognitive walkthrough
TheeUsability.testingsFriends kFamily
yeta.l.l. ucn’ *
SXSW see”
Analytics
s cr
expert review
A/B testing
s shhhh Funnel analysis
sssDogfooding
remote testing Contextual interviews
Beta
click/5sec tests
Focus groups Surveys Show to
David Pogue
*More diplomatically: they have strengths and weaknesses

13. Head in sand Guess
Ask Dave
Ask Enrique
Which method?
Demo at
H ere’s a dirty “Launch and cognitive walkthrough
TheeUsability.testingsFriends kFamily
yeta.l.l. ucn’ *
SXSW see”
Analytics
s cr
expert review
A/B testing
s shhhh Funnel analysis
sssDogfooding
remote testing Contextual interviews
Beta
Focus groups Surveys Especiallytests one
click/5sec this
Show to
David Pogue
*More diplomatically: they have strengths and weaknesses

14. WTF? ... thats not very
helpful ...

15. Sol ution:
Trian gulate
Mult iple M ethods
Analytics
Usability
Offer Janice beer

16. A nalytic s suck
*Analytics are useful but have strengths and weaknesses

17. Why Analytics Suck
1. Tell you <what> not <why>
2. Hard to isolate specific effects
3. Which numbers do I pay
attention to?

18. Oooo
AAARRRR, got a
dashboard?
Link: Metrics
for Pirates

19. Example Conversion Dashboard
(note: *not* actuals… your mileage may vary)
Stage Conversion Status Conv. Est. Value
% (*not* cost)
Acquisition Visitors -> Site/Widget/Landing Page 60% $.05
(2+ pages, 10+ sec, 1+ clicks = don’t abandon)
Activation “Happy” 1st Visit; Usage/Signup 15% $.25
(clicks/time/pages, email/profile reg, feature usage)
Retention Users Come Back; Multiple Visits 5% $1
(1-3x visits/mo; email/feed open rate / CTR)
Referral Users Refer Others 1% $5
(cust sat >=8; viral K factor > 1; )
Revenue Users Pay / Generate $$$ 2% $50
(first txn, break-even, target profitability)
Link: Metrics for Pirates

20. Re mote ( autom ated)
Usa bility S ucks*
*Remote usability is useful but has strengths and weaknesses

21. Why Remote
(Automated)
Usability Sucks
1. Participant quality
2. Hard to dig deep
3. Need high-fidelity prototype
4. Don’t get to meet fun people
(i.e. miss out on useful context)

22. Why Remote
(Automated)
Usability Sucks
1. Participant quality
Useful tool but
2. Hard to dig deep
dangerous too
3. Need high-fidelity prototype
4. Don’t get to meet fun people
(i.e. miss out on useful context)

23. Live U sability
S ucks*
*Live usability is useful but has strengths and weaknesses

24. Why Live Usability
Sucks
1. Pricey? ($$$ and time)
2. Hard to cover large sample
3. You have to talk to people

25. Usability testing
• Watch your product in action
with a target user
• Follow some tricks to make as
objective as possible
erview bonus : opportunity
Live int
for m ore generative research!

26. Usability Howto
1. Identify research goal -> Scenario
2. Recruit
3. Prepare interview outline, pilot test!
4. Test!
- Intro (welcome/confidential/reassure/thinkaloud)
- Record / take notes
- Debrief
5. Review, GET CODING, and iterate

27. Golden Rule of
Usability

28. Golden Rules of
Usability
Participant should do 90% of the talking
Shut up

29. Golden Rules of
Usability
Participant should do 90% of the talking
An Sh
d ITE RuTEup
At !

30. Things to look for
• Obvious usability issues
• Bugs
• Opportunities
• Don’t forget Delights!

31. Lean UX questions
1. What gear do I need?
2. How many users?
3.Which users?

32. observation
test room room
1-way mirror
eyetracking
UI
usb mic
Eng
Moderator User
PM
32

33. Observer /
notetaker /
prototype
debugger /
Moderator User coffee maker
(optional but nice)
33

34. Lean UX questions
1. What gear do I need?
2. How many users?
3.Which users?

35. Lean UX questions
1. What gear do I need?
2. How many users? >=0*
3.Which users?
* 2: Ideally, small n and iterate. If no one available, do a
walkthrough!

36. Lean UX questions
1. What gear do I need?
2. How many users? >=0*
3.Which users? Any*
* 2: Ideally, small n and iterate. If no one available, do a
walkthrough! Focus on core interaction flow.
* 3: Start with your neighbour or Janice, then focus.

37. Awesome
2 hour read
http://www.sensible.com/rocketsurgery/index.html

38. Interview Practice

39. Interview Practice
I’m super
nervous ... am I
being tested?

40. Interview Practice
What do I do
next?
Should I sign up?
Is this OK?

41. Interview Practice
I HATE IT.
THIS IS &^**(}ED
UP, MAN.

42. Interview Practice
I LOVE THIS. ITS THE
BEST THING EVER

43. Interview Practice
<click>
<click> <done>
<silence>

44. Interview Practice
Well it looks like you can
use this menu to browse and
you can use that button to
signup
<i.e. here’s a guided tour>

45. Interview Practice
Weather wafflewaffle
waffle Superbowl waffle
waffle waffle waffle waffle
waffle waffle waffle waffle
waffle waffle waffle
Superbowl ...

46. Interview Practice
?

47. Interview Practice
Let me show you the
product! What we’re
trying to do here ...

48. Interview Practice
This is my baby. What
do you love about it?

49. Interview Practice
This is my baby. What
annoys you about it?

50. Interview Practice
Weather wafflewaffle
waffle Superbowl waffle
waffle waffle waffle waffle
waffle waffle waffle waffle
waffle waffle waffle
Superbowl ...

51. Interview Practice
Lets do a focus group

52. Interview Practice
?

53. GO FORTH & LTYFU!
Q & A!