Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
No IM Please, We’re Testing Richard Boardman CHI 2006, Montreal, April 24th 2006
Instant Messaging in Usability Studies <ul><li>Personal Experiences </li></ul><ul><ul><ul><li>(including a few stories fro...
Usability Lab Setup Interview Room Observation Room  1-way mirror
Usability Lab Setup Interview Room Observation Room Moderator User  1-way mirror
Usability Lab Setup Interview Room Observation Room Moderator User  1-way mirror Eng PM UI
Usability Lab Setup  + Instant Messaging Eng PM Moderator User UI  1-way mirror Interview Room Observation Room
Personal Experiences with IM in Usability
The Usability Nightmare - **BUGS** UI Eng PM Moderator User ??? Oh no - I broke their  computer. I hope  they still pay me...
Complex Multi-step Coordination UI Eng PM move on yet? keep her talking, We’re updating the database This is boring,  I wa...
Questions from Team UI Eng PM Moderator User Any more questions on this page? does she like the colours? What should I spe...
The Chatterbox PM UI Eng PM Moderator User pls turn off the pm’s laptop VIA TESTED IM:   Baseball … blah blah … VIA TESTED...
Survey
What do others think of this technique? <ul><li>Email survey </li></ul><ul><ul><ul><li>11 moderators </li></ul></ul></ul><...
Negative Feedback - IM as a distraction
“ It distracts the moderator” UI Eng PM Moderator User Blah blah blah blah … Uh oh, what did the user do just then? **Amaz...
“ It distracts the observers” UI Eng PM Moderator User What does “http” mean? 101101 … **Amazing revelation** Wow - I hope...
Moderators: “Danger of back seat drivers …” UI Eng PM Moderator User Yeah right … I’m in charge here! Move onto page2. Pro...
Moderators: “It distracts the user” UI Eng PM Moderator User Is he doing his email? blah … <html> …
Moderators: “Observers may not turn up” Moderator User This team really buys into Usability! Its great to have lots of obs...
Positive Feedback - IM as a Tool
“ Handy for when the UI goes belly up” UI Eng PM Moderator User Sorry, I just rebooted the server Grrr … Grrrr … Grrr … I’...
“ Handy for extra questions” UI Eng PM Moderator User any more q’s b4 we  wrap up? Aggh, when is this going to finish??
Moderators: “Useful for training” Backup moderator Eng PM Moderator User Try saying this …  << smooth diplomatic phrase>> ...
Observers: “So I can call in from the pool” Moderator User Its nice of UI to check in from the NY office … how’s it going?...
Guidelines
Guideline: 1-way IM only UI Eng PM Moderator User Why won’t he reply??
Guideline: One to one IM UI Eng PM Moderator User pls ask about  feature foo ask others for extra questions
Create a special, secure IM account UI Eng PM Moderator User **Hello Richard This is your mother** Why is he blushing?
End of Session only UI Eng PM Moderator User Oops, we used the old version, can we start again? any more q’s b4 we  wrap up?
Don’t distract user UI Eng PM Moderator User That’s a nice chat alert! I wonder if I can get it as a ringtone for my phone?
Conclusions <ul><li>Effective Usage </li></ul><ul><ul><ul><li>Follow guidelines </li></ul></ul></ul><ul><ul><ul><li>Buggy/...
CHI 2006, Montreal, April 24th 2006 No IM Please, We’re Testing Rick Boardman [email_address] PS: We’re recruiting Thank y...
Go back
Go back (really)
You’ve finished this document.
Upcoming SlideShare
Inntrykk fra ux week 2010 Eli Toftøy-Andersen slide share
Next
Upcoming SlideShare
Inntrykk fra ux week 2010 Eli Toftøy-Andersen slide share
Next

Share

No IM Please, Were Testing

This paper discusses the use of instant messaging (IM) as a communication tool during usability studies � primarily between the interview and observation rooms. The benefits and challenges associated with providing an IM link are discussed, based on feedback from a survey of study moderators and observers. Observers were much more positive about the use of IM than the moderators. A key concern to moderators was the potential distraction to themselves, participants and observers. In contrast, observers greatly welcomed the opportunity to ask questions and help deal with buggy prototypes. Guidelines are outlined for the effective use of IM within a usability context, and contexts outlined when an IM link is most appropriate.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

No IM Please, Were Testing

  1. 1. No IM Please, We’re Testing Richard Boardman CHI 2006, Montreal, April 24th 2006
  2. 2. Instant Messaging in Usability Studies <ul><li>Personal Experiences </li></ul><ul><ul><ul><li>(including a few stories from the Google labs) </li></ul></ul></ul><ul><li>Survey </li></ul><ul><ul><ul><li>Problems </li></ul></ul></ul><ul><ul><ul><li>Opportunities </li></ul></ul></ul><ul><li>Guidelines </li></ul>
  3. 3. Usability Lab Setup Interview Room Observation Room  1-way mirror
  4. 4. Usability Lab Setup Interview Room Observation Room Moderator User  1-way mirror
  5. 5. Usability Lab Setup Interview Room Observation Room Moderator User  1-way mirror Eng PM UI
  6. 6. Usability Lab Setup + Instant Messaging Eng PM Moderator User UI  1-way mirror Interview Room Observation Room
  7. 7. Personal Experiences with IM in Usability
  8. 8. The Usability Nightmare - **BUGS** UI Eng PM Moderator User ??? Oh no - I broke their computer. I hope they still pay me sorry, I just rebooted the server
  9. 9. Complex Multi-step Coordination UI Eng PM move on yet? keep her talking, We’re updating the database This is boring, I want my $$$ Moderator User
  10. 10. Questions from Team UI Eng PM Moderator User Any more questions on this page? does she like the colours? What should I spend my $$$ on?
  11. 11. The Chatterbox PM UI Eng PM Moderator User pls turn off the pm’s laptop VIA TESTED IM: Baseball … blah blah … VIA TESTED IM: Nice to meet another Giants fan … blah blah …
  12. 12. Survey
  13. 13. What do others think of this technique? <ul><li>Email survey </li></ul><ul><ul><ul><li>11 moderators </li></ul></ul></ul><ul><ul><ul><li>15 observers (designers, product managers, engineers, tech writers, …) </li></ul></ul></ul><ul><li>Is IM a useful tool in usability? (1  5) </li></ul><ul><ul><ul><li>Moderators: 2.3 </li></ul></ul></ul><ul><ul><ul><li>Observers: 3.9 </li></ul></ul></ul>
  14. 14. Negative Feedback - IM as a distraction
  15. 15. “ It distracts the moderator” UI Eng PM Moderator User Blah blah blah blah … Uh oh, what did the user do just then? **Amazing revelation**
  16. 16. “ It distracts the observers” UI Eng PM Moderator User What does “http” mean? 101101 … **Amazing revelation** Wow - I hope they fix that ! <html> …
  17. 17. Moderators: “Danger of back seat drivers …” UI Eng PM Moderator User Yeah right … I’m in charge here! Move onto page2. Prompt them to do Y. Ask if they love the tool
  18. 18. Moderators: “It distracts the user” UI Eng PM Moderator User Is he doing his email? blah … <html> …
  19. 19. Moderators: “Observers may not turn up” Moderator User This team really buys into Usability! Its great to have lots of observers its sooooo sunny in santa cruz. hows the test goin?
  20. 20. Positive Feedback - IM as a Tool
  21. 21. “ Handy for when the UI goes belly up” UI Eng PM Moderator User Sorry, I just rebooted the server Grrr … Grrrr … Grrr … I’m sorry, Did I break Your computer?
  22. 22. “ Handy for extra questions” UI Eng PM Moderator User any more q’s b4 we wrap up? Aggh, when is this going to finish??
  23. 23. Moderators: “Useful for training” Backup moderator Eng PM Moderator User Try saying this … << smooth diplomatic phrase>> Help!!! Rant! Rave! Gibber! $%&*! This makes me SO ANGRY!!!! Someone better call security
  24. 24. Observers: “So I can call in from the pool” Moderator User Its nice of UI to check in from the NY office … how’s it going? NY is really cold
  25. 25. Guidelines
  26. 26. Guideline: 1-way IM only UI Eng PM Moderator User Why won’t he reply??
  27. 27. Guideline: One to one IM UI Eng PM Moderator User pls ask about feature foo ask others for extra questions
  28. 28. Create a special, secure IM account UI Eng PM Moderator User **Hello Richard This is your mother** Why is he blushing?
  29. 29. End of Session only UI Eng PM Moderator User Oops, we used the old version, can we start again? any more q’s b4 we wrap up?
  30. 30. Don’t distract user UI Eng PM Moderator User That’s a nice chat alert! I wonder if I can get it as a ringtone for my phone?
  31. 31. Conclusions <ul><li>Effective Usage </li></ul><ul><ul><ul><li>Follow guidelines </li></ul></ul></ul><ul><ul><ul><li>Buggy/experimental UI </li></ul></ul></ul><ul><ul><ul><li>Training new moderators </li></ul></ul></ul><ul><ul><ul><li>Remote observers </li></ul></ul></ul><ul><li>Towards Agile Usability </li></ul><ul><ul><ul><li>Engage observers  collaboration </li></ul></ul></ul><ul><ul><ul><li>Exploratory studies  Promote rapid iteration </li></ul></ul></ul>
  32. 32. CHI 2006, Montreal, April 24th 2006 No IM Please, We’re Testing Rick Boardman [email_address] PS: We’re recruiting Thank you to survey respondents!
  33. 33. Go back
  34. 34. Go back (really)
  • gregorysmith

    Dec. 20, 2009
  • dshaw

    Feb. 24, 2009
  • christianikolai

    Sep. 2, 2007
  • Noots

    Aug. 29, 2007
  • PaRaP

    Aug. 21, 2007
  • jessenfelix

    Mar. 23, 2007
  • lethia

    Dec. 3, 2006
  • jboutelle

    Nov. 2, 2006

This paper discusses the use of instant messaging (IM) as a communication tool during usability studies � primarily between the interview and observation rooms. The benefits and challenges associated with providing an IM link are discussed, based on feedback from a survey of study moderators and observers. Observers were much more positive about the use of IM than the moderators. A key concern to moderators was the potential distraction to themselves, participants and observers. In contrast, observers greatly welcomed the opportunity to ask questions and help deal with buggy prototypes. Guidelines are outlined for the effective use of IM within a usability context, and contexts outlined when an IM link is most appropriate.

Views

Total views

4,660

On Slideshare

0

From embeds

0

Number of embeds

111

Actions

Downloads

2

Shares

0

Comments

0

Likes

8

×