On Tuesday, December 13th, Bill Albert, co-author of Beyond the Usability Lab, and one of the most knowledgable experts in Remote Usability Testing, presented an update on Remote User Testing methods and tools to optimize your UX Roadmap. If you were not able to attend the live webinar please feel free to view the slideshare below!
The webinar was co-hosted with 3 special guests:
Dave Garr, Co-Founder, UserTesting.com
Andrew Mayfield, CEO, Optimal Workshop
Matt Paulus, Director of US Sales, UserZoom
1. Bill Albert, Director of Design & Usability
Center at Bentley University presents a
complimentary webinar on Remote Testing
Methods and Tools.
With special guests:
Webinar:
Remote Testing Methods & Tools
December 13th, 2011
2. Webinar Overview
1. Brief introductions & goal of the webinar
2. Updated overview of competitive landscape, updated figures, etc
3. The different methods / categories / tools. When to use what?
4. Types of research goals
1. Categories of tools available
2. Overview of some of those tools
5. Examples:
– Usertesting.com
– Optimal Workshop
– UserZoom
6. Takeaways, Q&A
2
3. Introduction
• Speakers today:
o Bill Albert, Director of Design & Usability Center, Bentley University
o Dave Garr, Co-Founder, UserTesting.com
o Andrew Mayfield, CEO, Optimal Workshop
o Matt Paulus, Director of US Sales, UserZoom
• Goals of today’s session
3
4. Remote Testing Methods and Tools
Bill Albert
December 13, 2011
walbert@bentley.edu
781.891.2500 | www.bentley.edu/usability
9. Defining Remote Testing Tools
What do we mean by “remote testing tools”?
• A remote testing tool is any technology that allows a researcher to collect
data from users about their experience in using an interface, without
direct, face-to-face contact.
There are two ways of organizing remote testing tools
• Moderated vs. Unmoderated
• Qualitative vs. Quantitative-based
Moderated Unmoderated
Qualitative-based Qualitative-based
Quantitative-based Quantitative-based
9
10. Unmoderated Testing Tools
Full-service
Self-service
Card
Quantitative- Sorting/IA
based
Unmoderated Surveys
tools
Click/Mouse
Video tools
Qualitative-
based Reporting
Expert
Reviews
10
12. Why Should You Care?
• Saves time
o Lab study takes 2-4 weeks from start to finish, unmoderated typically takes
hours to a few days*
• Saves money
o Participants compensation typically a lot less ($10 vs. $100)
o Tools are becoming very inexpensive
• Reliable metrics
o Only (reasonable) way to collect UX data from large sample sizes
• Geography is not a limitation
o Collect feedback from customers all over the world
• Greater Customer insight
o Richest dataset about the customer experience
12
13. Why is this important NOW?
• Questions from senior management are becoming more
complex and time sensitive
• Traditional usability is no longer enough
• Push to measure the UX
• Convergence with market research and web analytics to paint a
more complete picture of the UX
• Budgets/timeline constraints
13
14. Which one goes first?
Lab first, then Unmoderated Unmoderated first, then Lab
Identify/fix “low hanging fruit”, then Identify the most significant issues
focus on remaining tasks with large online through metrics, then use lab
sample size study to gather deeper qualitative
understanding of those issues
Generate new concepts, ideas, Collect video clips or more quotes of
questions through lab testing, then users to help bring metrics to life
test/validate online
Validate attitudes/preferences Gather all the metrics to validate
observed in lab testing design – if it tests well, then no
need to bring users into the lab
14
16. Unmoderated Testing Tools - Quantitative
Full-service
Self-service
Card
Quantitative- Sorting/IA
based
Unmoderated Surveys
tools
Click/Mouse
Video tools
Qualitative-
based Reporting
Expert
Reviews
16
17. Overview
Common Research Questions:
• What are the usability issues, and how big?
• Which design is better, and by how much?
• How do customer segments differ?
• What are user design preferences?
• Is the new design better than the old design?
• Where are users most likely to abandon a transaction?
Types of Studies: Typical Metrics:
• Comprehensive evaluation • Task success
• UX benchmark • Task time
• Competitive evaluation • Self-report ratings such as ease of
• Live site vs. prototype comparison use, confidence, satisfaction
• Feature/function test • SUS
• Discovery • Click paths
• Abandonment
17
18. Strengths / Limitations / Myths
Strengths:
• Comparing products
• Measuring user experience
• Finding the right participants
• Focusing on design improvements
• Insight into the user’s real
experience
Limitations: Myths:
• Not well suited to rapid, iterative • Only test with websites
design • Very expensive
• Need a deep understanding of • Only gather quantitative data
issues • A lot of noise in the data
• Studies that require long sessions • Same as any market research study
• Lose control over prototypes • Comparing products
• Internet access
18
25. Unmoderated Testing Tools - Qualitative
Full-service
Self-service
Card
Quantitative- Sorting/IA
based
Unmoderated Surveys
tools
Click/Mouse
Video tools
Qualitative-
based Reporting
Expert
Reviews
25
26. Overview
Common Research Questions:
• What are the big pain usability pain points, and
wins?
• How do users think, and feel about the design?
• Are we on the right (design) track?
• What is the overall experience like?
• Why might users abandon a transaction?
Types of Studies: Common Metrics:
• Low fidelity prototype • Frequency of issues
• High fidelity prototype • Verbatim comments
• Competitive evaluation • Task success (via video)
• Comprehensive evaluation • Task time (via video)
• Feature/function test
• Discovery
26
43. “I’d like to look at these three star reviews, … it doesn’t let me
do that… I’d have to go through all of the reviews.”
What do you want to test?
43
68. Key Takeaways
• Remote testing tools are becoming more advanced every day, and are a
critical part of the UX toolkit
• Find the right type of tool based on your research question
• Demo a select number of remote testing tools to find the one that will
deliver the results you need, within budget and timeframe
68
70. Thank You!
Bill Albert Dave Garr Andrew Mayfield Matt Paulus
Director of Design Co-Founder CEO Director of US
& Usability Center UserTesting.com Optimal Sales
Bentley University Workshop UserZoom
Floor open to Q&A
Notes de l'éditeur
Introduce myself and DUC Feel free to ask questions Motivation – sort through powerful tools, trying to find the right ones for the right job
Where it fits in Highly adaptive based on the needs This gives it POWER – it creates the missing link
Saves time – Very fast, thousands on panels, Money – essence of quick and dirty. Techniques for dealing with noise, unrealistic to be in the lab that long. Combines both qual/quant and attitudes and behavior.
All the flexibility you need to set up a study and analyzing the dataSignificant support in designing study and analysis. Pricing is all project based – typically very expensive – good choice for a large benchmark study
- These tools are self-service, with some amount of supportCan do largely the same as RV and Keynote, and becoming more powerful by the day Nice visualizations now Loop 11 is very basic, and UZ is much more flexible fraction of the cost = several hundred to a few thousand.
- Sort into groups
You get a video (typically about 15 minutes)This is a test of Walmart.com and her task is to find an ink cartridge for her printerShe doesn’t know what’s going on, and she doesn’t know how to fix it.
She spends more than four minutes and can’t figure out which cartridge works with her (very popular) HP deskjet 1000 printer.
She spends more than four minutes and can’t figure out which cartridge works with her (very popular) HP deskjet 1000 printer.
Download a clip and just drag it into Powerpoiont.
Download a clip and just drag it into Powerpoiont.
Download a clip and just drag it into Powerpoiont.