Eye tracking research was conducted on a mobile app called Fleet. The research involved 74 participants completing tasks on the app both with and without eye tracking. Key findings from the eye tracking data included:
- Novice and experienced users focused on different elements of the app interface.
- Larger interface elements were not necessarily noticed more than smaller elements.
- Interface designs that matched users' mental models were easier for users to process.
- Users did not read dense blocks of text or instruction text, focusing only on actionable areas instead.
- Error messages needed to be more helpful in indicating where users failed to complete mandatory fields.
Eye Tracking the UX of Mobile: What You Need to Know
1. Eye tracking the UX of mobile:
What you need to know
Jen Romano Bergstrom
UX Researcher
Facebook
@romanocog
1
Jon Strohl
UX Researcher
Fors Marsh Group
@jonstrohl
March
23,
2015
ModevUX
|
Arlington
VA
2. We will be answering these questions!
2
Ø Why is mobile important?
Ø What makes mobile different?
Ø Do I need to test my design?
• How do I get started?
• What are the benefits?
Ø Do I need to include eye tracking?
• How do I get started?
• What are the benefits?
• Can you show me how it’s done?
• Are there any examples?
3. The rise of mobile
3
@romanocog @jonstrohl
#modevux
7. 7
What we design for…and reality
@romanocog @jonstrohl
#modevux
Krug, S. (2000) Don’t Make Me Think. Pearson Education *NEW 3rd Edition (2014)*
@skrug
8. 8
8
Ø Controlled
environment
Ø All
par>cipants
have
the
same
experience
Ø Record
and
communicate
from
control
room
Ø Observers
watch
from
control
room
and
provide
addi>onal
probes
(via
moderator)
in
real
>me
Ø Incorporate
physiological
measures
(e.g.,
eye
tracking,
EDA)
Ø No
travel
costs
LABORATORY
REMOTE
IN
THE
FIELD
Ø Par>cipants
tend
to
be
more
comfortable
in
their
natural
environments
Ø Recruit
hard-‐to-‐reach
popula>ons
(e.g.,
children,
doctors)
Ø Moderator
travels
to
various
loca>ons
Ø Bring
equipment
(e.g.,
eye
tracker)
Ø Natural
observa>ons
Ø Par>cipants
in
their
natural
environments
(e.g.,
home,
work)
Ø Use
video
chat
(moderated
sessions)
or
online
programs
(unmoderated)
Ø Conduct
many
sessions
quickly
Ø Recruit
par>cipants
in
many
loca>ons
(e.g.,
states,
countries)
> FMG Capabilities
Where to test
12. User Experience (UX) Defined
12
@romanocog @jonstrohl
#modevux
Adapted from Morville, P. Semantic Studios. http://semanticstudios.com/user_experience_design/
13. A combination of measures
are used in UX research.
@romanocog @jonstrohl
#modevux
Using multiple measures allows
researchers to capture the most
detailed and complete picture of
the user experience.
14. 14
OBSERVATIONAL
First click accuracy
Task accuracy
Time on page/task
Selection/click behavior
Conversion rate
SELF-REPORT
Satisfaction ratings
Difficulty ratings
Think-aloud protocol
Debriefing interview
UX Measures
@romanocog @jonstrohl
#modevux
17. …our thumbs are only so big
17
http://gizmodo.com/how-to-design-for-thumbs-in-the-era-of-huge-screens-1636955848
18. …and people use them differently.
18
@romanocog @jonstrohl
#modevux
Hoober (2013). UX Matters. http://www.uxmatters.com/mt/archives/2013/02/how-do-users-really-hold-mobile-devices.php
19. Eye tracking and attention
19
@romanocog @jonstrohl
#modevux
@agabojko
Bojko, Aga (2013). Designing the User Experience. Rosenfeld Media
20. User Experience (UX) Defined
20
Eye tracking
helps to
explain how
usable,
engaging, and
desirable a
product is.
@romanocog @jonstrohl
#modevux
22. User Experience (UX) Measures
• Self-report metrics tell us why participants focus on
certain site aspects.
22
• Eye tracking tells us what, how long, and how often
participants focus on design elements.
• The combination of self-report, observational, and implicit
data allows us to better understand perceptions and
behavior.
• Observational metrics tell us how participants
navigate and interact.
We do not use eye tracking in isolation.
@romanocog @jonstrohl
#modevux
24. User Experience Testing + Eye Tracking
24
UX Testing UX Testing with Eye Tracking
Data while
completing tasks
Mouse movements,
clicks, comments
Scan paths, dwell time, first
fixation, re-fixations. Eye-tracking
helps determine what led to
these behaviors and comments
Findings Usability problems
are detected
A more thorough understanding
of usability issues and the
sources of these issues
Recommendations Quality recommendations
based on available data
More insightful and precise
recommendations
@romanocog @jonstrohl
#modevux
25. Eye-Tracking Metrics ENGAGEMENT
Number of fixations
Total dwell time
Percentage of time on
an area
FINDABILITY
Time to first fixation
Number of fixations
prior to first fixation
PROCESSING
Fixation
durations
WORKLOAD/EXCITEMENT
Pupil dilation
PROCESSING
ORDER
Gaze path
COMPREHENSION
Repeat fixations
25
35. Moderator & Participant
35
Slide from: Alayoubi, L. & Romano Bergstrom, J. (2014). Paying for Education: User-Centered Research on
the Paying for College Website and Tools. Presentation at edUi, Richmond, VA, Sept 2014.
Example Methodology – Formative Evaluation
Eye Tracker
Participants:
• N = 9
• Mix of gender, ethnicity, income.
• Formative evaluation with a single
web prototype.
User Experience Testing session:
• Participants read a description of
the study.
• The moderator gave instructions
and calibrated the eye tracker.
• Participants completed a set of
tasks provided individually by the
moderator.
• End-of-session satisfaction
questionnaire.
• Debriefing interview.
36. 36
Formative Studies
Ø Are <10 participants appropriate?
• Very often. Typical for finding large usability problems.
• Problem occurrence is 25% or higher.
• Problem discovery goal is 75% or less.
Ø Are >20 participants appropriate?
• Not as common, but sometimes used when testing more mature
applications or trying to find smaller, less frequent issues.
• Problem occurrence is 10% or higher.
• Problem discovery goal is 90%.
Ø If eye tracking, over recruit by ~20%
1.Calculations derived from:
Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical statistics for user research. Elsevier.
Bojko, A. (2013). Eye Tracking the User Experience. Rosenfield Media.
Determining Sample Size1
@romanocog @jonstrohl
#modevux
37. UX Test with Eye Tracking Demo
37
@romanocog @jonstrohl
#modevux
FLEET
• Beta
• Flyfleet.org
Features:
• Real-time flight status updates
• Let friends and family track your flight
• Get delay times at check-in and
security
• Find stores and restaurants near you
Fleet was designed and developed by Rob Youmans, Daniel Gartenburg, Melissa Smith, Peter Lee, Jordan Higgins, Octavian, and Vincent Mamo
38. 38
Example Methodology – Summative Evaluation
Participants:
• N = 74 | Average Age = 37
• Mix of gender, ethnicity, income.
• Summative Evaluation with a between
groups design.
• New, Old, Prototype, and Bilingual
media diaries.
User Experience Testing session:
• Participants read a description of the
study.
• The moderator gave instructions and
calibrated the eye tracker.
• Participants completed Steps 1-5 in the
diary at their own pace.
• End-of-session satisfaction questionnaire.
• Debriefing interview.
Eye Tracker
Moderators worked from another room.
Control Room
Slide from: Walton, L., Romano Bergstrom, J., Hawkins, D. & Pierce, C. (2014). User Experience
and Eye-Tracking Study: Paper Diary Design Decisions. Paper presentation at the American
Association for Public Opinion Research (AAPOR) Conference, Anaheim, CA, May 2014.
39. 39
1.Calculations derived from:
Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical statistics for user research. Elsevier.
Bojko, A. (2013). Eye Tracking the User Experience. Rosenfield Media.
Summative Studies
Ø Are <10 participants appropriate?
• Very rarely, but sometimes when trying to detect very large differences.
• Difference between groups is 60% or more.
• Assuming our confidence goal is 90%.
• Assuming our power is 80%.
Ø Are >20 participants appropriate?
• Yes, very often. Typical for detecting moderate differences with
confidence.
• Difference between groups is 30% or more.
• Assuming our confidence goal is 90%.
• Assuming our power is 80%.
Ø If eye tracking, over recruit by ~20%
Determining Sample Size1
@romanocog @jonstrohl
#modevux
40. Example Methodology – Naturalistic environment
40
Participants:
• N = 8
• Mix of gender, income.
• Participants use their own
device and Internet
connectivity.
User Experience Testing session:
• Sessions occurred in participant’s home.
• The moderator gave instructions.
• Sessions began with an interview about
how participants use the Internet.
• Participants completed a set of tasks
provided individually by the moderator.
• Debriefing interview.
42. 42
Fixation count heat map of 10 seconds
of interaction with the page. 20
participants included.*
Fixation count heat map of 10 seconds
of interaction with the page. 10
Novice participants included.*
Fixation count heat map of 10 seconds
of interaction with the page. 9
Experienced participants included.*
All Participants Novice Experienced
*Heatmaps include participants who located the link from the Refunds page only.
Fixation count refers to the number of times the eye fixates, or directs itself towards, an area of the page.
Novice and experienced users
focus on different elements.
44. 44
New/Old DVR Grid New/Old DVR Grid Prototype Grid
Grids that match users’ mental
model are easier to process.
Walton, L., Romano Bergstrom, J., Hawkins, D., & Pierce, C. (2014). User experience testing and eye tracking inform paper diary design.
Proceedings from the Human Computer Interaction International Conference, June 2014, Crete, Greece.
@romanocog @jonstrohl
#modevux
45. 45
Gaze plot of 60 seconds of engagement from four
participants on the Loan Rehabilitation page who scanned
the text without reading it thoroughly.
Gaze plot of 60 seconds of engagement from four
participants on the Contact Your Servicer page who read
most of the text.
People do not read
dense text.
@romanocog @jonstrohl
#modevux
46. 46
The text within the
orange circle indicates
the information to enter
on the page. The red
circled area is where
the user enters the ID
number. The participant
skips the instructions
and introductory text
and moves on to the
actionable areas.
People do not read instructions.
He, J., Siu, C., Chaparro, B. & Strohl, J. (2014). Mobile. In J. Romano Bergstrom & A. Schall (Eds.), Eye Tracking in User Experience Design. San
Francisco, CA: Morgan Kaufmann.
@romanocog @jonstrohl
#modevux
47. 47
If the mandatory fields were
not completed, participants
received an error message
that was not helpful. This gaze
plot begins after the
participant clicks “Ok.” The
participant looks all over the
screen, searching for the
missing mandatory fields. The
error message did not indicate
which fields were missing.
People need to know
what to do next
He, J., Siu, C., Chaparro, B. & Strohl, J. (2014). Mobile. In J. Romano Bergstrom & A. Schall (Eds.), Eye Tracking in User Experience Design. San
Francisco, CA: Morgan Kaufmann.
@romanocog @jonstrohl
#modevux
50. 50
Eye-Tracking Decision Progression
Research goals
and plan
Form hypotheses Will eye tracking
add value?
Design the study
Collect data
Clean & prepare
data
Visualize
qualitative data
Analyze
quantitative data
51. 51
Evidence: Younger and middle-age social media users access the sites/apps largely
through smartphones, whereas older users access largely through desktop
computers. It is important to test across user groups and across devices.
Older AdultsMiddle Age AdultsYounger Adults
People
fixate
elements
of
displays
differently
across
devices.
Different user groups access via
different interfaces
@romanocog @jonstrohl
#modevux
52. 52
Android App HomepageTablet Web HomepageTablet App Homepage
People
fixate
elements
of
displays
differently
across
devices.
People attend to elements of
displays differently across devices
Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and
Web Data Collection Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014.
@romanocog @jonstrohl
#modevux
53. Thank you!
Jen Romano Bergstrom
UX Researcher
Facebook
@romanocog
JenRB@fb.com
Jon Strohl
UX Researcher
Fors Marsh Group
@jonstrohl
jstrohl@forsmarshgroup.com
March
23,
2015
ModevUX
|
Arlington
VA
54. 54
Website
www.forsmarshgroup.com
LinkedIn
hXp://www.linkedin.com/company/fors-‐marsh-‐group
Blog
www.forsmarshgroup.com/index.php/blog
TwiXer
@forsmarshgroup