More Related Content Similar to World Usability Day - Measuring Engagement Similar to World Usability Day - Measuring Engagement (20) 2. Agenda
• What We are Researching
• Technology and Methodology
• Qualitative Findings
• Quantitative Analysis
• What Worked Well
• Challenges / Recommendations
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
3. Education and Tablet Technology
What is happening now?
• Pilot programs on using tablets in schools and higher education are
underway
– NYC, Chicago, Virginia, California, Boston and Internationally
– Intended to be used at home and school in place of textbooks
• Benefits include
– Light-weight
– Variety of applications
– Interactive Learning
• Not many studies done to verify if students learn more, faster or
better with tablets
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
4. What We Are Researching
– Engagement is a key part of learning effectively
• Is the increased use of technology affecting engagement?
– Do new or traditional technology engage students
differently?
• Digital Textbook v. Paper Textbook
– Measuring engagement is complex because it
involves biological and cognitive processes
– Requires an array of methods to be applied
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
6. Pearson Education E-book
Pearson Education
- One of the most innovative leaders in education
Inkling Interactive Digital Textbook Technology
- Same book in digital version and paper version
- Allows for the same interaction (highlighting, post it notes etc…)
Digital Book Paper Book
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
7. Research Methods
What methods did we choose to apply?
Combination of different methods:
- Affectiva’s Q Sensor
- SMI Eye tracking
- Product Reaction Cards
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
8. Q Sensor: Measuring EDA
Q Sensor
Created by Affectiva, a company founded by MIT Media Lab
faculty and students
What is the Q-Sensor?
A wireless biosensor that measures emotional arousal via skin
conductance (Electro Dermal Activity).
High arousal = High User Engagement
Low arousal = Low User Engagement
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
9. Q Sensor: Research Technique
Challenges in using the Q Sensor
The data in the Q Sensor tells you there is some sort of engagement, but not
which type of engagement. High engagement could mean:
Frustration Excitement Stress Delight
The Analysis:
1. Calculate average
EDA Value
2. Calculate Peaks
per minute
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
10. SMI Eye Tracking: Background
• Created by SMI
(SensoMotoric Instruments)
• Advantageous for use on
mobile or tablet devices
• Binocular tracking delivers
eye tracking data over all
distances
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
11. SMI Eye Tracking: Data Analysis
• Combined with Q Sensor data, SMI gives us the ability to try to pin
point engagement
• Ability to analyze areas of interest across participants
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
12. Product Reaction Cards: Background
– Developed by Joey Benedek and Trish
Miner (Microsoft)
– A set of 118 words
• 60% positive
• 40% negative/neutral
– Easy to administer and easy to analyze
– Does not rely on a questionnaire or
rating scales
– Users don’t need to generate words
themselves
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
13. Product Reaction Cards: Technique Used
Process:
– Users were asked to circle 10 words from the list of 118 words
– This process was repeated after each object of study:
• Paper Textbook
• Digital Textbook
– We used the response cards to understand and categorize Affectiva’s Q
Sensor EDA/skin conductance data
Challenges:
– Real time reactions not available
– Participant misinterpretation of word meaning
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
15. Qualitative Findings
Participants
• 10 undergraduate students who owned iPads and used the
internet for at least 10 hours per week
Common iPad Activities
• Most common: accessing the web, checking email, gaming and
social networking
• Additional: eBooks, music, course maintenance, movies and
looking at powerpoints during class
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
16. Which do you prefer and why?
Digital Paper
Versus
50% of students preferred each
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
17. What Do You Like About The Paper vs. The Digital Textbook?
Preferences on Paper Textbook
• Concerns about getting tired
looking at the iPad screen &
getting tired of swiping to flip
pages
• Pictures were clear & didn't
require enlarging to see
• Preferred for financial reasons:
reselling & cheaper to purchase
• Allowed for more freedom and
were more comfortable
• More reliable, as not dependent
on batteries
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
18. What Do You Like About The Digital vs. the Paper Textbook?
Prefer Digital Textbook
• Easier to search for content
• Easier navigation
• Ability to access internet to
do supplemental research
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
20. Quantitative Findings
We plan to use the following information to help us in understanding
user engagement:
- Electro-Dermal Activity Information
- Magnitude = a measure of peak height
- Area Under the Curve = represents peak volume
- Peaks per Minute = is a measure of peak frequency
- Heatmaps of user engagement throughout session
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
21. Quantitative Findings
With the addition of SMI eye tracking, we can:
- Pin point engagement areas and moments effectively
- Get the number of fixations on screen and paper textbooks across
participants
- Create heat maps to determine points of engagement
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
23. What Worked Well
• Ability to observe:
– top-down (Cognitive Perception – post moment) - “I felt…”
– bottom-up (Q Sensor Spikes – at the moment) - emotional reactions
• Affectiva
- Easy to set up
- Extremely detailed data about skin temperature changes, skin
conductance
• Use of SMI Glasses
- Easy to set up
- Eye tracking – precise view of how things worked
- Data analysis capabilities are an advantage - Intuitive
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
25. Challenges
- Coordinating the technology during the tests
- Synching issues
- Video loses small frames (due to editing)
- Combining the data
- 4 cameras (videos)
- Affectiva data
- SMI Data
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
27. Other Methodology Applications
• Areas of Application:
- User Experience, Usability, Product Design, Market Research, Education
Research, Psychology Research, Visual Perception Research
• Product Design
- Use it to find what emotionally engages people
• Usability
- Use it to find out what frustrates people
• Prioritization of design features
- Find what is more frustrating/exciting for the user
Caution: Labor intensive process to combine three methods (but doable)
© Copyright Diego Mendes Amanda Davis Vignesh Krubai
28. Questions?
Thank you to our sponsors who are here and excited to meet you!
The Design and Usability Center
Contact:
Diego Mendes – mendes.diego1@gmail.com
Amanda Davis –adavis.bentley@gmail.com
© Copyright Diego Mendes Amanda Davis Vignesh Krubai Vignesh Krubai – krubai_vign@bentley.edu
Editor's Notes Students are using more and more technology to aid their education, but is that always the best way to go? Do users engage more utilizing new technology, such as Inkling Ipad Textbook than when using traditional technology, such as paper textbook EDA = Electro dermal Activity Reasons iPad was preferred over Paper TextbookEasier to search for contentEasier navigationAbility to access internet to do supplemental researchReasons Paper Textbook was preferred over iPadConcerns about getting tired looking at the iPad screen & getting tired of swiping to flip pagesPictures were clear & didn't require enlarging to seePreferred for reselling & cheaper to purchaseAllowed for more freedom and were more comfortableMore reliable, as not dependent on batteries Reasons iPad was preferred over Paper TextbookEasier to search for contentEasier navigationAbility to access internet to do supplemental researchReasons Paper Textbook was preferred over iPadConcerns about getting tired looking at the iPad screen & getting tired of swiping to flip pagesPictures were clear & didn't require enlarging to seePreferred for reselling & cheaper to purchaseAllowed for more freedom and were more comfortableMore reliable, as not dependent on batteries Reasons iPad was preferred over Paper TextbookEasier to search for contentEasier navigationAbility to access internet to do supplemental researchReasons Paper Textbook was preferred over iPadConcerns about getting tired looking at the iPad screen & getting tired of swiping to flip pagesPictures were clear & didn't require enlarging to seePreferred for reselling & cheaper to purchaseAllowed for more freedom and were more comfortableMore reliable, as not dependent on batteries Add Question Mark
