Empathic Extended Reality Exploring Physiological Data Sharing in Virtual Environments by Arindam Dey

1. Empathic Extended Reality:
Exploring Physiological Data
Sharing in Virtual Environments
ARINDAM DEY
EMPATHIC EXTENDED REALITY AND PERVASIVE COMPUTING LAB
UNIVERSITY OF QUEENSLAND
A.DEY@UQ.EDU.AU

3. https://www.youtube.com/watch?v=t3g2IBSaYb4

6. Empathy
“Seeing with the Eyes of another,
Listening with the Ears of another,
and Feeling with the Heart of another …”
Alfred Adler

7. Empathic Computing
Understanding
Systems that can understand your feeling and emotions
Experiencing
Systems that can help you better experience the world
Sharing
Systems that help you better share your experiences with others
Sensors
Extended
Reality

8. “Virtual Reality is the ultimate empathy
machine …”
- Chris Milk

9. https://www.youtube.com/watch?v=digE62wpHOk
Project Syria – Nonny de la Peña

10. Empathic XR
Extended Reality Systems that can
Measure | Share | Adapt to | Manipulate
Emotion and Cognition in real time
-Physiological (ECG, EDA, EMG, Pupil)
-Neurological (EEG)
-Behavioural (Speech, Posture, Movement)

11. Sharing Physiological States in Collaborative VR
ACM CHI’17

13. Data Collected
- Raw heart-rate
- Positive and negative affect schedule (PANAS)
- Relative head orientation
Results
- Higher positive affect
- More communication between collaborators
- Observer interaction needed
Sharing Physiological States in Collaborative VR
ACM CHI’17

14. Multi-Sensory Heart Rate Feedback

15. Multi-Sensory Heart Rate Feedback
Results
- Audio-Haptic ranked best
- More interaction needed for higher presence
Data Collected
- PANAS
- Ranking
- Raw heart-rate
OzCHI’18

16. Manipulating Heart Rate Feedback in VR
ACM CHI Play’18
PANAS

17. Multi-Sensory Heart Rate Feedback in Collaborative VR
IEEE ISMAR’18

18. Multi-Sensory Heart Rate Feedback in Collaborative VR
IEEE ISMAR’18
PANAS
Social
Presence

19. Shared and Manipulated Heartrate Feedback in VR
IEEE ISMAR’19
Active
Passive
Manipulation -20%, 0%, +20%

20. Shared and Manipulated Heartrate Feedback in VR
Social
Presence
IEEE ISMAR’19

21. Lessons Learned
Social presences increases with
- Shared awareness cues
- Higher interaction
Shared physiological feedback can
- Increase communication
- Alter emotions but not real signals
- Increase awareness but may not change action

22. Brain Signals and VR

23. Emotion Sensing HMDs
Looxid Emteq Neurable
HMDs equipped with neurological and physiological sensors

24. Cognitively Adaptive Training in VR
Looxid
Training difficulty level adjusted based on alpha activity in real time
Low level High level
IEEE VR’19

25. Cognitively Adaptive Training in VR
Looxid
IEEE VR’19
Alpha activity changed without significant change in task completion time

26. Quantifying Presence
Looxid
Self-reported Questionnaires vs. real-time Physiological and Neurological Data

27. Quantifying Presence with Brain Signals
Looxid
High Presence
Low Presence

28. Quantifying Presence with Brain Signals
Looxid
High
Presence
Low
Presence

29. Next steps: Using EEG and other physiological sensors
Looxid
• Emotionally and cognitively adapt interfaces for learning and training
• Understand how brain synchronization works in collaborative XR
• Effectively share emotional and cognitive states to create empathy in
collaborative XR

30. Thank You!
Looxid
Arindam Dey
Empathic Extended Reality and Pervasive Computing Lab
Co-Innovation Group | University of Queensland
http://empathicxr.org | a.dey@uq.edu.au