Presentation at XR Workshop in Wellington, NZ 18 February 2020

1. Wellington 18 Feb 2020
Virtual and Augmented Reality in
Health Care and Rehabilitation

2. Joaquim Jorge
Instituto Superior Técnico
Universidade de Lisboa
Visualization and Multimodal Interfaces
@ INESC-ID Lisboa
http://web.ist.utl.pt/jorgej
ACM/SIGGRAPH Special Conferences
Committee Chair
Editor Computers & Graphics
Research Interests:
Calligraphic Interaction,
Multimodal Interfaces,
Graphical Modeling

3. Pedro F. Campos
University of Madeira
Associate Professor with Habil.
Interactive Technologies Institute
www.experienceaugmentation.com
PI, SUAVE project: Nudging users
towards healthier lifestyles
PI, Augmented Crutches: AR for
helping patients to learn how to
walk with crutches
PI, SENSE:SEAT, Inducing Improved
Mood and Cognition through
Multisensorial Priming
Research Interests:
Virtual Reality
Health and Well-being
Persuasive Computing
Behavior Change Technologies
Multimodal Interaction

4. Daniel Simões Lopes
Instituto Superior Técnico
UT Austin / Portugal
Assistant Professor
Heads Biomedical Research Line
PI of project IT-MEDEX
http://web.ist.utl.pt/daniel.s.lopes
Research Interests:
Computational Geometry,
Interactive Computer Graphics,
Computer-Aided Healthcare,
Interactive Visualization

5. IEEE VR 2020
27th IEEE Conference on Virtual Reality & 3D User Interfaces
March 22nd - 26th, 2020 Atlanta, Georgia, USA.

6. IEEE VR 2021 Lisbon Portugal
March

7. Introduction and Motivation

8. Medical UI/UX

9. Reality–Virtuality Continuum
Real Reality Immersive VR
Sem
i-im
m
ersive
VR
D
im
inished
R
ealitySpatialAR
See-Through
AR
Augm
ented
Virtuality
Mixed Reality
Augmented Reality
Extended Reality

10. Challenges
Building a network of healthcare professionals
Identifying current practices
Linguistic Barrier
Big Picture but no Specs
“Automagic worshippers”

11. Motivation

12. Motivation

13. http://jnm.snmjournals.org/content/45/2/279/F2.expansion.html
http://jnm.snmjournals.org/content/52/10/1501/F1.expansion.html
http://jnm.snmjournals.org/content/51/8/1198/F3.expansion.html
Motivation

14. Extend the Reality: Learning

15. Healthcare Scenarios and Collaboration

16. Scenario: Radiodiagnostics

17. VRRRRoom: Virtual Reality for Radiologists in the
Reading Room CHI 2017

18. VR for Radiologists in the Reading ROOM

19. VR for Radiologists in the Reading Room

20. VR for Radiologists in the Reading Room

21. VR for Radiologists in the Reading Room
https://www.youtube.com/watch?v=bsNwV4BhJOw

22. VR 4 Radiologists in the Reading ROOM
Evaluation
Radiology Neuroradiology
Gynecology
Surgery Dental Implantology
Obstetrics
Medical Specialities
5 no previous experience in VR

23. Virtual Colonoscopy

24. Problem

25. Problematic

26. Interaction Techniques for Immersive CT
Colonography: A Professional Assessment
MICCAI / VCBM 2018

27. https://youtu.be/_9EMhjI3kYw

28. Professional Assessment
5M+2F Doctors
Diagnostic: Minutes << HOURS!

29. Evaluation
+ immersion
+ freedom of movement
+ World In Miniature navigation
- limited camera control
- Improvements needed

30. Augmented Dissection
Table
Table for two … plus one

33. Anatomy Studio: Augmented Virtual Dissection Sessions

35. Anatomy Studio: Augmented Virtual Dissection Sessions

42. Healthcare Scenarios Unesco Chair Of Digital Anatomy
Université Paris Descartes
Unité de recherche en Développement, Imagerie et
Anatomie - Prof. Vincent Delmas et Dr. J.F Uhl

43. Forthcoming Springer HCIS Book
Digital Anatomy

44. VR in the Operations Room

45. Contactless Surgical
Navigation
Germ-free interfaces

46. ?
Conventional Systems: Dimensional Gap

47. Source: O’Hara et al., 2014
VOXEL Explorer: Asepsis

48. VOXEL Explorer: Simple Gestures

49. Novel Spatial Interaction Techniques for
Exploring 3D Medical Images
49
VOXEL Explorer: Track your Body

50. VOXEL Explorer: Gameboy Metaphor

52. Augmented Surgery
More than meets the scalpel

53. The Promise of an Augmented Reality

54. Motivation

55. Laparoscopic HUD
Augmented Laparoscopic
Navigation

56. Motivation

57. Motivation

58. Motivation

59. Laparoscopic Head Mounted Display

62. Interactive
Rehabilitation
Easing the patient’s patience

63. Motivation

64. REPETITION – FEEDBACK – MOTIVATION
Motivation

65. $
COST WEARABLES
Motivation

66. COMPENSATORY
MOVEMENTS
COLLABORATION
Motivation

68. SleeveAR: Augmented Reality for
Rehabilitation
using Realtime Feedback
João Vieira, Maurício Sousa, Artur Arsénio and Joaquim Jorge

69. Motivation

70. Augmented Reality

71. SleeveAR

72. Sleeve

73. Process
Recording
Movement
Guidance
Performance
Review

74. x§x

75. Locomotiver
Exergames for
Locomotion Rehabilitation
in VR

76. VR for Locomotion

79. Augmented Crutches

80. Augmented Crutches
Mobile-projection
system, aimed at
helping people to learn
how to walk with
crutches

81. Personal Experience
1

82. Research
1
Personal Experience
2
Physical
Therapist
2

83. How it works
1 2 3
Answer
questionnaire
Connect to
projector
Walk with
projected visual
cues
3

84. Como funciona

86. Challenges Ahead

87. Nonstandard Equipment

88. Funny User Interfaces

89. Disorientation

90. Fatigue

91. Nausea

92. Bright Future Ahead

93. Raskar | MIT Media Lab
Multi-ModeMulti-Format
IoT
Perception
Ergo,
Hygiene
Gestural Language

94. Interactive Tablets for
Collaborative Scenarios
Related to 3D Medical Image
Exploration
jorgej@acm.org
http://it-medex.inesc-id.pt/home

95. Acknowledgments
ARCADE Afonso Faria, André Domingues
LOCOMOTIVER Alexandre Gordo
VRRRRoom Maurício Sousa, Daniel Mendes, Soraia Paulo
CAVE COLON Pedro Borges, Daniel Medeiros, Soraia Paulo
Voxel TIPs Rita Mendes, Pedro Parreira, Soraia Paulo
Anatomy Studio Maurício Sousa, Daniel Mendes, Rafael dos
Anjos, Soraia Paulo
Voxel Explorer Pedro Parreira, Soraia Paulo
TOOTHFAIRY Filipe Relvas, Soraia Paulo
Laparoscopic HUD Miguel Belo, Soraia Paulo
Minimally Invasive Surgery José Miguel Gomes
ImplantAR Tiago Jerónimo
Plausible Box Figures Nuno Matias
Imperceptible Breathing Filipe Marques

96. Special Thanks to
Mauricio de Sousa
Daniel Pires de Sá Medeiros
Daniel Tavares Mendes
Rafael Kuffner dos Anjos
Soraia Figueiredo Paulo
Beatriz Peres

97. Questions?

98. jorgej@acm.org

99. References

100. D.S. Lopes, et al, Interaction Techniques for Immersive CT
Colonography: A Professional Assessment, Medical Image
Computing and Computer Assisted Intervention – MICCAI 2018.
M. Sousa, D. Mendes, S. Paulo, N. Matela, J. Jorge, D.S. Lopes,
VRRRRoom: Virtual Reality for radiologists in the
reading room, (CHI 2017),
D.S. Lopes, et al, On the utility of 3D hand cursors to explore medical
volume datasets with a touchless interface, Journal of Biomedical
Informatics, 72, Pages 140–149, 2017.
D.S. Lopes, et al, Explicit design of transfer functions for
volume-rendered images by combining histograms,
thumbnails, and sketch-based interaction, The Visual
Computer, December 2018,

101. S F Paulo, et al, Touchless interaction with medical images
based on 3D hand cursors supported by single-foot
input: A case study in dentistry, Journal of Biomedical
Informatics, Volume 100, 2019, 103316, DOI: 10.1016/j.jbi.2019.103316
DS Lopes, et al. 2019. FEETICHE: FEET Input for Contactless
Hand gEsture Interaction. VRCAI ’19, Brisbane,
E R. Zorzal, et al, Anatomy Studio: a Tool for Virtual
Dissection Through Augmented 3D Reconstruction,
Computers & Graphics, DOI: 10.1016/j.cag.2019.09.006

102. If you want to know moreBeatriz Peres et al A Persuasive Approach in
Using Visual Cues to Facilitate Mobility Using
Forearm Crutches.
CHI 18 EA: Digitally Augmenting the Physical
Ground Space with Timed Visual Cues for
Crutch-Assisted Walking.
A Digitally-Augmented Ground Space with
Timed Visual Cues for Facilitating Forearm
Crutches’ Mobility, Full paper , Interact 2019
.