This document provides a summary of a lecture on perception in augmented and virtual reality. It discusses the history of disappearing computers from room-sized to handheld. It reviews the key concepts of augmented reality, virtual reality, and mixed reality on Milgram's continuum. It discusses how perception of reality works through our senses and how virtual reality aims to create an illusion of reality. It covers factors that influence the sense of presence such as immersion, interaction, and realism.

1. PERCEPTION
COMP 4010 Lecture Two
Mark Billinghurst
August 3rd 2021
mark.billinghurst@unisa.edu.au

2. REVIEW

3. The Incredible Disappearing Computer
1960-70’s
Room
1970-80’s
Desk
1980-90’s
Lap
1990-2000’s
Hand
2010 -
Head

4. Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented interaction with real world environments.
Making Interfaces Invisible
(c) Internet of Things

5. Internet of Things (IoT)..
• Embed computing and sensing in real world
• Smart objects, sensors, etc..
(c) Internet of Things

6. Virtual Reality (VR)
• Users immersed in Computer Generated environment
• HMD, gloves, 3D graphics, body tracking

7. Augmented Reality (AR)
• Virtual Images blended with the real world
• See-through HMD, handheld display, viewpoint tracking, etc..

8. AR vs VR

9. Milgram’s Mixed Reality (MR) Continuum
Augmented Reality Virtual Reality
Real World Virtual World
Mixed Reality
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays
Internet of Things

10. Extended Reality (XR)
Augmented Reality Virtual Reality
Real World Virtual World
Mixed Reality
Extended Reality
Internet of Things

11. Metaverse Components
• Four Key Components
• Virtual Worlds
• Augmented Reality
• Mirror Worlds
• Lifelogging

12. History Timeline
https://immersivelifeblog.files.wordpress.com/2015/04/vr_history.jpg

13. Ivan Sutherland (1960s)
1
3
Ivan Sutherland’s Head-Mounted Display (1968)

14. Super Cockpit (1965-80’s)
• US Airforce Research Program
• Wright Patterson Air Force Base
• Tom Furness III
• Multisensory
• Visual, auditory, tactile
• Head, eye, speech, and hand input
• Addressing pilot information overload
• Flight controls and tasks too complicated
• Research only
• big system, not safe for ejecting

15. VPL Research (1985 – 1999)
• First Commercial VR Company
• Jaron Lanier, Jean-Jacques Grimaud
• Provided complete systems
• Displays, software, gloves, etc
• DataGlove, EyePhone, AudioSphere

16. First Industrial Use of AR (1990’s)
• 1992: Tom Caudell at Boeing coined the term “AR.”
• Wire harness assembly application begun
• Lead by Tom Caudell, and David Mizell

17. Desktop VR - 1995
• Expensive - $150,000+
• 2 million polys/sec
• VGA HMD – 30 Hz
• Magnetic tracking

18. Mobile/Wearable Systems (1995)
• 1995 Navicam (Rekimoto)
• Handheld AR
• 1997 Touring Machine (Feiner)
• Backpack AR, GPS, see-through display
• 1998 Tinmith (Thomas, UniSA)
• Outdoor gaming, CAD

19. Rise of Commercial VR Companies
• W Industries/Virtuality (1985 - 97)
• Location based entertainment
• Virtuality VR Arcades
• Division (1989 – 1998)
• Turn key VR systems
• Visual programming tools
• Virtual i-O (1993 -1997)
• Inexpensive gamer HMDs
• Sense8 (1990 - 1998)
• WorldToolKit, WorldUp
• VR authoring tools

20. Mobile Phone AR (2005)
• Mobile Phones
• camera
• processor
• display
• AR on Mobile Phones
• Simple graphics
• Optimized computer vision
• Collaborative Interaction

21. 2008 - Browser BasedAR
• Flash + camera + 3D graphics
• ARToolKit ported to Flash
• High impact
• High marketing value
• Large potential install base
• 1.6 Billion web users
• Ease of development
• Lots of developers, mature tools
• Low cost of entry
• Browser, web camera

22. 2008: Location Aware Phones
Nokia Navigator
Motorola Droid

23. VR Second Wave (2010 - )
• Palmer Luckey
• HMD hacker
• Mixed Reality Lab (MxR) intern
• Oculus Rift (2011 - )
• 2012 - $2.4 million kickstarter
• 2014 - $2B acquisition FaceBook
• $350 USD, 110o FOV

24. Desktop VR in 2016
• Graphics Desktop
• $1,500 USD
• >4 Billion poly/sec
• $600 HMD
• 1080x1200, 90Hz
• Optical tracking
• Room scale

25. Oculus Rift
Sony Morpheus
HTC/Valve Vive
2016 - Rise of Consumer HMDs

26. Social Mobile Camera AR Apps (2015 - )
• SnapChat - Lenses, World Lenses
• Cinco de Mayo lens > 225 million views
• Facebook - Camera Effects
• Google – Word Lens/Translate

27. Hololens (2016)
• Integrated system – Windows
• Stereo see-through display
• Depth sensing tracking
• Voice and gesture interaction
• Note: Hololens2 coming September 2019

28. ARKit/ARcore (2017)
• Visual Inertial Odometry (VIO) systems
• Mobile phone pose tracked by
• Camera (Visual), Accelerometer & Gyroscope (Intertial)
• Features
• Plane detection, lighting detection, hardware optimisation
• Links
• https://developer.apple.com/arkit/
• https://developers.google.com/ar/

29. History Summary
• 1960’s – 80’s: Early Experimentation
• 1980’s – 90’s: Basic Research
• Tracking, displays
• 1995 – 2005: Tools/Applications
• Interaction, usability, theory
• 2005 - : Commercial Applications
• Mobile, Games, Medical, Industry

30. THE BUSINESS OF AR/VR

31. Why 2021 won’t be like 1996
• It’s not just VR anymore
• Huge amount of investment
• Inexpensive hardware platforms
• Easy to use content creation tools
• New devices for input and output
• Proven use cases – no more Hype!
• Most important: Focus on User Experience

32. Example: Pokemon GO
Killer Combo: brand + social + mobile + geo-location + AR

33. Pokemon GO Effect
• Fastest App to reach $500 million in Revenue
• Only 63 days after launch, > $1 Billion in 6 months
• Over 500 million downloads, > 25 million DAU
• Nintendo stock price up by 50% (gain of $9 Billion USD)

34. Augmented Reality in 2021
• Large growing market
• > $7Billion USD in 2020
• Many available devices
• HMD, phones, tablets, HUDs
• Robust developer tools
• Vuforia, ARToolKit, Unity, Wikitude, etc
• Large number of applications
• > 150K developers, > 100K mobile apps
• Strong research/business communities
• ISMAR, AWE conferences, AugmentedReality.org, etc

40. Large Growing Industry

42. Conclusion
• AR/VR has a long history
• > 50 years of HMDs, simulators
• Key elements for were in place by early 1990’s
• Displays, tracking, input, graphics
• Strong support from military, government, universities
• First commercial wave failed in late 1990’s
• Too expensive, bad user experience, poor technology, etc
• We are now in second commercial wave
• Better experience, Affordable hardware
• Large commercial investment, Significant installed user base
• Will XR be a commercial success this time?

43. CONCLUSIONS

44. Conclusions
• Mixed Reality continuum
• AR/VR complimentary technologies
• Key Enabling Technologies
• Display, Interaction, Tracking
• Many possible applications
• Industry, Education, Medicine, etc.
• Rapidly growing market
• Many areas for commercial application

45. PERCEPTION

46. What is Reality?

47. How do We Perceive Reality?
• We understand the world through
our senses:
• Sight, Hearing, Touch, Taste, Smell
(and others..)
• Two basic processes:
• Sensation – Gathering information
• Perception – Interpreting information

48. Simple Sensing/Perception Model

49. Goal of Virtual Reality
“.. to make it feel like you’re actually in a place that
you are not.”
Palmer Luckey
Co-founder, Oculus

50. Creating the Illusion of Reality
• Fooling human perception by using
technology to generate artificial sensations
• Computer generated sights, sounds, smell, etc

51. Reality vs. Virtual Reality
• In a VR system there are input and output devices
between human perception and action

52. Example Birdly - http://www.somniacs.co/
• Create illusion of flying like a bird
• Multisensory VR experience
• Visual, audio, wind, haptic

53. Birdly Demo
• https://www.youtube.com/watch?v=gHE6H62GHoM

54. PRESENCE

55. Presence ..
“The subjective experience of being in one place or
environment even when physically situated in another”
Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence
questionnaire. Presence: Teleoperators and virtual environments, 7(3), 225-240.

56. Immersion vs. Presence
• Immersion: describes the extent to which technology is capable of
delivering a vivid illusion of reality to the senses of a human participant.
• Presence: a state of consciousness, the (psychological) sense of being
in the virtual environment.
• So Immersion, defined in technical terms, is capable of producing a
sensation of Presence
• Goal of VR: Create a high degree of Presence
• Make people believe they are really in Virtual Environment
Slater, M., & Wilbur, S. (1997). A framework for immersive virtual environments (FIVE): Speculations on the role
of presence in virtual environments. Presence: Teleoperators and virtual environments, 6(6), 603-616.

57. How to Create Strong Presence?
• Use Multiple Dimensions of Presence
• Create rich multi-sensory VR experiences
• Include social actors/agents that interact with the user
• Have environment respond to the user
• What Influences Presence
• Vividness – ability to provide rich experience (Steuer 1992)
• Using Virtual Body – user can see themselves (Slater 1993)
• Internal factors – individual user differences (Sadowski 2002)
• Interactivity – how much users can interact (Steuer 1992)
• Sensory, Realism factors (Witmer 1998)

58. Five Key Technical Requirements for Presence
• Persistence
• > 90 Hz refresh, < 3 ms persistence, avoid retinal blur
• Optics
• Wide FOV > 90 degrees, comfortable eyebox, good calibration
• Tracking
• 6 DOF, 360 tracking, sub-mm accuracy, no jitter, good tracking volume
• Resolution
• Correct stereo, > 1K x 1K resolution, no visible pixels
• Latency
• < 20 ms latency, fuse optical tracking and IMU, minimize tracking loop
http://www.roadtovr.com/oculus-shares-5-key-ingredients-for-presence-in-virtual-reality/

59. Example: UNC Pit Room
• Key Features
• Training room and pit room
• Physical walking
• Fast, accurate, room scale tracking
• Haptic feedback – feel edge of pit, walls
• Strong visual and 3D audio cues
• Task
• Carry object across pit
• Walk across or walk around
• Dropping virtual balls at targets in pit
• http://wwwx.cs.unc.edu/Research/eve/walk_exp/

60. Typical Subject Behaviour
• Note – from another pit experiment
• https://www.youtube.com/watch?v=VVAO0DkoD-8

61. Richie’s Plank
• https://www.youtube.com/watch?v=4M92kfnpg-k

62. Benefits of High Presence
• Leads to greater engagement, excitement and satisfaction
• Increased reaction to actions in VR
• People more likely to behave like in the real world
• E.g. people scared of heights in real world will be scared in VR
• More natural communication (Social Presence)
• Use same cues as face-to-face conversation
• Note: The relationship between Presence and Performance is
unclear – still an active area of research

63. Measuring Presence
• Presence is very subjective so there is a lot of debate
among researchers about how to measure it
• Subjective Measures
• Self report questionnaire
• University College London Questionnaire (Slater 1999)
• Witmer and Singer Presence Questionnaire (Witmer 1998)
• ITC Sense Of Presence Inventory (Lessiter 2000)
• Continuous measure
• Person moves slider bar in VE depending on Presence felt
• Objective Measures
• Behavioural
• reflex/flinch measure, startle response
• Physiological measures
• change in heart rate, skin conductance, skin temperature
Presence Slider

64. Types of Presence
• Spatial Presence
• Feeling that you are in another space
• Object Presence
• Feeling that an object is really in your space
• Social Presence
• Feeling that someone is really with you

65. Object Presence
• What makes an object appear real?
• Touch/Haptic feedback
• Appearance
• Lighting
• Audio cues
• Occlusion
• Etc..

67. Social Presence
• What makes a Person appear real?
• Interactivity
• Visual appearance
• Audio cues
• Touch
• Contextual cues
• Etc..
Oh, C. S., Bailenson, J. N., & Welch, G. F. (2018). A systematic review of social presence:
Definition, antecedents, and implications. Frontiers in Robotics and AI, 5, 114.

69. PERCEPTION

70. Motivation
• Understand: In order to create a strong sense of Presence
we need to understand the Human Perception system
• Stimulate: We need to be able to use technology to provide
real world sensory inputs, and create the VR illusion
VR Hardware Human Senses

71. Senses
• How an organism obtains information for perception:
• Sensation part of Somatic Division of Peripheral Nervous System
• Integration and perception requires the Central Nervous System
• Five major senses (but there are more..):
• Sight (Opthalamoception)
• Hearing (Audioception)
• Taste (Gustaoception)
• Smell (Olfacaoception)
• Touch (Tactioception)

72. Relative Importance of Each Sense
• Percentage of neurons in
brain devoted to each
sense
• Sight – 30%
• Touch – 8%
• Hearing – 2%
• Smell - < 1%
• Over 60% of brain involved
with vision in some way

73. Other Lessor Known Senses..
• Proprioception = sense of body position
• what is your body doing right now
• Equilibrium = balance
• Acceleration
• Nociception = sense of pain
• Temperature
• Satiety (the quality or state of being fed or gratified to or beyond capacity)
• Thirst
• Micturition
• Amount of CO2 and Na in blood

74. Sight

75. The Human Visual System
• Purpose is to convert visual input to signals in the brain

76. The Human Eye
• Light passes through cornea and lens onto retina
• Photoreceptors in retina convert light into electrochemical signals

77. Photoreceptors – Rods and Cones
• Retina photoreceptors come in two types, Rods and Cones
• Rods – 125 million, periphery of retina, no colour detection, night vision
• Cones – 4-6 million, center of retina, colour vision, day vision

78. Human Horizontal and Vertical FOV
• Humans can see ~135
o
vertical (60
o
above, 75
o
below)
• See up to ~ 210o
horizontal FOV, ~ 115o
stereo overlap
• Colour/stereo in centre, Black & White/mono in periphery

79. Vergence + Accommodation
• saas

80. Vergence/Accommodation Demo
• https://www.youtube.com/watch?v=p_xLO7yxgOk

81. Vergence-Accommodation Conflict
• Looking at real objects, vergence and focal distance match
• In VR, vergence and accommodation can miss-match
• Focusing on HMD screen, but accommodating for virtual object behind screen

82. Visual Acuity
Visual Acuity Test Targets
• Ability to resolve details
• Several types of visual acuity
• detection, separation, etc
• Normal eyesight can see a 50 cent coin at 80m
• Corresponds to 1 arc min (1/60th of a degree)
• Max acuity = 0.4 arc min

83. Stereo Perception/Stereopsis
• Eyes separated by IPD
• Inter pupillary distance
• 5 – 7.5cm (avge. 6.5cm)
• Each eye sees diff. image
• Separated by image parallax
• Images fused to create 3D
stereo view

85. Depth Perception
• The visual system uses a range of different Stereoscopic
and Monocular cues for depth perception
Stereoscopic Monocular
eye convergence angle
disparity between left
and right images
diplopia
eye accommodation
perspective
atmospheric artifacts (fog)
relative sizes
image blur
occlusion
motion parallax
shadows
texture
Parallax can be more important for depth perception!
Stereoscopy is important for size and distance evaluation

86. Common Depth Cues

87. Depth Perception Distances
• i.e. convergence/accommodation used for depth perception < 10m

88. Properties of the Human Visual System
• visual acuity: 20/20 is ~1 arc min
• field of view: ~200° monocular, ~120° binocular, ~135° vertical
• resolution of eye: ~576 megapixels
• temporal resolution: ~60 Hz (depends on contrast, luminance)
• dynamic range: instantaneous 6.5 f-stops, adapt to 46.5 f-stops
• colour: everything in CIE xy diagram
• depth cues in 3D displays: vergence, focus, (dis)comfort
• accommodation range: ~8cm to ∞, degrades with age

89. Creating the Perfect Illusion
Cuervo, E., Chintalapudi, K., & Kotaru, M. (2018,
February). Creating the perfect illusion: What will it
take to create life-like virtual reality headsets?.
In Proceedings of the 19th International Workshop on
Mobile Computing Systems & Applications (pp. 7-12).
• Technology to create life-like VR HMDs
• Compared to current HMDs
• 6 − 10× higher pixel density
• 20 − 30× higher frame rate

90. Comparison between Eyes and HMD

91. Hearing

92. Anatomy of the Ear

93. Auditory Thresholds
• Humans hear frequencies from 20 – 22,000 Hz
• Most everyday sounds from 80 – 90 dB

94. Sound Localization
• Humans have two ears
• localize sound in space
• Sound can be localized
using 3 coordinates
• Azimuth, elevation,
distance

95. Sound Localization
• https://www.youtube.com/watch?v=FIU1bNSlbxk

96. Sound Localization (Azimuth Cues)
Interaural Time Difference

97. HRTF (Elevation Cue)
• Pinna and head shape affect frequency intensities
• Sound intensities measured with microphones in ear and
compared to intensities at sound source
• Difference is HRTF, gives clue as to sound source location

98. Accuracy of Sound Localization
• People can locate sound
• Most accurately in front of them
• 2-3° error in front of head
• Least accurately to sides and behind head
• Up to 20° error to side of head
• Largest errors occur above/below elevations and behind head
• Front/back confusion is an issue
• Up to 10% of sounds presented in the front are perceived
coming from behind and vice versa (more in headphones)
BUTEAN, A., Bălan, O., NEGOI, I., Moldoveanu, F., & Moldoveanu, A. (2015). COMPARATIVE RESEARCH
ON SOUND LOCALIZATION ACCURACY IN THE FREE-FIELD AND VIRTUAL AUDITORY DISPLAYS.
InConference proceedings of» eLearning and Software for Education «(eLSE)(No. 01, pp. 540-548).
Universitatea Nationala de Aparare Carol I.

99. Touch

100. Haptic Sensation
• Somatosensory System
• complex system of nerve cells that responds to changes to
the surface or internal state of the body
• Skin is the largest organ
• 1.3-1.7 square m in adults
• Tactile: Surface properties
• Receptors not evenly spread
• Most densely populated area is the tongue
• Kinesthetic: Muscles, Tendons, etc.
• Also known as proprioception

101. Cutaneous System
• Skin – heaviest organ in the body
• Epidermis outer layer, dead skin cells
• Dermis inner layer, with four kinds of mechanoreceptors

102. Mechanoreceptors
• Cells that respond to pressure, stretching, and vibration
• Slow Acting (SA), Rapidly Acting (RA)
• Type I at surface – light discriminate touch
• Type II deep in dermis – heavy and continuous touch
Receptor Type Rate of
Acting
Stimulus
Frequency
Receptive Field Detection Function
Merkel Discs SA-I 0 – 10 Hz Small, well defined Edges, intensity
Ruffini
corpuscles
SA-II 0 – 10 Hz Large, indistinct Static force,
skin stretch
Meissner
corpuscles
RA-I 20 – 50 Hz Small, well defined Velocity, edges
Pacinian
corpuscles
RA-II 100 – 300 Hz Large, indistinct Acceleration,
vibration

103. Spatial Resolution
• Sensitivity varies greatly
• Two-point discrimination
Body
Site
Threshold
Distance
Finger 2-3mm
Cheek 6mm
Nose 7mm
Palm 10mm
Forehead 15mm
Foot 20mm
Belly 30mm
Forearm 35mm
Upper Arm 39mm
Back 39mm
Shoulder 41mm
Thigh 42mm
Calf 45mm
http://faculty.washington.edu/chudler/chsense.html

104. Proprioception/Kinaesthesia
• Proprioception (joint position sense)
• Awareness of movement and positions of body parts
• Due to nerve endings and Pacinian and Ruffini corpuscles at joints
• Enables us to touch nose with eyes closed
• Joints closer to body more accurately sensed
• Users know hand position accurate to 8cm without looking at them
• Kinaesthesia (joint movement sense)
• Sensing muscle contraction or stretching
• Cutaneous mechanoreceptors measuring skin stretching
• Helps with force sensation

105. www.empathiccomputing.org
@marknb00
mark.billinghurst@auckland.ac.nz