A presentation given by Mark Billinghurst on April 21st 2015 at the CHI 2015 conference. This talk presents highlights from the journal paper: M. Billinghurst, A. Clark, and G. Lee. A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction. Vol. 8, No. 1 (2015) 1–202, 2015 Available at :http://www.nowpublishers.com/article/Details/HCI-049

1. A Survey of Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
April 21st 2014
mark.billinghurst@hitlabnz.org

2. Highlights from:
M. Billinghurst, A. Clark, and G. Lee. A Survey
of Augmented Reality, Foundations and
Trends in Human-Computer Interaction.
Vol. 8, No. 1 (2015) 1–202, 2015
http://www.nowpublishers.com/article/Details/HCI-049

3. Introduction

4. 1977 – Star Wars

5. Augmented Reality Definition
! Defining Characteristics [Azuma 97]
! Combines Real and Virtual Images
- Both can be seen at the same time
! Interactive in real-time
- The virtual content can be interacted with
! Registered in 3D
- Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

6. 2008 - CNN

7. ! Put AR pictures here
Augmented Reality Examples

8. Virtual Reality
! Immersive VR
! Head mounted display, gloves
! Separation from the real world

9. AR vs VR

10. Making Interfaces Invisible
Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented
interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on
User interface and Software Technology. UIST '95. ACM, New York, NY, 29-36.

11. From Reality to Virtual Reality
Ubiquitous Computing Augmented Reality Virtual Reality

12. Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

13. Metaverse
! Neal Stephenson’s “SnowCrash”
! The Metaverse is the convergence of:
! 1) virtually enhanced physical reality
! 2) physically persistent virtual space
! Metaverse Roadmap
! http://metaverseroadmap.org/

15. Summary
! Augmented Reality has three key features
! Combines Real and Virtual Images
! Interactive in real-time
! Registered in 3D
! AR can be classified alongside other technologies
! Invisible Interfaces
! Milgram’s Mixed Reality continuum
! Stephenson’s MetaVerse

16. History

17. Pepper’s Ghost (1862)
! Dates back to Giambattista della Porta (1584)

18. Early HMDs and HUDs (1960’s)
Sutherland / Sproull’s see-
through HMD (1965)
Bucaneer HUD (1958)

19. AR History
! 1960’s – 80’s: Early Experimentation
• Military, Academic labs
! 1980’s – 90’s: Basic Research
• Tracking, Displays
! 1995 – 2005: Tools/Applications
• Interaction, Usability, Theory
! 2005 - : Commercial Applications
• Games, Medical, Industry, Mobile

20. Important Milestones
! 1992: T. Caudell coined term “AR” (Boeing)
! 1996: First collaborative AR systems
! 1999: ARToolKit released – tracking library
! 1997: Feiner First outdoor AR system
! 1997 – 2001: Mixed Reality Systems Lab (Japan)
! 1999 – 2003: ARVIKA project (Germany)
! 2002: ISMAR conference started

21. AR Enters Mainstream (2007 -)
! Magazines
! MIT Tech. Review (Mar 2007)
- 10 most exciting technologies
! Economist (Dec. 2007)
- Reality, only better
! Games
! Sony “Eye of Judgement”
- 300,000+ units shipped
! Broadcast TV
! Sports broadcasting

22. Google Searches for AR

23. Browser Based AR (2008 - )
! Flash + Camera + 3D graphics
! High impact
! High marketing value
! Large potential install base
! 1.6 Billion web users
! Ease of development
! Lots of developers, mature tools
! Low cost of entry
! Browser, web camera

24. Mobile AR (2005 - )
! Mobile Phones
! Camera, processor, display
! Computer vision based AR
! Advertising
! HIT Lab NZ (2007)
! AR print advertisement
! Txt to download app

25. Mobile Outdoor AR (2009 - )
! Mobile phones with GPS
! Tag real world locations
! GPS + Compass input
! Overlay graphics data on live video
! Applications
! Travel guide, Advertising, etc
! Wikitude, Layar, Junaio, etc..
! Android/iOS based, Public API released
Motorola Droid

26. AR Today
! Key Technologies Available
- Robust tracking (Computer Vision, GPS/sensors)
- Display (Handheld, HMDs)
- Input Devices (Kinect, etc)
- Developer tools (Qualcomm, Metaio, Wikitude)
! Commercial Business Growing
- Gaming, GPS/Mobile, Online Advertisement
• >$5 Billion USD by 2016 (Markets andMarkets)
• >$1.5 Billion USD in Mobile AR by 2014 (Juniper Research)

27. AR Business Today
! Marketing
! Web-based, mobile
! Mobile AR
! Geo-located information and service
! Gaming
! Mobile, Physical input (Kinect, PS Move)
! Upcoming areas
! Manufacturing, Medical, Military

28. Summary
! Augmented Reality has a long history going
back to the 1960’s
! Interest in AR has exploded over the last few
years and is being commercialized quickly
! AR is growing in a number of areas
! Mobile AR
! Web based AR
! Marketing experiences

29. Technology

30. Augmented Reality Definition
! Defining Characteristics
! Combines Real and Virtual Images
- Display Technology
! Interactive in real-time
- Interaction Technology
! Registered in 3D
- Tracking Technology

31. Display

32. Display Technologies
" Types (Bimber/Raskar 2003)
" Head attached
• Head mounted display/projector
" Body attached
• Handheld display/projector
" Spatial
• Spatially aligned projector/monitor

33. Display Taxonomy

34. Types of Head Mounted Displays
Occluded
See-thru
Multiplexed

35. Optical see-through HMD
Virtual images
from monitors
Real
World
Optical
Combiners

36. Epson Moverio BT-200
▪ Stereo see-through display ($700)
▪ 960 x 540 pixels, 23 degree FOV, 60Hz, 88g
▪ Android Powered, separate controller
▪ VGA camera, GPS, gyro, accelerometer

37. Strengths of optical see-through
! Simpler (cheaper)
! Direct view of real world
! Full resolution, no time delay (for real world)
! Safety
! Lower distortion
! No eye displacement
! see directly through display

38. Video see-through HMD
Video
cameras
Monitors
Graphics
Combiner
Video

39. Vuzix Wrap 1200DXAR
▪ Stereo video see-through display ($1500)
■ Twin 852 x 480 LCD displays, 35 deg. FOV
■ Stereo VGA cameras
■ 3 DOF head tracking

40. Strengths of Video See-Through
! True occlusion
! Block image of real world
! Digitized image of real world
! Flexibility in composition
! Matchable time delays
! More registration, calibration strategies
! Wide FOV is easier to support
! wide FOV camera

41. Multiplexed Displays
! Above or below line of sight
! Strengths
! User has unobstructed view of real world
! Simple optics/cheap
! Weaknesses
! Direct information overlay difficult
• Display/camera offset from eyeline
! Wide FOV difficult

42. Google Glass
▪ Monocular see-through multiplexed display
▪ 640 x 360 microprojector, 15 degree FOV
▪ 5 MP camera, gyro, accelerometer

43. Display Technology
! Curved Mirror
! off-axis projection
! curved mirrors in front of eye
! high distortion, small eye-box
! Waveguide
! use internal reflection
! unobstructed view of world
! large eye-box

44. See-through thin displays
! Waveguide techniques for thin see-through displays
! Wider FOV, enable AR applications
! Social acceptability
Opinvent Ora
Lumus DK40

45. Spatial Augmented Reality
! Project onto irregular surfaces
! Geometric Registration
! Projector blending, High dynamic range
! Book: Bimber, Rasker “Spatial Augmented Reality”

46. Projector-based AR
Examples:
Raskar, MIT Media Lab
Inami, Tachi Lab, U. Tokyo
Projector
Real objects
with retroreflective
covering
User (possibly
head-tracked)

47. Example of projector-based AR
Ramesh Raskar (MIT Media Lab)

48. Head Mounted Projector
! NVIS P-50 HMPD
! 1280x1024/eye
! Stereoscopic
! 50 degree FOV
! www.nvis.com

49. HMD vs. HMPD
Head Mounted Display Head Mounted Projected Display

50. CastAR - http://technicalillusions.com/
! Stereo head worn projectors
! Interactive wand
! Rollable retro-reflective sheet

51. ! Designed for shared interaction

52. Video Monitor AR
Video
cameras Monitor
Graphics Combiner
Video
Stereo
glasses

53. Examples

54. Handheld Displays
! Mobile Phones
! Camera
! Display
! Input

55. Tracking

56. Objects Registered in 3D
! Registration
! Positioning virtual object wrt real world
! Tracking
! Continually locating the users viewpoint
- Position (x,y,z), Orientation (r,p,y)

57. Tracking Technologies
! Active
• Mechanical, Magnetic, Ultrasonic
• GPS, Wifi, cell location
! Passive
• Inertial sensors (compass, accelerometer, gyro)
• Computer Vision
• Marker based, Natural feature tracking
! Hybrid Tracking
• Combined sensors (eg Vision + Inertial)

58. Tracking Types
Magnetic
Tracker
Inertial
Tracker
Ultrasonic
Tracker
Optical
Tracker
Marker-Based
Tracking
Markerless
Tracking
Specialized
Tracking
Edge-Based
Tracking
Template-Based
Tracking
Interest Point
Tracking
Mechanical
Tracker

59. Example: Marker tracking
! Available for more than 10 years
! Several open source solutions exist
! ARToolKit, ARTag, ATK+, etc
! Fairly simple to implement
! Standard computer vision methods
! A rectangle provides 4 corner points
! Enough for pose estimation!

60. Marker Based Tracking: ARToolKit
http://artoolkit.sourceforge.net/

61. Coordinate Systems

62. Tracking challenges in ARToolKit
False positives and inter-marker confusion
(image by M. Fiala)
Image noise
(e.g. poor lens, block coding /
compression, neon tube)
Unfocused camera,
motion blur
Dark/unevenly lit
scene, vignetting
Jittering
(Photoshop illustration)
Occlusion
(image by M. Fiala)

64. Markerless Tracking
Magnetic Tracker Inertial
Tracker
Ultrasonic
Tracker
Optical
Tracker
Marker-Based
Tracking
Markerless
Tracking
Specialized
Tracking
Edge-Based
Tracking
Template-Based
Tracking
Interest Point
Tracking
! No more Markers! #Markerless Tracking
Mechanical
Tracker

65. Natural Feature Tracking
! Use Natural Cues of Real Elements
! Edges
! Surface Texture
! Interest Points
! Model or Model-Free
! No visual pollution
Contours
Features Points
Surfaces

66. Texture Tracking

67. Edge Based Tracking
! RAPiD [Drummond et al. 02]
! Initialization, Control Points, Pose Prediction (Global Method)

68. Line Based Tracking
! Visual Servoing [Comport et al. 2004]

69. Model Based Tracking
! OpenTL - www.opentl.org
! General purpose library for model based visual tracking

70. Marker vs. natural feature tracking
! Marker tracking
! ++ Markers can be an eye-catcher
! ++ Tracking is less demanding
! -- The environment must be instrumented with markers
! -- Markers usually work only when fully in view
! Natural feature tracking
! -- A database of keypoints must be stored/downloaded
! ++ Natural feature targets might catch the attention less
! ++ Natural feature targets are potentially everywhere
! ++ Natural feature targets work also if partially in view

71. Example: Outdoor Hybrid Tracking
! Combines
! computer vision
- natural feature tracking
! inertial gyroscope sensors
! Both correct for each other
! Inertial gyro - provides frame to frame
prediction of camera orientation
! Computer vision - correct for gyro drift

72. Robust Outdoor Tracking
! Hybrid Tracking
! Computer Vision, GPS, inertial
! Going Out
! Reitmayr & Drummond (Univ. Cambridge)
Reitmayr, G., & Drummond, T. W. (2006). Going out: robust model-based tracking for outdoor augmented
reality. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp.
109-118). IEEE.

73. Handheld Display

74. Interaction

75. ! Interface Components
! Physical components
! Display elements
- Visual/audio
! Interaction metaphors
Physical
Elements
Display
ElementsInteraction
MetaphorInput Output
AR Interface Elements

76. AR Design Space
Reality Virtual Reality
Augmented Reality
Physical Design Virtual Design

77. Interface Design Path
1/ Prototype Demonstration
2/ Adoption of Interaction Techniques from other
interface metaphors
3/ Development of new interface metaphors
appropriate to the medium
4/ Development of formal theoretical models for
predicting and modeling user actions
Desktop WIMP
Virtual Reality
Augmented Reality

78. Interaction Development
! Information Browsing
! Camera movement
! Limited interaction
! 3D AR Interaction
! HMD, hand tracking
! 3D UI/VR techniques
! Specialized input devices

79. Tangible User Interfaces (Ishii 97)
! Augmented Surfaces
! Rekimoto 1998
! Multiple projection surfaces
! Tangible prop interaction
! i/O Brush (2004)
! Ryokai, Marti, Ishii
! Sensor enhanced real brush

80. Other Examples
! Triangles (Gorbert 1998)
! Triangular based story telling
! ActiveCube (Kitamura 2000-)
! Cubes with sensors

81. Lessons from Tangible Interfaces
! Benefits
! Physical objects make us smart (affordances)
! Objects aid collaboration
! Objects increase understanding (cognitive artifacts)
! Limitations
! Difficult to change object properties
! Limited display capabilities (2D surface)
! Separation between object and display

82. Orthogonal Nature of AR Interfaces

83. Tangible AR Interaction
! AR overcomes limitation of TUIs
! enhance display possibilities
! merge task/display space
! provide public and private views
! TUI + AR = Tangible AR
! Apply TUI methods to AR interface design

84. Tangible AR Design Principles
! Tangible AR Interfaces use TUI principles
! Physical controllers for moving virtual content
! Support for spatial 3D interaction techniques
! Support for multi-handed interaction
! Match object affordances to task requirements
! Support parallel activity with multiple objects
! Allow collaboration between multiple users

85. VOMAR - Tangible AR Interface
! Use of natural physical object
to control virtual objects
! Physical objects
! Catalog book:
- Turn over the page
! Paddle operation:
- Push, shake, incline, hit, scoop
Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000). Virtual object manipulation on
a table-top AR environment. In Augmented Reality, 2000.(ISAR 2000). Proceedings. IEEE and ACM
International Symposium on (pp. 111-119). Ieee.

86. Wrap-up
! Browsing Interfaces
! simple (conceptually!), unobtrusive
! 3D AR Interfaces
! expressive, creative, require attention
! Tangible Interfaces
! Embedded into conventional environments
! Tangible AR
! Combines TUI input + AR display

87. Typical Applications

88. ! Web based AR
! Flash, HTML 5 based AR
! Marketing, education
! Outdoor Mobile AR
! GPS, compass tracking
! Viewing Points of Interest in real world
! Eg: Junaio, Layar, Wikitude
! Handheld AR
! Vision based tracking
! Marketing, gaming
! Location Based Experiences
! HMD, fixed screens
! Museums, point of sale, advertising
Typical AR Experiences

89. CityViewAR Application
! Visualize Christchurch before the earthquakes

90. User Experience
! Multiple Views
! Map View, AR View, List View
! Multiple Data Types
! 2D images, 3D content, text, panoramas

91. Demo: Warp Runner
! Puzzle solving game
! Deform real world terrain

92. Demo: colAR
! Turn colouring books pages into AR scenes
! Markerless tracking, use your own colours..
! Try it yourself: http://www.colARapp.com/

93. What Makes a Good AR Experience?
! Compelling
! Engaging, ‘Magic’ moment
! Intuitive, ease of use
! Uses existing skills
! Anchored in physical world
! Seamless combination of real and digital

94. Research Opportunities

95. Previous Research Themes
! Zhou et. al. (2008) analyzed all ISMAR papers
Zhou, F., Duh, H. B. L., & Billinghurst, M. (2008). Trends in augmented reality tracking, interaction and display:
A review of ten years of ISMAR. In Proceedings ISMAR 2008, (pp. 193-202). IEEE Computer Society.

96. Possible Research Directions
! Tracking
! Wide area, Reliable indoor, Ubiquitous tracking
! Interaction
! Intelligent systems, Gesture, Collaborative systems
! Displays
! Wide FOV, Retinal Scanning, Contact Lens
! Social Acceptance
! Wearable AR, Handheld AR, Social Interactions

97. Wide Area Tracking
! Process
! Combine panorama’s into point cloud model (offline)
! Initialize camera tracking from point cloud
! Update pose by aligning camera image to point cloud
! Accurate to 25 cm, 0.5 degree over wide area
Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking.In Mixed
and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.

98. Gesture Based Interaction
! Use free hand gestures to interact
! Depth camera, scene capture
! Multimodal input
! Combining speech and gesture
HIT Lab NZ Microsoft Hololens
Meta SpaceGlasses

99. Wide FOV Displays
! Wide FOV see-through display for AR
! LCD panel + edge light point light sources
! 110 degree FOV
Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays: wide
field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH 2014
Emerging Technologies (p. 20). ACM.

100. Social Acceptance
! People don’t want to look silly
! Only 12% of 4,600 adults would be willing to wear AR glasses
! 20% of mobile AR browser users experience social issues
! Acceptance more due to Social than Technical issues
! Needs further study (ethnographic, field tests, longitudinal)

101. Conclusions

102. Conclusion
! AR seamlessly blends real and virtual imagery
! Interactive in real time, fixed in space
! AR has developed into a mass market technology
! Education, engineering, entertainment
! The technologies to create AR are available
! Display, tracking, interaction
! There are opportunities for significant research

103. Gartner Hype Cycle (2014)

104. Market Forecast
Up to $120B by 2020 – 5 x VR market

105. User Forecast

106. More Information
• Mark Billinghurst
– Email: mark.billinghurst@hitlabnz.org
– Twitter: @marknb00
• Website
– http://www.hitlabnz.org/