A short course on how to develop AR and VR experiences using Unity. Using Unity 2017.2, Google 1.100 VR SDK, and Vuforia. Taught by Mark Billinghurst on November 7th 2017.
5. What is Virtual Reality?
Virtual reality is..
a computer technology that replicates an
environment, real or imagined, and simulates a
user's physical presence and environment to
allow for user interaction. (Wikipedia)
• Defining Characteristics
• Environment simulation
• Presence
• Interaction
6. First VR Experience
• “This is so real..”
• https://www.youtube.com/watch?v=pAC5SeNH8jw
10. Augmented Reality Definition
•Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
14. Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.
19. MobileAR 2009 - Outdoor Overlay
• Mobile phone based
• Tag real world locations
• GPS + Compass input
• Overlay graphics on live video
• Applications
• Travel guide,Advertising, etc
• Wikitude, Layar, etc..
• iOS/Android, Public API released
21. Vuforia Featured Apps (2015)
• Mobile AR using Vuforia image tracking
• https://www.youtube.com/watch?v=rohLgtLQrK4
22. Mobile Camera AR Apps (2015 - )
• SnapChat - Lenses, World Lenses
• Cinco de Mayo lens > 225 million views
• Facebook - Camera Effects
• Google – Word Lens/Translate
23. ARKit/ARcore (2017)
• Visual Inertial Odometry (VIO) systems
• Mobile phone pose tracked by
• Camera (Visual), Accelerometer & Gyroscope (Intertial)
• Features
• Plane detection, lighting detection, hardware optimisation
• Links
• https://developer.apple.com/arkit/ https://developers.google.com/ar/
30. more reading at http://www.gizmag.com/google-cardboard-2-review-initial/37777/
Presses the
screen
Figure: google cardboard v2 supports larger phone.
Google Cardboard (v2)
32. Types of VR Experiences
• Immersive Spaces
• 360 Panorama’s/Movies
• High visual quality
• Limited interactivity
• Changing viewpoint orientation
• Immersive Experiences
• 3D graphics
• Lower visual quality
• High interactivity
• Movement in space
• Interact with objects
33. Immersive Panorama
• High quality 360 image or video surrounding user
• User can turn head to see different views
• Fixed position
34. Example Applications
• Within – Storytelling for VR
• https://with.in/
• High quality 360 VR content
• New York Times VR Experience
• NYTVR application
• Documentary experiences
• Vimeo360
• https://join.vimeo.com/360/
• Immersive 360 movies
36. Applications: Sports and Entertainment
•www.Nexvr.com
•Live streaming events
•NBA Basketball VR
•(1 game per week)
•US Open Tennis VR
•Live Nation concerts
37. Capturing Panoramas
• Stitching photos together
• Image Composite Editor (Microsoft)
• AutoPano (Kolor)
• Using 360 camera
• Ricoh Theta-S
• Fly360
38. Capturing 360 images
Kodak 360 Fly 360 Gear 360 Theta S Nikon
LG 360 Pointgrey Ladybug Panono 360 Bublcam
39. Example: Cardboard Camera
• Capture 360 panoramas
• Stitch together images on phone
• View in VR on Cardboard
41. Google Cardboard App
• 7 default experiences
• Earth: Fly on Google Earth
• Tour Guide: Visit sites with guides
• YouTube: Watch popular videos
• Exhibit: Examine cultural artifacts
• Photo Sphere: Immersive photos
• Street View: Drive along a street
• Windy Day: Interactive short story
47. Tools for Non-Programmers
• Focus on Design, ease of use
• Visual Programming, content arrangement
• Examples
• Insta-VR – 360 panoramas
• http://www.instavr.co/
• Vizor – VR on the Web
• http://vizor.io/
• A-frame – HTML based
• https://aframe.io/
• ENTiTi – Both AR and VR authoring
• http://www.wakingapp.com/
• Eon Creator – Drag and drop tool for AR/VR
• http://www.eonreality.com/eon-creator/
57. Building Scenes
• Use GameObjects:
• Containers that hold different components
• Eg 3D model, texture, animation
• Use Inspector
• View and edit object properties and other settings
• Use Scene View
• Position objects, camera, lights, other GameObjects etc
• Scripting
• Adding interaction, user input, events, etc
58. GameObjects
• Every object in Scene is a GameObject
• GameObjects contain Components
• Eg Transform Component, Camera Component
59. Adding 3D Content
• Create 3D asset using modeling package, or download
• Fbx, Obj file format for 3D models
• Add file to Assets folder in Project
• When project opened 3D model added to Project View
• Drag mesh from Project View into Hierarchy or Scene View
• Creates a game object
63. Making a Simple Scene
1. Create New Project
2. Create Game Object
3. Moving main camera position
4. Adding lights
5. Adding more objects
6. Adding physics
7. Changing object materials
8. Adding script behaviour
73. Example C# Script
GameObject Rotation
using UnityEngine;
using System.Collections;
public class spin : MonoBehaviour {
// Use this for initialization
void Start () {
}
// Update is called once per frame
void Update () {
this.gameObject.transform.Rotate(Vector3.up*10);
}
}
74. Scripting C# Unity 3D
• void Awake():
• Is called when the first scene is loaded and the game object is active
• void Start():
• Called on first frame update
• void FixedUpdate():
• Called before physics calculations are made
• void Update():
• Called every frame before rendering
• void LateUpdate():
• Once per frame after update finished
76. Resources
• Unity Main site
• http://www.unity3d.com/
• Holistic Development with Unity
• http://holistic3d.com
• Official Unity Tutorials
• http://unity3d.com/learn/tutorials
• Unity Coder Blog
• http://unitycoder.com
78. Key Steps
1. Creating new project
2. Loading Google VR package
3. Adding VR view emulation
4. Getting a panorama image
5. Creating a Skybox material
6. Adding a Skybox
7. Testing application
8. Deploying to mobile phone
79. 1. New Project
• Double click Unity icon
• or within Unity File -> New Project
91. 7. Testing Application
• Select Maximize on Play
• Hit play arrow and use mouse + Alt/Ctrl/Command keys
92. 8. Deploying to Mobile Phone
• Go to Build Settings
• File -> Build Settings
• Change platform to Android
• Add open scenes
• Check Development Build
• In Player Settings
• XR Settings -> VR supported
• Other Settings
• Change Package Name
• E.g. com.BDVApano.PanoTest
94. Key Steps
1. Create New Scene + add GvrEditorEmulator
2. Create body object and add camera to it
3. Add 3D objects to scene
4. Test 3D scene
95. 1. Create Scene
• File -> New Scene
• Drag GvrEditorEmulator into hierarchy
• Prefabs -> GvrEditorEmulator
• Save scene (File -> Save Scene)
96. 2. Create Body
• Needed for correct camera motion
• Create empty object and name it Body
• GameObject -> Create Empty
• Change y value of Body to 1.0 (or similar height)
• Drag Main Camera under Body
97. 3. Create Scene
• Add a ground plane
• Game Object -> 3D Object -> Plane
• Add material to ground plane
• Assets -> Create Material
• Add cube, sphere, cylinder + materials, position around plane
• Move Main Camera to (0,0,0) point
98. 4. Test Scene
• Select Maximize on Play
• Hit play arrow and use mouse + Alt/Ctrl/Command keys
100. Key Steps
1. Creating a new project
2. Load Cardboard SDK
3. Replace camera with CardboardMain
4. Loading in 3D asset packages
5. Loading a SkyDome
6. Adding a plane floor
103. Load Asset + Add to Scene
• Assets -> Import Package -> Custom Package
• Look for MagicLamp.unitypackage (If not installed already)
• Drag MagicLamp_LOD0 to Hierarchy
• Position and rotate
110. Gaze Interaction
• Using Gaze interaction is natural for VR
• Without an eye-tracker it’s really head point
• Goal is to have objects that respond to Gaze input
• E.g. look at objects to have them change colour
• Gaze + button input is particularly powerful
• Gaze to highlight, then button click to select
111. Key Steps
•Using previous scene:
1. Add Event System
2. Add Gaze Reticle to Main Camera
3. Add Physics Raycaster to Camera
4. Add Event Trigger to Object
5. Create Gaze Input Script
6. Connect Script Functions
7. Test Scene
112. 1. Add Event System
• Need an Event System object to monitor events
• E.g. Gaze going on and off an object
• Google VR package has built in Event System
• GoogleVR à Prefabs à EventSystem
• Drag GvrEventSystem prefab into hierarchy
113. 2. Add Gaze Reticle to Camera
• Gaze reticle shows the center of the camera view
• Gaze reticle prefabs
• GoogleVR à Prefabs à Cardboard
• Drag GvrReticlePointer under Main Camera
• In inspector click on ReticlePointer to set colour (red in this case)
114. 3. Add Physics Raycaster to Camera
• Used to send ray from camera to collide
with objects in the scene
• Select Main Camera
• Add Gvr Pointer Raycaster Component
• Add Component à GvrPointerPhysicsRaycaster
115. 4. Add Event Trigger to Object
• Used to cause event to trigger when gazing at object
• Select Cube
• Add Event Trigger Component
• Add Component à Event Trigger
• Now when playing reticle changes shape over cube
116. Add Event Trigger Type
• We want something to happen when gaze at object
• Change object colour when person looking at it
• Add new event type to event trigger component
• Hit Add New Event Type button
• Add Pointer Enter and Point Exit events
• Need to add function scripts to events
117. 5. Create Gaze Input Script
• Create a new Script
• Cube à Add Component à New Script
• Name script GazeInput
• Edit Script – adding functions for changing object colour
119. 6. Connect Script Functions
• In event trigger drag Cube into object form
• In Pointer Enter add HighlightColour Function
• Gaze Input à HighlightColour()
• In Pointer Exit add ResetColour Function
• Gaze Input à ResetColour()
120. 7. Test Scene
• When gaze falls on Cube, Cube highlights white
• Resets to original colour when gaze moves off
123. Key Steps
1. Create simple scene
2. Add Character Controller to scene
3. Creating walk script
4. Connect Character Controller to script
5. Testing scene
124. 1. Create Simple Scene
• Create new scene
• Add GvrEditorEmulator to hierarchy
• Prefabs à GvrEditorEmulator
• Add plane and objects
125. 2. Add Character Controller to Scene
• Add empty Body object
• Add camera to Body
• Add Character Controller to Body
• Add Component à Character Controller
• Move camera to top of Character Controller
126. 3. Create Walking Script
• Want to toggle walking on when user looks down
• When looking down more than 30 degrees
• Move forward in the direction the user is looking
• Move the Body object
• Add script to Main Camera
• Add component à New Script
132. What you will learn
• Introduction to Vuforia
• Platform and features
• How to install/set-up Vuforia
• Vuforia Basics
• Marker Tracking, Object tracking
• Deploying to Mobile Device
• Android, iOS
134. Vuforia Overview
• Platform for Mobile Computer Vision
• https://www.qualcomm.com/products/vuforia
• Released by Qualcomm in 2010, acquired by PTC 2015
• Used by over 200K developers, >20K applications
• Main Features:
• Recognition
• Image, text, object recognition
• Tracking
• Image, marker, scene, object
135. Vuforia Provides
• Android
• iOS
• Unity Extension
Device SDK
• Target Management System
• App Development Guide
• Vuforia Web Services
Tools & Services
• Dedicated technical support
engineers
• Thousands of posts
Support Forum
144. Key Steps
1. Enable Vuforia in project
2. Load AR Camera
3. Configure camera
4. Load Tracking Image
5. Add 3D model
6. Test AR Scene
145. 1. Enable Vuforia in project
• If checked the Vuforia AR box on Unity install:
• just check Vuforia Augmented Reality in player settings
• Build Settings à Player Settings à XR Settings
• If didn’t install, need to download and install package
• See https://developer.vuforia.com/downloads/sdk
• Download vuforia-unity-6-2-10.unitypackage
146. 2. Add AR Camera to Scene
• Add Vuforia camera object
• Game Object à Vuforia à AR Camera
• Delete Main Camera object
147. 3. Configure Camera
• Need to add Vuforia License key
• Open AR Camera inspection
• Open Vuforia Configuration
148. Create Vuforia Developer Account
• Need to create a Vuforia Developer account
• https://developer.vuforia.com/
• Register for account
149. License Manager Portal
• Login to Vuforia account
• Got to license manager
• Develop à License Manager
158. Google Design Guidelines
• Google’s Guidelines for good VR experiences:
• Physiological Considerations
• Interactive Patterns
• Setup
• Controls
• Feedback
• Display Reticle
• From http://www.google.com/design/spec-vr/designing-
for-google-cardboard/a-new-dimension.html
159. Physiological Considerations
• Factors to Consider
• Head tracking
• User control of movement
• Use constant velocity
• Grounding with fixed objects
• Brightness changes
160. Interactive Patterns - Setup
• Setup factors to consider:
• Entering and exiting
• Headset adaptation
• Full Screen mode
• API calls
• Indicating VR apps
162. Interactive Patterns - Feedback
• Use audio and haptic feedback
• Reduce visual overload
• Audio alerts
• 3D spatial sound
• Phone vibrations
163. Interactive Patterns - Display Reticle
• Easier for users to target objects with a display reticle
• Can display reticle only when near target object
• Highlight objects (e.g. with light source) that user can target
164. Cardboard Design Lab Application
• Use Cardboard Design Lab app to explore design ideas
166. Many Areas for AR/VR Research
• Display
• Wide field of view, new display technologies
• Tracking
• Precise tracking, wide area
• Interaction
• Natural gesture interaction, human factors
• Authoring Tools
• Content capture, authoring
• Applications
• Collaboration, large scale
167. Bare Hands
• Using computer vision to track bare hand input
• Creates compelling sense of Presence, natural interaction
• Challenges need to be solved
• Not having sense of touch
• Line of sight required to sensor
• Fatigue from holding hands in front of sensor
169. Eye Tracking
• Technology
• Shine IR light into eye and look for reflections
• Advantages
• Provides natural hands-free input
• Gaze provides cues as to user attention
• Can be combined with other input technologies
176. Conclusion
• VR and AR have a long history
• Early prototypes from 1960’s onwards
• Having second phase of commercial success
• Projected to grow to over $120B market by 2020
• Mostly mobile AR/VR in near term
• It is easier than ever before to develop AR/VR apps
• Multiple easy to use tools for non-programmers
• Powerful developer tools for programmers
• There are many opportunities for research
• Collaboration, interaction, navigation, etc
178. Google Poly for 3D models
• https://poly.google.com/
• Repository of user contributed low polygon objects
• Free to download for AR/VR applications
179. Resources
• Excellent book
• 3D User Interfaces: Theory and Practice
• Doug Bowman, Ernst Kruijff, Joseph, LaViola, Ivan Poupyrev
• Great Website
• http://www.uxofvr.com/
• 3D UI research at Virginia Tech.
• research.cs.vt.edu/3di/
180. Other Books
• Augmented Reality for Developers: Build
practical augmented reality applications
with Unity, ARCore, ARKit, and Vuforia
• Jonathan Linowes, Krystian Babilinski
• Unity Virtual Reality Projects
• Jonathan Linowes
• Holistic Game Development with Unity: An
All-in-One Guide to Implementing Game
Mechanics, Art, Design and Programming
• Penny de Byl
181. UX of VR Website - www.uxofvr.com
• Many examples of great interaction techniques
• Videos, books, articles, slides, code, etc..
182. Useful VR Resources
• Unity VR Tutorials
• https://unity3d.com/learn/tutorials/s/virtual-reality
• Google Cardboard main page
• https://vr.google.com/cardboard/
• Developer Website
• https://vr.google.com/cardboard/developers/
• Building a VR app for Cardboard (Old SDK)
• http://www.sitepoint.com/building-a-google-cardboard-vr-app-in-unity/
• Moving in VR space (Old SDK)
• http://www.instructables.com/id/Prototyping-Interactive-Environments-
in-Virtual-Re/