This panel will take a detailed look at the needs for a complex production pipeline for AR apps that include rich character-driven stories, which are expected to increase as Augmented Reality becomes more accepted and grows into a wider range of diverse application areas. Borrowing from traditional animation, effects and virtual reality pipelines, the team from The Augmented Traveler will share their experiences with creating a pipeline that can support narrative-driven AR. Supporting this pipeline are specialized Artificial Intelligence techniques for character navigation in a combined real and virtual world simultaneously, and for dialogue / narrative enhancement based on user interaction and current surroundings.
3. Introduction
Our work at Augmented Traveler addresses the
increasing popularity of AR apps that include rich
character-driven stories
A complex pipeline is necessary to manage the
production of these apps
This panel will discuss the experience of creating a
pipeline for PastPort™, a time-traveling interactive
history AR application
4. Pipeline Overview
Creative Maya Unity Vuforia Deployment
Script development
Location scouting
Storyboards/Concept art
Cast voice actors
Audio (ambient, FX, &
music)
UX design/Experience
flow
Scene building
Character
behaviors &
pathing
Prop behaviors &
pathing
UI/UX for scene
navigation
Design/Build models
(characters & props)
Create textures
Rig models
Animate models
Export to Unity
Determine AR targets
Overlays of virtual objects
Create AR behaviors
Tie in to social media
backend
Customize for select
partners (branding)
Determine distribution
with partners
Launch subscription
services
Launch customized
PastPort™ services
5. AR Design Targets, markers, SLAM AR integration
Story concept “It’s 1499 in Venice…” Script Audio recordings
Scratch & final Narration & dialogue
Ambient & effects Music
Conceptual Assets
Creative
Character designs & costumes 3D models Rigged Models
Concept to Final Model Animation
Action design Storyboards Model Choreography
Characters/Props Path planning
UIX Design Buttons, widgets, look & flow UI integration
Scene design Stand-in scene model
Production
6. Maya
Model Texture Facial BlendShapes
for characters: Mixamo Auto Rig
for props: rig by hand
Character Avatar Export char.fbx Unity
Actor Dialogue .wav
Shot List / Storyboard
Animation Sequence
Baked Animation char@anim.fbx Unity
bone rotations/blendShapes
strip controls
Rig Leonardo
Salai
Leonardo’s Giant Crossbow
Rigging
Animation
7. Unity
Mecanim
avatars
animation controller
animation sequences
animation events
Scripts
dialogueSync.cs
screenplayController.cs
navigateTargets.cs
eventInteraction.cs
.fbxMaya
meshes
joints
textures
blendShapes
animations
Other tools
Google Drive
Trello
Collaborate
Playmaker
Skype
Open Broadcaster
World Clock
Scenes
environment
characters
props
camera
materials
physics
navmesh
audio sources
UI
Vuforia
Screenplay
Unity
8. 3D Virtual Objects [Maya / Unity]
Leonardo
Salai
Crossbow
Vuforia Unity plugin package [Download] Import package to Unity
AR Components Intermediate Steps
Vuforia
AR Camera
[in Vuforia Assets]
Target (marker) assets
2D texture: Piazza [Creative]
3D model target (Maya/Unity)
Deployment
Add developer license
[from Vuforia License Manager]
Add AR Camera to Unity scene
Configure webcam for editor debugging
Vuforia Target Manager
Upload images
Download as Vuforia ImageTarget db
Add to Unity
Position Piazza ImageTarget in scene
Connect db to Vuforia ARCamera
Add GameObjects to scene
In hierarchy, set as children of
Piazza ImageTarget
9. Deployment!
User testing & refinement
Customize for select partners (branding)
Determine distribution with partners
Launch subscription services
Launch customized PastPort™ services
Export s master character for a Unity avatar
Rigging allows the animator to focus on the performance
Export each animation sequence to remix the performance in Unity
* Export a master character for a Unity avatar – we have to plan early for all possible animations
* Rigging allows the animator (Charles) to focus on the performance
* Export each animation sequence to remix the performance in Unity – it’s easy to do updates, proxies
* Mixamo is a wonderful tool to standardize rigs, animations, and to share motion
* The Mixamo AutoControlRig script would not complete, so facial animation had to be a separate process
Unity balances three interactions: the physical design (Scene), animation and event control (Mecanim), and programming logic (Scripts)
.fbx is a sure way to publish assets that you mean to publish – one envelope to contain many useful things
Unity has the Asset Store & significant training resources to solve technical problems
Even when using a technical setup, we reference the screenplay and the actors performance, historical research
We have a team scattered around the world (Los Angeles, China, Italy, Dallas) and cloud tools make it possible