Natural User Interface Workshop

NATURAL UI
Innovation & Advanced Technology Group
Thursday, October 25, 12

2005
Keyboard
Mouse
until
Screen

2012
Motion
Gyroscope
GPS
Proximity Sensor
Optical LensTouch Screen
Microphone & Speaker
Biometrics

Transition
Command Line
Interface
(CLI)
Graphical User
Interface
(GUI)
Natural User
Interface
(NUI)
Time
Interactionswithtechnology
We are here

Multi-Touch
Body Gestures &
Movement
Body Sensors
(i.e. Heart rate)
Natural = Human

Presence
Emotion Tracking
Eye Tracking &
Control
Natural = Human

Voice Control
Audible Feedback
Natural = Human
Haptics

A NUI is a user interface
that is effectively invisible.

NUI NOW
Voice & Sound
Sight
Touch & Gesture
Location & Motion
Biometrics
Opportunities

Voice &
Sound
Voice &
Sound
Audible Feedback
Voice Control

Voice &
Sound
Voice control
Use of language creates an efﬁcient way to access a large set of
commands.
Conversational agent
Users can interact with a system without prior knowledge of the system’s
commands.
Audio security
Voice signature shifts the burden of identiﬁcation away from the user to
the system.
New data channels
Utilizing audio channels opens up additional ways to transmit data.
Thursday,August 23, 12

Unlock your mobile phone or computer
using your voice.
Voice &
Sound

Voice &
Sound
Ask Siri any question,
she’ll try to help you
answer it.

1.If you don’t have an iPhone 4S, buddy up or use
the phone on the table
2.Launch Siri, by press and holding the home key
3.Tap the Siri icon
4.Ask Siri a Question
• What's on my calendar for Friday?
• Wake me up tomorrow at 7am
• How do I get home?
• Note that I spent $12 on lunch
• How many days until Christmas?
• How much wood would a woodchuck chuck?
Voice &
Sound

Sight
Sight
Visualization
Eye Tracking

Augmented reality
Additional intelligence layered on top of natural life.
Visual security
Shifts the burden of identiﬁcation away from the user to the system.
Emotion tracking
The system proactively reads the user and can adapt in a natural way.
Sight

Explore the world around using with help
fromYelp and augmented reality
Sight

Real-time translate Spanish to English using
your mobile device and camera.
Sight

1.Hold the phone in portrait mode
2.Try Word Lens Demo or the phone on the table
3.Launch Word Lens
4.Hold word lens up to the sample text
5.Watch word lens translate from English to Spanish
Sight

Automatically measure an entire room’s
dimension and generate a blue print.
Sight

Sight

Touch & Gesture
Touch &
Gestures
Body Gestures
Multi-Touch

Touch &
Gestures
Powerful collaboration
Users share control over the interface in a way not possible with single input
devices. (mouse, keyboard, remote control, etc)
Utilizing three dimensions
Traditional inputs track X/Y coordinates. NUI’s create a more direct
connection to content by taking advantage of depth in movement.
Affordances
The digital representation of a physical object (book) allows the user to take
advantage of all the things a user already knows about how to use the object.

A powerful and
intuitive gesture
To Do list app.
Touch &
Gestures

A gesture based
browser. Make a G
go to Google.com
Touch &
Gestures

Touch &
Gestures
1.Launch Dolphin Browser
2.Tap the on the bottom tool bar
3.Draw a Capital G (to go to Google)
5.Draw a Capital N (open a new browser tab)
2.Draw a Capital M (to go to Mint)

Using the camera
users can swipe to
turn pages.Avoiding
sticky hands on the
screen.
Touch &
Gestures

Using the camera to capture body size to
shop for clothing online.
Touch &
Gestures

Natural = Human

Location &
Motion
Location &
Motion
Movement
Presence

Location &
Motion
Physical controls
Physical inputs (weight, tile, rotation) are translated into digital outputs.
Relational awareness
Devices know where you are in relation to other people and act as an agent
for the user.
Monitor physical
movements
This additional source of information can be used to help systems (and users)
understand more about themselves in order to learn and adapt.

Perform CPR the correct
way to save a life.
Location &
Motion

1.Launch PocketCPR
2.Tap Start CPR Hands-Only
3.Tap right arrow to advance tutorial
4.Practice CPR using pillow
Location &
Motion

Users can track their jump height and
energy levels.
Location &
Motion

Biometrics
Biometrics
Body Sensors
Haptics

Biometrics
Haptics
Digitial feedback can mimic physical sensations, creating a wide
range of experiences from a single device.
Body monitoring
Systems can start to “read” body language as an input.
Gestures can move beyond the hand and ﬁnger.
Biometric security
Shifts the burden of identiﬁcation away from the user to the
system. Users don’t need to remember their password, they simply

A revolutionary way to
check your heart rate
using the phone’s camera
Biometrics

1.Launch Instant Heart Rate **
2.Select Get Started
3.Cover entire camera lens using your ﬁnger tip
Don’t press too hard.
4.Wait until app reads heart rate
Biometrics

All your sleep information
monitored and tracked
using your iPhone
Biometrics

Emotiv lets you control an interface
using just a thought.
Biometrics

Appendix

The power of three dimensional
gesture. Going beyond touch.
CTOF2012
Next
Gen Leap Motion

Augmented reality you always
wear. (Google I/O demo)
Next
Gen Google Goggles

Create a GUI on any surface and
interact immediately.
Next
Gen Omni Touch

Spatially aware display that supports
intuitive interaction with volumetric data
Next
Gen Media Lab T(ether)

Natural User Interface Workshop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Natural User Interface Workshop

Similar to Natural User Interface Workshop (20)

More from Stephen Gay

More from Stephen Gay (12)

Recently uploaded

Recently uploaded (20)

Natural User Interface Workshop