Learn about new patterns, behaviors, and design approaches for touch and gesture interfaces from a practitioners point of view. Learn early lessons from applied knowledge of touch applications, devices, and design methods.

1. Untold Stories of Touch, Gesture, & NUI
Joe Fletcher
Design Manager, Microsoft Surface
3/31/2009 Joe Fletcher MIX09, Las Vegas
1

2. 3/31/2009 Joe Fletcher MIX09, Las Vegas
2

3. Touch is great
Touch is not Touch is a
for specific
good for computing
aspects
everything evolution
New technology has enabled touch
It is not an end all solution It is an addition element for users to
systems, which can enhance certain
interact with their content in new and
activities
unique ways
3/31/2009 Joe Fletcher MIX09, Las Vegas
3

4. 3/31/2009 Joe Fletcher MIX09, Las Vegas
4

5. Microsoft Surface
Microsoft Surface is a revolutionary
computer that responds to natural hand
gestures and real-world objects, creating
unique new ways for people to
collaboratively interact with digital
content.
3/31/2009 Joe Fletcher MIX09, Las Vegas
5

6. Microsoft Surface
3/31/2009 Joe Fletcher MIX09, Las Vegas
6

7. 3/31/2009 Joe Fletcher MIX09, Las Vegas
7

8. Vision Strategy Design
3/31/2009 Joe Fletcher MIX09, Las Vegas
8

9. Surface Massive Multi-touch, Multi-user
Unique
Spot
Windows 7 PC’s, Form Factors, Technology
3/31/2009 Joe Fletcher MIX09, Las Vegas
9

10. Larger Surface Team
3/31/2009 Joe Fletcher MIX09, Las Vegas
10

11. Touch is great for
Touch is not good Touch is an
specific aspects
for everything evolution
3/31/2009 Joe Fletcher MIX09, Las Vegas
11

12. Touch Gesture Manipulation
The act of directly touching your digital Hand motions to take an action Alteration or freeform movement of
content digital content using touch
Natural User Interface
The combination of these and other
elements the create a unique way to
interact with your digital content
3/31/2009 Joe Fletcher MIX09, Las Vegas
12

13. History and Surface Design Touch
Ecology Principles Challenges
3/31/2009 Joe Fletcher MIX09, Las Vegas
13

14. Surface Design
History and Ecology Touch Challenges
Principles
3/31/2009 Joe Fletcher MIX09, Las Vegas
14

15. Why Evolve?
3/31/2009 Joe Fletcher MIX09, Las Vegas
15

16. Command Line Interface (CLI)
3/31/2009 Joe Fletcher MIX09, Las Vegas
16

17. Graphic User Interface (GUI)
3/31/2009 Joe Fletcher MIX09, Las Vegas
17

18. Natural User Interface (NUI)
3/31/2009 Joe Fletcher MIX09, Las Vegas
18

19. Natural User Interface (NUI)
3/31/2009 Joe Fletcher MIX09, Las Vegas
19

20. Natural User Interface (NUI)
3/31/2009 Joe Fletcher MIX09, Las Vegas
20

21. Microsoft Surface and NUI
Microsoft Surface computing uses sensing and display technology to infuse everyday
surfaces with digital content and is comprised of the following unique attributes:
Direct interaction Multi-touch Multi-user Object recognition
3/31/2009 Joe Fletcher MIX09, Las Vegas
21

22. NUI
vs.
Web 2.0
3/31/2009 Joe Fletcher MIX09, Las Vegas
22

23. NUI & Touch
should not be a
novelty
3/31/2009 Joe Fletcher MIX09, Las Vegas
23

24. 3/31/2009 Joe Fletcher MIX09, Las Vegas
24

25. Input Ecology
Keyboard Mouse Stylus Single point Multi point
Good for Good for Good for Good for Good for
Accuracy Moving across the Freeform input Direct object selection Manipulating objects
screen efficiently Social
Con Con
Con Con Con
Indirect Not for input
Limited Low accuracy Not for quick input
Low accuracy
Need large controls Low accuracy
Need large controls
Need of large controls
Support of hands and
Mis-hits from
screen
fingernails of perceived
hit area
3/31/2009 Joe Fletcher MIX09, Las Vegas
25

26. [Partial] Touch Ecology
Multi Touch NUI
Multi point
(screen) (Gesture & Physical/Digital)
(screen)
Reacts to multiple Reacts to multiple
Reacts to direct input
inputs inputs
Orients UI to the Spatial Recognition -
direction of the touch Orients UI to the
direction of the touch
Spatial recognition of
system Object recognition -
Blurring physical and
digital
3/31/2009 Joe Fletcher MIX09, Las Vegas
26

27. Tap is not the new click
≠
3/31/2009 Joe Fletcher MIX09, Las Vegas
27

28. Context, task, and use define the solution
3/31/2009 Joe Fletcher MIX09, Las Vegas
28

29. Context, task, and use define the solution
Touch is not a
solution in search of
a problem
3/31/2009 Joe Fletcher MIX09, Las Vegas
29

30. Touch is great for
Touch is not good Touch is an
specific aspects
for everything evolution
3/31/2009 Joe Fletcher MIX09, Las Vegas
30

31. Surface Design
History and
Touch Challenges
Ecology
Principles
3/31/2009 Joe Fletcher MIX09, Las Vegas
31

32. Create an
inflection point
3/31/2009 Joe Fletcher MIX09, Las Vegas
32

33. The start of almost every
successful technological
innovation is marked by an
inflection point, which defines a
moment of dramatic change.
3/31/2009 Joe Fletcher MIX09, Las Vegas
33

34. Companies spend
90% of their energy
trying to imitate
3/31/2009 Joe Fletcher MIX09, Las Vegas
34

35. Microsoft Surface and principle driven design
Principle Driven Design
3/31/2009 Joe Fletcher MIX09, Las Vegas
35

36. Microsoft Surface and principle driven design
Principle Driven Design
Using history to develop the future
3/31/2009 Joe Fletcher MIX09, Las Vegas
36

37. CLI GUI NUI
Text Graphics Objects
3/31/2009 Joe Fletcher MIX09, Las Vegas
37

38. Recall Recognition Intuition
CLI GUI NUI
Text Graphics Objects
3/31/2009 Joe Fletcher MIX09, Las Vegas
38

39. Directed Exploratory Contextual
Recall Recognition Intuition
CLI GUI NUI
Text Graphics Objects
3/31/2009 Joe Fletcher MIX09, Las Vegas
39

40. High-Low Double Medium Fast Few
Directed Exploratory Contextual
Recall Recognition Intuition
CLI GUI NUI
Text Graphics Objects
3/31/2009 Joe Fletcher MIX09, Las Vegas
40

41. Disconnected Indirect Unmediated
High-Low Double Medium Fast Few
Directed Exploratory Contextual
Recall Recognition Intuition
CLI GUI NUI
Text Graphics Objects
3/31/2009 Joe Fletcher MIX09, Las Vegas
41

42. Static Responsive Evocative
Disconnected Indirect Unmediated
High-Low Double Medium Fast Few
Directed Exploratory Contextual
Recall Recognition Intuition
CLI GUI NUI
Text Graphics Objects
3/31/2009 Joe Fletcher MIX09, Las Vegas
42

43. Principle of Performance Aesthetics Evocative
Unmediated
Principle of Direct Manipulation
Principle of Scaffolding Fast Few
Principle of Contextual Environments Contextual
Principle of the Super Real Intuition
NUI
3/31/2009 Joe Fletcher MIX09, Las Vegas
43

44. Microsoft Surface and principle driven design
Principle Driven Design
3/31/2009 Joe Fletcher MIX09, Las Vegas
44

45. Microsoft Surface and principle driven design
Three Super Principles
3/31/2009 Joe Fletcher MIX09, Las Vegas
45

46. Social Multiple simultaneous users
3/31/2009 Joe Fletcher MIX09, Las Vegas
46

47. 3/31/2009 Joe Fletcher MIX09, Las Vegas
47

48. Social Multiple simultaneous users
Seamless Digital and Physical combined
3/31/2009 Joe Fletcher MIX09, Las Vegas
48

49. Seamless
3/31/2009 Joe Fletcher MIX09, Las Vegas
49

50. Social Multiple simultaneous users
Seamless Digital and Physical combined
Spatial Kinesiology
3/31/2009 Joe Fletcher MIX09, Las Vegas
50

51. 3/31/2009 Joe Fletcher MIX09, Las Vegas
51

52. Overcome
Homeostasis
3/31/2009 Joe Fletcher MIX09, Las Vegas
52

53. Touch is great for
Touch is not good Touch is an
specific aspects
for everything evolution
3/31/2009 Joe Fletcher MIX09, Las Vegas
53

54. History and Surface Design
Touch Challenges
Ecology Principles
3/31/2009 Joe Fletcher MIX09, Las Vegas
54

55. Let’s become
practitioners
3/31/2009 Joe Fletcher MIX09, Las Vegas
55

56. Unique
Spot
PC’s, Form Factors, Technology
3/31/2009 Joe Fletcher MIX09, Las Vegas
56

57. Challenge Form Factor & Technology
3/31/2009 Joe Fletcher MIX09, Las Vegas
57

58. Touch Hardware Ecology
3/31/2009 Joe Fletcher MIX09, Las Vegas
58

59. Touch Hardware Ecology
Stylus
Capacitive Capacitive Infrared Vision system
3/31/2009 Joe Fletcher MIX09, Las Vegas
59

60. Touch Hardware Ecology
Stylus
Capacitive Capacitive Infrared Vision system
Wacom (1)
N-Trig (2-10) N-trig (2-10) HP (2) Surface (52)
3/31/2009 Joe Fletcher MIX09, Las Vegas
60

61. Challenge Patterns
3/31/2009 Joe Fletcher MIX09, Las Vegas
61

62. FireFox
Swipe Left: Go back in history
Swipe Up: Go to the top of the page
3/31/2009 Joe Fletcher MIX09, Las Vegas
62

63. Apple iPhone
Swipe Left: Go Forward in History
Swipe up: Go to the bottom of a page
3/31/2009 Joe Fletcher MIX09, Las Vegas
63

64. Challenge Orientation
3/31/2009 Joe Fletcher MIX09, Las Vegas
64

65. Horizontal Vertical
3/31/2009 Joe Fletcher MIX09, Las Vegas
65

66. Challenge Cardio Effect
3/31/2009 Joe Fletcher MIX09, Las Vegas
66

67. Challenge Paradigms
3/31/2009 Joe Fletcher MIX09, Las Vegas
67

68. To button
or
not to button
3/31/2009 Joe Fletcher MIX09, Las Vegas
68

69. Challenge Getting people to touch
3/31/2009 Joe Fletcher MIX09, Las Vegas
69

70. Exploration Instruct
3/31/2009 Joe Fletcher MIX09, Las Vegas
70

71. Challenge Visualizing Touch
3/31/2009 Joe Fletcher MIX09, Las Vegas
71

72. Is it working?
3/31/2009 Joe Fletcher MIX09, Las Vegas
72

73. Future Where do we go?
3/31/2009 Joe Fletcher MIX09, Las Vegas
73

74. Your effort to remain
what you are is what
limits you
Ghost in the Shell
3/31/2009 Joe Fletcher MIX09, Las Vegas
74

75. Debating the future
of touch is useless
3/31/2009 Joe Fletcher MIX09, Las Vegas
75

76. Creating the future
is more interesting
3/31/2009 Joe Fletcher MIX09, Las Vegas
76

77. Touch is great for
Touch is not good Touch is an
specific aspects
for everything evolution
3/31/2009 Joe Fletcher MIX09, Las Vegas
77

78. Take advantage of Microsoft Surface
Download SDK • Available from the Surface Community site
Workstation Edition
• Learn how to build applications from scratch on Surface
Develop for Surface by viewing videos on Surface.com
• Interested in purchasing a Surface? Email to
Purchase a Surface sbizdesk@microsoft.com
3/31/2009 Joe Fletcher MIX09, Las Vegas
78

79. Future Vision
2019
3/31/2009 Joe Fletcher MIX09, Las Vegas
79

80. 3/31/2009 Joe Fletcher MIX09, Las Vegas
80

81. Las Vegas, Thank you!
Joe Fletcher
joeflet@microsoft.com
Design Manager, Microsoft Surface
3/31/2009 Joe Fletcher MIX09, Las Vegas
81

82. Your feedback is important!