The 2015-06-15 Operational Insight presentation, without presenter notes (because the way Keynote handles presenter notes makes them dominate the presentation)

1. Operational InsightJune 15, 2015
Roy Rapoport
@royrapoport / linkedin.com/in/royrapoport / rrapoport@netflix.com

2. Oh, The Places
We’ll Go!

4. John Boyd

6. Observe

7. Observe
Orient

8. Observe
Orient
Decide

9. Observe
Orient
Decide
Act

10. Observe
Orient
Decide
Act
OODA

11. Observe
Orient
Decide
Act
OODA
“This approach favors agility over raw power in dealing with human
opponents in any endeavor” - Wikipedia

12. This Is What We
Do

16. OODA KPI

17. OODA KPI
Speed

18. OODA KPI
Speed Effort

19. OODA KPI
Speed Effort Reliability

20. Winning
Speed Effort Reliability

21. Winning
Speed
Effort Reliability

22. Winning
Speed
Effort
Reliability

23. Winning
Speed
Effort
Reliability

24. Implications …
for Observation (aka measurement, telemetry, metrics)

25. Implications …
for Observation (aka measurement, telemetry, metrics)
• Make It Easy

26. Implications …
for Observation (aka measurement, telemetry, metrics)
• Make It Easy
• Make It Scalable

27. Implications …
for Observation (aka measurement, telemetry, metrics)
• Make It Easy
• Make It Scalable
• Make it pluggable

28. Implications …
for Observation (aka measurement, telemetry, metrics)
• Make It Easy
• Make It Scalable
• Make it pluggable
• (Eventually) Ruthlessly Cull

29. Implications …
for Observation (aka measurement, telemetry, metrics)
• Make It Easy
• Make It Scalable
• Make it pluggable
• (Eventually) Ruthlessly Cull
“What decision will this help me make?”

30. A Joke

32. 52
48

33. % of servers in major region
with an even IP address

34. Implications …
for Orientation (aka graphing, visualization)

35. Implications …
for Orientation (aka graphing, visualization)
• First-class product

36. Implications …
for Orientation (aka graphing, visualization)
• First-class product
• Different decisions require different viz

37. Implications …
for Orientation (aka graphing, visualization)
• First-class product
• Different decisions require different viz
• Low cognitive load better than

38. Implications …
for Orientation (aka graphing, visualization)
• First-class product
• Different decisions require different viz
• Low cognitive load better than
• High refresh rates

39. Implications …
for Orientation (aka graphing, visualization)
• First-class product
• Different decisions require different viz
• Low cognitive load better than
• High refresh rates
• Deep data density

40. Better Like This …

41. Or Better Like That …

42. Implications …
for Decisions (aka alerting, real-time analytics, etc)

43. Implications …
for Decisions (aka alerting, real-time analytics, etc)
• You already have (some of) this

44. Implications …
for Decisions (aka alerting, real-time analytics, etc)
• You already have (some of) this
• Incremental improvement

45. Implications …
for Decisions (aka alerting, real-time analytics, etc)
• You already have (some of) this
• Incremental improvement
• Sky’s the limit

46. Implications …
for Decisions (aka alerting, real-time analytics, etc)
• You already have (some of) this
• Incremental improvement
• Sky’s the limit
• For benefits

47. Implications …
for Decisions (aka alerting, real-time analytics, etc)
• You already have (some of) this
• Incremental improvement
• Sky’s the limit
• For benefits
• For cost

48. Implications …
for Action

49. Implications …
for Action
1. Humans beat bureaucracy

50. Implications …
for Action
1. Humans beat bureaucracy
2. Machines beat humans

51. Implications …
for Action
1. Humans beat bureaucracy
2. Machines beat humans
3. Repeatability beats one-offs

52. Implications …
for Action
1. Humans beat bureaucracy
2. Machines beat humans
3. Repeatability beats one-offs
Repeatable machine processes TROUNCE one-off human
bureaucracy

53. Implications …
for Action
1. Humans beat bureaucracy
2. Machines beat humans
3. Repeatability beats one-offs
4. Start with humans
Repeatable machine processes TROUNCE one-off human
bureaucracy

54. Implications …
for Action
1. Humans beat bureaucracy
2. Machines beat humans
3. Repeatability beats one-offs
4. Start with humans
5. If IFTTT, deprecate humans
Repeatable machine processes TROUNCE one-off human
bureaucracy

55. Decision:
Do I Have Enough
Instances?

58. Decision:
Is My Canary Good?

59. 25

60. Been there.
Done that.
Manually.Artisanally.
25

61. Been there.
• Started in the Data Center
Done that.
Manually.Artisanally.
25

62. Been there.
• Started in the Data Center
• Manual, dashboard-driven
Done that.
Manually.Artisanally.
25

63. Been there.
Done that.
Manually.
26
CPURequestsErrors

64. Been there.
Done that.
Manually.
27

65. Been there.
Done that.
Manually.
• Context vs Precision
27

66. Been there.
Done that.
Manually.
• Context vs Precision
• No …
27

67. Been there.
Done that.
Manually.
• Context vs Precision
• No …
• Repeatability
27

68. Been there.
Done that.
Manually.
• Context vs Precision
• No …
• Repeatability
• Trending
27

69. Been there.
Done that.
Manually.
• Context vs Precision
• No …
• Repeatability
• Trending
• Manual effort is manual
27

70. So Now What?
28

71. So Now What?
• Automate Analysis
28

72. So Now What?
• Automate Analysis
• Took Some Effort
28

73. So Now What?
• Automate Analysis
• Took Some Effort
• Approach and analytics
28

74. So Now What?
• Automate Analysis
• Took Some Effort
• Approach and analytics
• Presentation matters
28

75. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
Pretty Pictures
29

76. Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
1 server
@ 1.0.2
Automated
Canary
Analysis
Pretty Pictures
29

77. 10 servers
@ 1.0.2
Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
Pretty Pictures
29

78. 1000
servers
@ 1.0.2
Version
Control
System
1000
servers
@ 1.0.1
Customers
Build &
Deployment
System
Automated
Canary
Analysis
Pretty Pictures
29

79. Versi
on
1000
servers
@ 1.0.1
Custome
Build &
Deployment
Automat
ed
1000
servers
@ 1.0.2
Pretty Pictures
30
Version
Control
System
Build &
Deployment
System
Automated
Canary
Analysis
Customers

80. Versi
on
Custome
Build &
Deployment
Automat
ed
1000
servers
@ 1.0.2
Pretty Pictures
30
Version
Control
System
Build &
Deployment
System
Automated
Canary
Analysis
Customers

81. Versi
on
1000
servers
@ 1.0.1
Custome
Build &
Deployment
Automat
ed
1000
servers
@ 1.0.2
Pretty Pictures
31
Version
Control
System
Build &
Deployment
System
Automated
Canary
Analysis

82. Versi
on
1000
servers
@ 1.0.1
Custome
Build &
Deployment
Automat
ed
1000
servers
@ 1.0.2
Pretty Pictures
31
Version
Control
System
Build &
Deployment
System
Automated
Canary
Analysis

83. Just The Stats
4-Week View

84. Just The Stats
4-Week View
6309 canary analysis cycles

85. Just The Stats
4-Week View
6309 canary analysis cycles
16% canaries failed

86. Decision:
Do I Have an Outlier?

87. Outlier Detection

88. Would You Like to Play a
Game?

89. Spot the Outlier

93. The
Outlier Is
“A”

94. Just The Stats
4-Week View

95. Just The Stats
4-Week View
739 Server Terminations

96. In a Nutshell
Observe
Orient
Decide
Act

97. In a Nutshell
Observe
Orient
Decide
Act
Need This First
http://bit.ly/nflx-atlas-2013
http://metrics20.org

98. In a Nutshell
Observe
Orient
Decide
Act
Need This First
http://bit.ly/nflx-atlas-2013
http://metrics20.org
Understand the decision
http://bit.ly/nflx-qcon-aca-2014

99. In a Nutshell
Observe
Orient
Decide
Act
Need This First
http://bit.ly/nflx-atlas-2013
http://metrics20.org
Understand the decision
http://bit.ly/nflx-qcon-aca-2014
Make it easier for humans

100. In a Nutshell
Observe
Orient
Decide
Act
Need This First
http://bit.ly/nflx-atlas-2013
http://metrics20.org
Understand the decision
http://bit.ly/nflx-qcon-aca-2014
Make it easier for humans
Make machines
do it

101. In a Nutshell
Observe
Orient
Decide
Act
Need This First
http://bit.ly/nflx-atlas-2013
http://metrics20.org
Understand the decision
http://bit.ly/nflx-qcon-aca-2014
Make it easier for humans
Make machines
do it
Higher speed
Lower effort
Higher reliability

102. Questions, Attributions, Feedback
42

103. Questions, Attributions, Feedback
@royrapoport
rsr@netflix.com
linkedin.com/in/royrapoport
?42