Dianne Marsh, Director of Engineering at Netflix, discusses Netflix's DevOps practices for managing their large and growing global ecosystem. Key aspects include building a blameless culture where developers are responsible for operations, extensive automation using tools like Spinnaker and Atlas, and chaos engineering practices like Chaos Monkey to test system reliability. Netflix also leverages machine learning for tasks like anomaly detection and automated canary analysis to improve operations.

1. Dianne Marsh
Director of Engineering
@dmarsh

2. DevOps
Photo Photo Credit: https://www.facebook.com/theprincessbride/photos_stream

3. DevOps
in
Three
Acts

4. Driven
by
Scale

5. Empowered
by
Culture

6. Supported
by
Tools

7. Approaching
Global
Reach
October - Spain, Portugal, Italy
Early 2016 - Korea, Taiwan, Singapore, Hong Kong
65m members à 100m
~60 counties à 200

8. Ne=lix
ecosystem
• 100s
of
microservices
• 1000s
of
daily
producBon
changes
• 10,000s
of
instances
• 100,000s
of
customer
interacBons/minute
• 1,000,000s
of
customers
• 1,000,000,000s
of
metrics
• 10,000,000,000
hours
of
streamed

9. Yet
…
• 10s
of
OperaBons
Engineers
• No
NOC

10. You
Build
It,
You
Run
It

11. Outages
24/7

12. • Developers
• CriBcal
OperaBons/Reliability
Engineering
team
(CORE)
• Crisis
Response
Manager

14. “Get
rid
of
the
safeguards.
Enable
the
most
knowledgeable
people
to
do
their
job
effecBvely.”

15. Blameless
Culture

16. Produc4on
Ready
• IdenBfy
criBcal
services
• Provide
context,
assistance
• Keep
number
small

17. Conformity
Monkey
IdenBfy
best
pracBces
NoBfy
service
owners

18. AutomaBon
and
Tools

19. It’s
Complicated
…

21. Common
RunBme
Services
and
Libraries
Eureka
Ribbon
Hystrix
Zuul

22. Hystrix:
Automate
Recovery

23. Delivery
Tools
Aminator
Spinnaker

24. • Cloud Management
• Delivery Engine
• Automation Platform

25. Global
Cloud
Management

26. Delivery
Pipelines

27. Automated
Global
Delivery

28. Insight
Atlas
Edda
Vector

29. Atlas:
Telemetry
Pla=orm

30. Insight

31. Insight
(Dashboards)

32. What
did
you
expect?

33. Been
Thro_led?

34. Performance
Monitoring

35. Vector

36. • DES on time series
data
• Predict the future
based on history
• Favor recent history
• Threshold-based alerts
• 6-8 minute delay
Anomaly Detection
Alert!

38. Finer Granularity, Shorter Time Windows

39. Ensemble
Learning

40. Median Absolute Deviation
IQR
Least Squares
HDI
Voting

41. Alert
Sooner
Alert!
From 6-8 minutes to < 1 minute

42. AcBon
was
an
Alert

43. Ge`ng
the
Humans
Out
of
the
EquaBon
is
BETTER

44. Outlier Detection & Remediation

45. Kepler
• Unsupervised
machine
learning
• Density-­‐based
clustering
algorithm
• AcBons
– Email,
page
– OOS,
detach,
terminate

46. An
ounce
of
prevenBon…

47. Old Version (v1.0)
New Version
(v1.1)
Load BalancerCustomers
100 Servers
5 Servers
95%
5%
Metrics
Canary
Release
Process

48. Old Version (v1.0)
New Version
(v1.1)
Load BalancerCustomers
0 Servers
100 Servers
100%
Metrics
Canary
Release
Process

49. Automated
Canary
Analysis
Define
• Metrics
• A
threshold
Every
n
minutes
● Classify
metrics
● Compute
score
● Make
a
decision

50. Chaos
Engineering
the
discipline
of
experimenBng
on
a
distributed
system
in
order
to
build
confidence
in
the
systems
capability
to
withstand
turbulent
condiBons
in
producBon.

51. Cluster A Cluster D
Edge Cluster
Cluster B
Cluster C
Imagine a monkey loose in your data center…

52. Xen
Hypervisor
vulnerability
–
9/25/14
218
out
of
2700+
Cassandra
nodes
rebooted
22
did
not
reboot
successfully
AutomaBon
recovered
those
A State of Xen – Chaos Monkey & Cassandra

53. Device
Service
B
Service
C
Internet
Edge
Zuul
Service
A
ELB
FIT
Fault-Injection Testing (FIT)
• Simulate service failures
• Override by device or account
• % of member traffic

54. Device
Service
B
Service
C
Internet
Edge
Zuul
Service
A
ELB
FIT
Fault-Injection Testing (FIT)
• Simulate service failures
• Override by device or account
• % of member traffic

55. Monkey
–
Single
Instance
Gorilla
–
Availability
Zone
Kong
-­‐
Region
More Chaos

56. US-EastUS-West
AZ1
EU-West
Global Traffic Management

58. Exercise
Regularly

59. DevOps
at
Ne=lix

60. How
do
you
think
about
DevOps?

61. Roll
the
Credits
Ne=lix.github.io
Dianne
Marsh,
Director
of
Engineering
dmarsh@ne=lix.com
@dmarsh