Is your service architecture and engineering velocity constrained by organizational concerns? Does it seem impossible to give priority to key initiatives regardless of intent? Are engineers switching tasks so often that they are just treading water? Are critical projects endlessly backlogged? Has staffing up pushed the limits of your team structure? Navigating through challenges like these can be daunting and solutions fraught with uncertainty. How do you know what, where, when to change. And whatever the answer is today it will most certainly vary over time. Effective organizations evolve, at key inflection points, to support critical business and technical goals. There is not only a strong relationship between organizations and the software they produce (Conway’s Law) but many organizational solutions can be derived from analogs in the technical realm. In other words, we can treat organizational improvement as a refactoring exercise. Over the last 20 years Netflix engineering has proven time and again an ability to adapt and grow, resulting in undisputed dominance over the global internet tv market. In this talk we’ll use Netflix as a case study to illustrate how specific strategies, framed as technical analogs, have been employed to maximize engineering agility, velocity, and impact. These powerful, yet simple strategies and solutions provide a useful blueprint for organizational success.

1. Josh Evans – (Former Netflix) Engineering Leader at Large
June 26, 2017
Refactoring Organizations
A Netflix Study

2. 2009

3. Devices
…

4. Queue Reader UX

5. DVD Foundation

6. Customer Device Netflix Data Center
NCCP
Electronic Delivery
LoadBalancer
Netflix App
Security
Activation
Playback
Platform (NRDP)
UI
XML/RPC
Ticket-based security
Custom responses
DB
HTTP/S
DVD Legacy
DVD Legacy

8. Netflix API

9. Let 1000 flowers bloom!

10. Netflix Data Center
API
Netflix API
LoadBalancer
REST API
JSON schema
HTTP response codes
Oauth
Content
Metadata
Application
HTTP/S

11. Customer Device
Netflix Data Center
API
Proposal
LB
Netflix App
Security
Activation
Playback
Platform (NRDP)
UI
Content
Metadata
NCCP
ED
LB

12. Pros
Separation of concerns
Bandwidth
Increased innovation
Cons
API reliability
Heterogeneous architecture
Lack of domain knowledge
Assessment

13. Neil Hunt, CPO

14. What was the reaction?

15. Concern
Anger

16. Tribalism

17. Conway’s Law

18. If you have four teams working on a compiler you will end
up with a four pass compiler

19. Today’s Premise
Conway’s Law describes dysfunction
We must embrace architecture before organization
Technical analogs drive better organizational solutions

20. Selfless Leadership
Company
Team
You
In that order!

21. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

22. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

23. 1999 – 2009
Ecommerce (DVD  Streaming)
2009 – 2013
Streaming Infrastructure
2013 - 2016
Operations Engineering
2017
Time off, exploring options
Josh Evans – Engineering Leader at Large
@ops_engineering
@

24. Global leader in subscription internet TV
Growing slate of original content
100 million members
190 countries, 10s of languages
1000s of device types
Microservices on AWS
Unique company culture

25. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

26. Why do we refactor?

27. Functionality
Engineering velocity
Functional and operational quality
As we scale!
We refactor to improve or sustain

28. The ability to enhance a system by adding new functionality at
minimal effort
Functional Scalability
The ease with which a system or component can be modified,
added, or removed, to accommodate changing load
Load Scalability

29. The ability for an organization to easily add people and domain
responsibilities in response to increased work and complexity
The ease with which an organization or team can adapt to shifts
in business strategy
Organizational Scalability

30. Common tasks are difficult
Strategic efforts are impractical or impossible
When do we refactor?

31. How do we refactor?

32. Technical Patterns
Object-oriented design
Micro-service architecture
Systems engineering

33. Example: Organizational Polymorphism

34. With the right people
Instead of a culture process adherence
We have a culture of creativity and self discipline,
freedom and responsibility
Netflix Culture

35. You build it
You run it

36. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

37. 2009

38. Devices in Production
…

39. Key Platforms in Progress

40. Anthony Park

41. Surprise!
John Funge

42. Big Picture
Device Ubiquity
Product Innovation
Cloud Migration
Internationalization
Service Reliability

43. How many engineers?
6

45. What would you do?

46. Prioritize & Queue
…
…
Task Queue
Completed Tasks
Thread Pool
… …

47. Prioritize
Service availability
Game consoles
Downloadable apps
CE expansion
Mobile
…
Queue
Audio & subtitles
International support
New codecs
…
Prioritize & Queue

48. Scale Up
Work profile
Roles & throughput
Team structure

49. Manager
Engineer Engineer Engineer Engineer Engineer Test/Ops
Monolithic Team
One leader
Undifferentiated roles
Ad hoc responsibilities

50. Monolithic Decomposition
Distinct modules & services
Workload partitioning
Dependency awareness
Loose coupling

51. Server
NRDP Features
Protocols
Security
Bootstrap
Key Platforms
Device integration
Device launch
Streaming Infrastructure
Device & partner-oriented load balancing

52. On Call Overload

53. Vicious Cycle
Philip was great at
Development
Test infrastructure
Project management
Troubleshooting
Philip Fisher-Ogden

54. You build it
You run it

55. Risks
Burnout
Slow progress on key initiatives
Philip Fisher-Ogden

56. Thread Starvation
… …
Shared
exclusive
resource
High priority/frequency
Other - blocked
Tasks

57. Context Switching
Process 1 Process 2
OS
Interrupt or system call
Save state - pcb1
..
Get state – pcb2
Interrupt or system call
Save state – pcb2
Get state – pcb1
..
Executing
Executing
Idle Executing
Idle
Idle

58. Thread Pool Isolation
Partition pools & locks
Distribute problematic workloads
… …
… …
… …
…
…

59. Organizational Solution
Deepen troubleshooting skills
Distribute escalations
Engineer operations

60. Key Platforms
Device integration
Device launch
Server
NRDP
Protocols
Security
Bootstrap
Insight/Tools
Delivery
Dashboards
Performance
Operational tools
Consolidating Operations Engineering

61. Cloud Migration

62. Rapid iteration v. systematic, long-cycle execution
Cloud v. Product

63. S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Traffic Batch Processes
Heterogeneous
Workloads

64. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Interference

65. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Interference

66. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Interference

67. . . .
Batch
S S S S. . .
DB DB DB DB. . .
. . .
Member Path
Member Path
Member Path
Batch
Batch
Interference
X

68. Batch
S S S S. . .
DB DB DB DB. . .
. . .
Member Path
Member Path
Member Path
Batch
Batch
Partitioning
Online Offline
. . .

69. Silverlight Migration

70. Partitioning & Domain Portability
Streaming Infrastructure Platform Engineering
Engineer as a Library
Ranjit Ranjit

71. Partitioning & Domain Portability
Platform Engineering
Engineer as a Library
Streaming Infrastructure

72. Systems
Cloud migration
Key Platforms
Device integration
Device launch
Server
NRDP
Protocols
Security
Bootstrap
Insight/Tools
Delivery
Dashboards
Performance
Ops tools
Streaming Infrastructure

73. Staffing

74. 6  24

75. Bottleneck

76. Systems
Cloud migration
Viewing history
Viewing sessions
Key Platforms
Device integration
Device launch
Server
NRDP
Protocols
Security
Bootstrap
Insight/Tools
Delivery
Dashboards
Performance
Ops tools
Cloning & Parallel Processing

77. By 2012
cloud migration
Canada, Latin America, UK
massive device expansion
major product improvements
Netflix CDN

78. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

79. IQ
Task-oriented
Logical
Literal
Detached
Autocratic
EQ
Feeling-oriented
Emotional
Social
Empathetic
Democratic
Bimodal Thinking

80. IQ
Design
Evaluation
Implementation
EQ
Inception
Socialization
Overcoming Tribalism

81. Flawed Inception

82. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

83. 2012

84. Customer Device
Netflix Data Center
API
This…
LB
Netflix App
Security
Activation
Playback
Platform (NRDP)
UI
Content
Metadata
NCCP
ED
LB

85. ELB
NCCP
API
…has become this
Zuul

86. ELB
…and this

87. Growing complexity
Duplication of effort
Engineering tax

88. Raising the Stakes
Playback start in 500ms
More UI/Playback scenarios
Faster rate of innovation
Better service reliability

89. Common tasks are difficult
Strategic efforts are impractical or impossible
When do we refactor?

90. If you have four teams working on a compiler you will end
up with a four pass compiler
Conway’s Revenge!
We had two teams and a two-service edge architecture

91. Mature API team
Robust API platform
Strong operational focus
Trust & respect
A Better Foundation
Daniel Jacobson

92. Josh: what’s the right architectural solution?
Peter: do you care about the organizational implications?
Moment of Truth

93. Selfless Leadership

94. Josh: what’s the right architectural solution?
Peter: do you care about the organizational implications?
Moment of Truth
Josh: no – we’ll figure that out later

96. ELB
NCCP
API
Before
Zuul

97. After
Integrated architecture
Distributed functionality
Shared services
Common practices

98. Edge Services
Zuul API Server
Playback
Services
Features Security Data Systems
Platform Insight/Tools
Edge Services
Shared services
Organized around microservices, functionality, shared services

99. Takeaways
Put architecture first
Leverage technical analogs
Know when to use IQ v EQ
Be selfless

100. www.linkedin.com/in/jevansnflx
Where to find me

101. ?
Refactoring Organizations