Netflix’s container management platform, Titus, powers critical aspects of the Netflix business, including video streaming, recommendations, machine learning, big data, content encoding, studio technology, internal engineering tools, and other Netflix workloads. Titus offers a convenient model for managing compute resources, enables developers to maintain just their application artifacts, and provides a consistent developer experience from a developer’s laptop to production by leveraging Netflix container-focused engineering tools.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Another Week, Another Million
Containers on Amazon EC2
Andrew Spyker
Software Engineering Manager
Netflix
C M P 3 7 6
Joe Hsieh
Principal Technical Account Manager
Amazon Web Services

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why containers?
Given our VM architecture comprised of …
Amazingly resilient
Microservice driven
Cloud native
CI/CD DevOps enabled
Elastically scalable
Do we really need containers?

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What was missing from our VM environment?
Packaging
• Simple to customize application focused artifacts
• Especially for growth of polyglot environments
• Notably for platforms with OS level dependencies
Local development
• Ability to run applications locally on developer laptops
Simple way to manage compute resources
• Especially for ad hoc batch processing

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titus, Netflix’s container management platform
Scheduling
• Service & batch job lifecycle
• Resource management
Container execution
• AWS Integration
• Netflix Ecosystem Support
Job and Fleet Management
Batch
Resource Management & Optimization
Container Execution
Service

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Titus team
• Design
• Develop
• Operate
• Support

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titus and containers product strategy
• Ordered priority focus on
• Developer velocity
• Reliability
• Cost efficiency
Easy migration from VMs to containers
Easy container integration with VMs and Amazon Services
Focus on just what Netflix needs

8. “Our focus is to leverage EC2 deeply in Titus,
not abstract it away or implement similar
features. We see this as a differentiator of
Titus versus other container management
solutions.”

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mesos
High level architecture
Titus Control Plane
• API
• Scheduling
• Job Lifecycle Control
Fenzo
Titus Agents
User Containers
Docker
Mesos Agent
Netflix System Services
AWS Virtual Machines
Docker Registry
Cassandra
AWS Auto Scaling

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EC2 virtual machine portability
Early on we decided a container MUST …
• Natively integrate with VPC for networking
• Natively integrate with security groups for firewalling
• Work with IAM based Amazon Web Services

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key leverage points

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EC2
GPUs - 10’s of p2.8xlarges
Memory optimized - 100’s of r4.16xlarges
General purpose - 1000’s of m4.16xlarges

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VPC and security groups
EC2 VM
ENI0
(to control plane)
ENI1
SG = w
ENI2
SG = x
ENIn
SG = z
Container 1
SG = w
ENI1 IP1
Container 2
SG = w
ENI1 IP2
Container 3
SG = y
ENI3 IP1
Titus
Container
Mgmt

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IAM based services
EC2 VM
ENI0
Container 1
eth0 ethMD
ENI1
Titus
Metadata
Proxy
Normal
networking 169.254.169.254
Amazon Metadata Service and
Security Token Service (STS)

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titus Host
Instance cryptographic identity
Metatron
Service
User
Container

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
All I really needed to know about
containers, I learned from Titus …

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Choices for Auto Scaling Titus applications
Use the two existing Netflix autoscaling engines we already had
• Pro: Code existed
• Con: Lacking features, we’d have to operate
Write a new one
Look for one from Amazon Web Services

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Choices for Auto Scaling Titus applications
Use the two existing Netflix autoscaling engines we already had
Write a new one
• Pro: Would be specific to our needs
• Con: Would be lacking features, we’d have to operate
Look for one from Amazon Web Services

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Choices for Auto Scaling Titus applications
Use the two existing Netflix autoscaling engines we already had
Write a new one
Look for one from Amazon Web Services
• Pro: Already well understood for VMs, feature-rich
• Con: Only works for VMs

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A true story

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A product manager introduction,
development team interchanges, and
multiple iterations later …

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application Auto Scaling with custom resources

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configuring Auto Scaling in Spinnaker

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titus and Application Auto Scaling integration
User Containers
Control Plane

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titus API call pattern
CreateNetworkInterface Total CreateNetworkInterface Throttled
AttachNetworkInterface Total AttachNetworkInterface Throttled
ModifyNetworkInterfaceAttribute Total ModifyNetworkInterfaceAttribute Throttled
AssignPrivateIpAddresses Total AssignPrivateIpAddresses Throttled

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto Scaling group Auto Scaling group Auto Scaling group
An infrastructure view of applications

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
An infrastructure view of applications
Auto Scaling group
VPC

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
API calls
RunInstances
CreateNetworkInterface
AttachNetworkInterface
AssignPrivateIpAddress
ModifyNetworkInterface

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Netflix regional failover
Kong evacuation of us-east-1
Traffic diverted to other regions
Fail back to us-east-1
Traffic moved back to us-east-1
us-east-1
eu-west-1

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Infrastructure challenge
• Increase capacity during scale up of savior region
• Launch 1000s of containers in seven minutes

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Easy right?
“we reduced time to schedule 30,000
pods onto 1,000 nodes from
8,780 seconds to 587 seconds”

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Easy right?
“we reduced time to schedule 30,000
pods onto 1,000 nodes from
8,780 seconds to 587 seconds”

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titus can do this by …
• Dynamically changeable scheduling behavior
• Fleet wide networking optimizations

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Normal scheduling
VM1
App 1
App 2
ENI 1 App 2
IP1 IP1
VM2
App 1
ENI 1
IP1
VMn
App 1
App 2
ENI 1 App 2
IP1 IP1
Trade-off for reliability

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failover scheduling
VM1
App 1
App 2
ENI 1 App 2
IP1 IP1
VM2
App 1
ENI 1
IP1
VMn
App 1
App 2
ENI 1 App 2
IP1 IP1
App 1
App 1
App 1
App 1
App 1
App 2
App 2
IP2, IP3 IP2, IP3, IP4 IP2, IP3
Trade-off for speed

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
On each host
Change when create and attach ENIs is performed
• Moved this to instance start time
• No longer needed on-demand
Need to burst allocate IP addresses
• Opportunistically batch allocate at container launch time
• Likely if one container was launched more are coming
• Garbage collect unused later

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titus API pattern
ModifyNetworkInterfaceAttribute Total ModifyNetworkInterfaceAttribute Throttled
AssignPrivateIpAddresses Total AssignPrivateIpAddresses Throttled

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results
us-east-1 / prod
containers started per minute
} 7500 Launched
in 5 minutes

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Netflix load balancing

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IP based Application Load Balancing

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configuring EC2 load balancers in Spinnaker

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Titus and Load Balancing integration
User Containers
Control Plane
IP Target
Group

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use cases on Titus
• Netflix API, Node.js Backend UI Scripts
• Machine Learning (GPUs) for personalization
• Encoding and Content use cases
• Netflix Studio use cases
• CDN tracking and planning
• Massively parallel CI system
• Data Pipeline routing and SPaaS
• Big Data platform use cases
Batch
Q4 15
Basic
Services
1Q 16
Production
Services
4Q 16
Customer
Facing
Services
2Q 17

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Q4 2018 container usage
Common
Jobs launched 255K jobs / day
Different applications 1K+ different images
Isolated Titus deployments 7 stacks
Services
Single app cluster size 5K (real), 12K containers (benchmark)
Hosts managed 7K VMs (435,000 CPUs)
Batch
Containers launched 450K / day (750K / day peak)
Hosts managed (autoscaled) 55K VMs / month

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Open Source
Open sourced April 2018
Help other communities by sharing our approach

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current and future work
Advanced CPU Isolation Opportunistic Workloads
Nitro and Bare Metal Instances Next Amazon and Netflix
Partnership

54. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Andrew Spyker
@aspyker
Joe Hsieh

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.