The document provides slides from a presentation given by Joe Sondow of Netflix about how Netflix uses Grails and AWS to deploy its services. Key points include:
- Netflix uses an internal tool called Asgard to manage deployments to AWS. Asgard was open sourced in 2012.
- Netflix services are deployed as microservices across many AWS accounts and regions. Asgard helps manage this complex cloud infrastructure.
- Custom domain models and conventions are used instead of databases to represent cloud resources and their relationships, avoiding inconsistencies.
- Techniques like blue/green deployments, auto-rollback, and caching of cloud metadata help Netflix deploy changes safely at scale across its global infrastructure.
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
Grails App that Deploys Netflix to the Cloud
1. // Feed your brain GR8Conf US 2012
gr8.technologies.each{ July 30-31
yourBrain << it Minneapolis, MN, US
}
The Grails App that Deploys Netflix to the Cloud
Extended Edition Slides
Joe Sondow, Netflix
@joesondow #gr8conf
14. Why am I here?
Sell you something
Discuss business plans
15. Why am I here?
Sell you something
Discuss business plans
Answer technical questions
16. Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
17. Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Give back to community
18. Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Give back to community
Steal your engineers
35. Joe Sondow Clay McCoy Jason Gritman
@joesondow @claymccoy @jgritman
Asgard Team
July 2012
36.
37. With more than 25 million streaming members in the
United States, Canada, Latin America, the United
Kingdom and Ireland, Netflix, Inc. (NASDAQ: NFLX) is
the world's leading internet subscription service for
enjoying movies and TV programs.
(from http://ir.netflix.com)
78. Inventing the Application
Problem:
Application is not an Amazon concept
Solution:
Create an Application domain in SimpleDB
Enforce naming conventions on Amazon objects
83. Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
84. Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Monitor results
85. Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Monitor results
Revert traffic quickly
95. Inventing the Cluster
Problem:
Two ASGs with one function but different names
Solution:
Append version number in reserved format
Parse ASG name to determine long-term “cluster”
96. Inventing the Cluster
Instead of keeping a database in sync, use naming
conventions to store the source in truth in Amazon’s API
api Application
api-usprod Cluster
api-usprod-v007 Auto Scaling Group
api-usprod-v008 Auto Scaling Group
110. Open Source, Closed Config
Asgard for
Netflix is
configured to
use company-
specific
extension
points such
as standard
utility links
for instances
111. Open Source, Closed Config
Out-of-the-box Asgard installation
has no instance utility links
112. Open Source, Closed Config
Netflix specific $ASGARD_HOME/Config.groovy
link {
// Avoid GStrings here because these Strings are stored dynamic templates for arbitrary server names.
String logUrlStart = 'http://${server}:7777'
String configUrlStart = 'http://${server}:9999/AdminConfig'
instanceLinkGroupingsToLinkTemplateLists = [
'Logs': [
new TextLinkTemplate(logUrlStart + '/Admin/list?view=tomcat/catalina.out', 'catalina.out'),
new TextLinkTemplate(logUrlStart + '/Admin/list', 'Log File Archive'),
new TextLinkTemplate(logUrlStart + '/Admin/threaddumps', 'Thread Dumps'),
new TextLinkTemplate(logUrlStart + '/AdminProxy', 'Admin Proxy Info'),
new TextLinkTemplate(logUrlStart + '/AdminStatus', 'Admin Proxy Status'),
new TextLinkTemplate(logUrlStart + '/GC/index', 'GC Visualization')
],
'Netflix Configuration': [
new TextLinkTemplate(configUrlStart + '/prop.html', 'NetflixConfiguration Properties Console'),
new TextLinkTemplate(configUrlStart + '/libs.html', 'Libraries Console'),
new TextLinkTemplate(configUrlStart + '/machineProps', 'Machine Readable Properties'),
new TextLinkTemplate(configUrlStart + '/webapp/META-INF/MANIFEST.MF', 'Manifest'),
]
]
}
113. Open Source, Closed Config
grails-app/conf/Config.groovy
references external configuration file
~/.asgard/Config.groovy
https://github.com/Netflix/asgard/blob/master/grails-app/conf/Config.groovy
asgardHome = System.getenv('ASGARD_HOME') ?: System.getProperty('ASGARD_HOME') ?:
"${System.getProperty('user.home')}/.asgard"
// Locations to search for config files that get merged into the main config.
// Config files can either be Java properties files or ConfigSlurper scripts.
grails.config.locations = [
"file:${asgardHome}/Config.groovy",
'classpath:sourceVersion.properties'
]
114. Open Source, Closed Config
External Config.groovy also hold the AWS account
credentials, or references for finding them.
grails {
awsAccounts=["178123456789"]
awsAccountNames=["178123456789":"prod"]
}
secret {
accessId="AKIAILAND0E3TMSJA8BA"
secretKey="Od0AMY/dEC3P0/wZQjpmRtm2MVZc9330epDk+pwm"
}
cloud {
accountName="prod"
publicResourceAccounts=["amazon"]
}
150. Visual Language for the Cloud
The indicators for the
current AWS region are
too easy to miss.
151. Visual Language for the Cloud
The indicators for the
current AWS region are
too easy to miss.
152. Visual Language for the Cloud
These availability zones
are important to
recognize at a glance
but their names look
similar, and they appear
on many screens.
153. Visual Language for the Cloud
These availability zones
are important to
recognize at a glance
but their names look
similar, and they appear
on many screens.