As a content delivery network, Fastly operates an edge environment for many large scale web properties and APIs. In order to deal with emerging threats to its network, Fastly needed to develop processes that allowed it to respond effectively to availability and security incidents at scale. The network engineering, SRE, and security teams at Fastly leverage a protocol called “Incident Command” to rapidly engage various teams across the company, and make sure customer properties are protected. Let the Fastly VP of SRE take you to the far side of the edge, and learn more about the challenges a large global network faces and the protocols that we found helped for us.

1. Incident Management At The Edge
Lisa Phillips VP, SRE
@lisaphillips
lisa@fastly.com

3. Incident Types
• Distributed Denial of Service attacks
• Critical security vulnerabilities
• Software bugs
• Capacity concerns
• Upstream network outages
• Datacenter issues
• “Operator Error”
• Third Party service provider events

4. Incident Response Framework
1. Develop definitions of impact
2. Define severity
3. Define response and communication requirements
4. Define post-incident activities

5. Severity levels
Severity App Delivery
Impact
Business
Operations Impact
Scope of Impact
SEV0 Critical Critical All Sites Affected
SEV1 Critical Critical Multiple Sites Affected, or Single Site
unavailable or suffering from severe degradation
SEV2 Major Major Multiple Sites Affected, Single Site
intermittently available or suffering from minor
degradation
SEV3 Minor Minor Single Site or limited customer impact

6. The people
• Engineers are Human
• Right people at the right time engaged
• Randomization is expensive

7. The Situation: No NOC
• Immediate escalation to engineers is required
• Global Customer Service Focused Engineer
• Decentralized on-call and monitoring
• Engineering teams own their own destiny and have control over their alerts
• Empowered to improve

8. Incident Commander
• Coordinate actions across multiple responders
• Alerts and updates stakeholders
• Evaluate the high-level issue and understand its impact
• Consult with team experts on necessary actions
• Calls off or delays other activities
• Clearly indicates when an Incident is resolved and leads follow-up

9. Communication: Wide Collaboration
• Involve all parts of the organization
• Quick communication and collaboration - establish next
steps and regroup time
• Identify quickly the questions to answer, and
communicate effectively to address them
• When in doubt, choose transparency

10. Lessons
• Start with the basics
• Empower your engineers to deal effectively with workload
• Exercise your engineers
• Continuously Improve
• Partner closely with all stakeholders
• Continue to let incidents teach you

11. Thank you
Lisa Phillips VP, SRE
@lisaphillips
lisa@fastly.com