Improve Your Incident Response Process
•
3 j'aime•1,631 vues
The document discusses improving incident response processes. It emphasizes the importance of preparing, executing, and improving the response process through monitoring, alerting, clearly defined roles and responsibilities, practice, and post-incident reviews. The prepare phase involves setting up monitoring, alerts, processes, and practicing the plan. The execute phase is assessing incidents and resolving them. The improve phase is reviewing incidents to learn and improve the process. Regular improvement of the response process helps ensure high reliability and availability.
Recommandé
Recommandé
Contenu connexe
En vedette
En vedette (20)
Similaire à Improve Your Incident Response Process
Similaire à Improve Your Incident Response Process (20)
Plus de Amazon Web Services
Plus de Amazon Web Services (20)
Dernier
Dernier (19)
Improve Your Incident Response Process
- 1. @esigler Eric Sigler, Head of DevOps, PagerDuty Incident Response & Coordination
- 3. @esigler Everyone can improve their Incident Response process
- 4. @esigler Take the time to clearly define your process today
- 5. @esigler Why should organizations invest time improving it?
- 6. @esigler Puppet / DORA “State of DevOps 2016 Report”
- 7. @esigler What is Incident Response?
- 8. @esigler Prepare Execute Improve Incident Response “Outer Loop”
- 12. @esigler “If Engineering at Etsy has a religion, it’s the Church of Graphs. If it moves, we track it.” @esigler
- 13. @esigler Don’t forget the business metrics
- 15. @esigler Setting up an alarm in AWS CloudWatch
- 16. @esigler
- 17. @esigler
- 18. @esigler
- 19. @esigler
- 20. @esigler
- 21. @esigler
- 22. @esigler “I don’t want to get woken up at 3AM.”
- 23. @esigler Scope down your alerts!
- 25. @esigler A problem in Production at 3AM? I’m there with bells on. A problem in Staging at 3AM? Maybe less so.
- 27. @esigler Humans are terrible shell script interpreters, especially at 3AM.
- 31. @esigler … so relate them to the business.
- 32. “We now have more time to focus on other proje Connie-Lynne Villani Senior Manager
- 33. @esigler Configure AWS CloudWatch to integrate with PagerDuty
- 35. @esigler
- 36. @esigler
- 37. @esigler
- 38. @esigler
- 39. @esigler AWS Incidents in PagerDuty
- 40. @esigler AWS Incidents in PagerDuty
- 42. @esigler “Like getting lawn care advice from the superintendent of Augusta National” response.pagerduty.com
- 45. @esigler Consider a volunteer IC schedule
- 47. @esigler Scribe
- 49. @esigler What criteria should you use to launch an Incident Response?
- 50. @esigler Post incident criteria widely. Don’t litigate during a call.
- 52. @esigler Practice your Incident Response plan beforehand
- 53. @esigler Consider injecting failure, and testing “all of the above”.
- 54. @esigler Don’t forget to include the rest of the business in your Incident Coordination
- 56. @esigler Assess & Triage Resolve or Remediate Learn & Review Incident Response “Inner Loop”
- 57. @esigler Assess & Triage Resolve or Remediate Learn & Review
- 58. @esigler Elect a leader (IC) at the beginning of the call
- 59. @esigler How to give a status update
- 60. @esigler Assess & Triage Resolve or Remediate Learn & Review
- 61. @esigler Delegating tasks on a call
- 62. @esigler Don’t forget to check back in.
- 63. @esigler Have a clear mechanism for making decisions.
- 64. @esigler “IC, I think we should do X” “The proposed action is X, is there any strong objection?”
- 66. @esigler Humor is best in context.
- 67. @esigler DT5: Roger that GND: Delta Tug 5, you can go right on bravo DT5: Right on bravo, taxi. (…): Testing, testing. 1-2-3-4. GND: Well, you can count to 4. It’s a step in the right direction. Find another frequency to test on now. (…): Sorry
- 68. @esigler Assess & Triage Resolve or Remediate Learn & Review
- 69. @esigler Capture everything, and call out what’s important now vs. later.
- 70. @esigler Decreasing the scope of a call
- 71. @esigler Capture everything for the postmortem / learning review.
- 73. @esigler “You can’t fire your way to reliability.” Ensure your postmortems are blameless
- 74. @esigler Beware of: Counterfactual Reasoning Normative Language Mechanistic Reasoning
- 75. @esigler Maintain every postmortem in a collection / archive.
- 76. @esigler Review your Incident Response process
- 77. @esigler
- 78. “We’ve handed out responsibility for handling alerts to the teams that know the most about the service. They’re the people who can generally fix things fastest.” Sam Eaton Vice President of Engineering
- 80. @esigler FD: “OK, why don’t, you gotta pass the data for the crew checklist anyway onboard, d MC: “Right” FD: “Don’tcha got a page update? Well why don't we read it up to them and that'll se MC: “Alright.” FD: “Both that mattered as well as what page you want it in the checklist?” MC: “OK.”
- 81. @esigler TELMU: "Flight, TELMU.” FD: "Go TELMU.” TELMU: "We show the LEM overhead hatch is closed, and the heater current looks n FD: "OK." GUIDE: "Flight, Guidance." FD: "Go Guidance" GUIDE: "We've had a hardware restart, I don't know what it was."
- 82. @esigler FD: "GNC, you wanna look at it? See if you've seen a problem" Lovell: "Houston, we've had a problem ..." FD: "Rog, we're copying it CAPCOM, we see a hardware restart" Lovell: "... Main B Bus undervolt" FD: "You see an AC bus undervolt there guidance, er, ah, EECOM?" EECOM: "Negative flight" FD: "I believe the crew reported it." ???: "We got a main B undervolt"
- 83. @esigler EECOM: "OK flight we've got some instrumentation issues ... let me add em up” FD: "Rog" CAPCOM: "OK stand by 13 we're looking at it" EECOM: "We may have had an instrumentation problem flight" FD: "Rog" INCO: "Flight, INCO” FD: "Go INCO” INCO: "We switched to wide beam about the time he had that problem"
- 84. @esigler Haise: "...the voltage is looking good. And we had a pretty large bang associated with FD: "OK" CAPCOM: "Roger, Fred." FD: "INCO, you said you went to wide beam with that?" INCO: "Yes" FD: "Let's see if we can correlate those times get the time when you went to wide-beam INCO: "OK"
- 85. @esigler
- 86. @esigler Challenge: Audit your alarms this week. Are they all immediately human actionable?
- 87. @esigler Puppet / DORA “State of DevOps 2016 Report”
- 88. @esigler You'll sleep better at night
- 89. See You At AWS Summit San Francisco! April 18-19 aws.amazon.com/summits/san-francisco/
- 91. @esigler Eric Sigler, Head of DevOps, PagerDuty Incident Response & Coordination
Notes de l'éditeur
- Howdy! I’m Eric Sigler, Head of DevOps at PagerDuty, and I’m here to talk to y’all today about Incident Response & Coordination.
- First, let’s start with a couple of audience questions. (Who currently uses PagerDuty?) Who here has been on an “outage call” of some kind? Who’s been on an outage call for more than 4 hours? Who’s been on an outage call with over 20 people? Who’s been on a call when the CEO or another high ranking person “swooped” in? These are all examples of … less than ideal incident response behaviors. As an industry, we inherited a lot of baggage and cruft from earlier eras. But I believe…
- … Everyone can improve their Incident Response process. We have modern ways of building stacks, like EC2 & ECS, and modern ways of structuring our applications (Microservices, Lambdas, etc). Organizations that want to be the best need to have modern ways of responding to problems as well.
- And I’m going to call out that every organization should do so, and do so now. An ounce of prevention today is worth, etc, etc. You’ll have an easier, faster time responding to issues, and a better time preventing small issues from cascading into large ones.
- So, as with everything else, there’s a cost here, right? Why should organizations invest in it? What’s the ROI on having a great Incident Coordination process?
- Well, for one thing, as you adopt a lot of these practices, you’ll find that you end up spending less time on unplanned work. Getting back 20% of your engineering time to spend on more features usually sounds pretty good.
- War Rooms, Outage Calls, Incident Response, Incident Command, Incident Coordination, there are lots of different ways of outlining the same set of topics. So, let’s put one together.
- Here’s one way to think about Incident Response that we’ve seen from a large number of our customers across various industries - as a series of loops. There’s an “outer loop”, if you will, that goes like this - prepare (through setting up monitoring, alerting, and processes), then execution (once you’ve actually come upon an incident), then improvement (learning from your incident, and making improvements to the next time) - this all closes out in a constant feedback cycle where one incident, better or worse, influences the next.
- So then, the first part of our outer loop - prepare. This is where I’ll point out that the more effort you spend here, the less impact and easier time you’ll have on the other two items.
- OK, so within prepare, there’s a ton of stuff. (For those of you keeping score, we’ve got lists and loops - I promise to stop before hash tables.) Here’s a few different things we’ll go over.
- First up, monitoring. It’s actually pretty easy - Etsy has it right …
- Log everything. _Everything_. The point here is to make this so cheap to collect, that it costs more NOT to. Common libraries, centralized / shared services, and convenience are key. Business, application, infrastructure, client, deployment pipeline, and so on.
- Order velocity, customer contact rate, these aren’t necessarily the first things you think about hooking up to your monitoring tool of choice, but they’re actually probably the most valuable, as we’ll get to in a bit.
- Now that you’ve got all this data, you need to alert when it’s not right!
- OK, so you’ve probably already seen this a few times today, but I wanted to walk through setting up an alarm in CloudWatch super quick for anyone who hasn’t…
- Go into the AWS Control Panel, select CloudWatch…
- Avoid flappy alarms!
- But don’t do that! Most of this should be in the form of code. CloudFormation, for example, lets you configure CloudWatch alarms programmatically. Preferably, code that’s under the control of your service-owning-teams. Remember - with responsibility comes authority, and they have to be allowed to make their own choices.
- And that brings us to the #1 most common complaint we’ve heard for all time - people don’t want to get alerted! I’ve got a quick solution…
- … don’t have as many!
- First, it should be an alarm that requires immediate attention. By that, I mean it shouldn’t be something that “can wait until tomorrow”. Alert appropriately. Another way of looking at it…
- … is like this. (YMMV on follow-the-sun issues.) There are tools out there that have urgencies and severities, make sure to leverage those where possible. Not all Immediate things are created equal.
- Second, you want to have something that requires a human. If “the first step in the run book” is to run a script to purge /tmp, then just do that automatically, because …
- … because Humans are terrible compilers. Don’t turn them into one, especially if you don’t have to. Lambda functions, autoscaling systems, heck even just cron triggers and touch files are all better than operational toil. The next thing to remember on an alert is …
- If the alert is basically “something happened”, well, that’s not super useful. It tends to very quickly condition people to ignore it, and that’s the worst thing of all to do. The classic version of this is the antipattern…
- … the “everything’s OK alarm”. In the Simpsons, Homer comes up with an alarm that goes off every 3 seconds, to indicate everything is OK. I know that sounds rediculous, but how many times have you asked yourself, “I haven’t gotten an alert in a while, is the monitoring system still working?” - which is a great indicator of this pattern. The last question you should ask when reviewing alerts is …
- Pulling it all together, here’s what should go through your head every time you create something that will try to get your attention. And it turns out that probably has a curious side effect…
- And when you go down this process of distilling your alerts, you’ll find that more often than not, they’ll tend to be related to your business needs. Huh. Guess collecting that business data was important after all.
- Here’s a screenshot of our product for a bit of context. Originally, this area below wasn’t this way - we accepted events from customers, but didn’t do a lot with the data we accepted. As we grew the product, we wanted to provide more value for our customers, and tease out more details. But there was a challenge …
- Next up, now that we have all this awesome data, and are only hearing about it when there’s an issue. We need some process to actually deal with all of it! First, let’s figure out the “who”.
- And here might be a good place to pause and take a 10 minute break. A lot of what I’m going to say is captured here at our open source response documentation - so if you want spoilers, feel free to read ahead.
- You need to define who’s going to do what. There’s a thing called the “National Incident Management System”, that’s used by firefighters and other first responders. If you think about it, this is a group of people who you pick up the phone, dial a number, and within 10 minutes receive several million dollars of hardware and a dozen well trained engineers. There’s probably something to learn here. (These terms are slightly military focused, because that’s where NIMS comes from.)
- And it starts with the “IC”. Not individual contributor, but Incident Commander. This is someone who _isn’t_ doing the work, but is responsible for the meta-process. The mechanical turning of the crank. They rally the team together, dispatch tasks, check in on status, update stakeholders (potentially), make choices if needed, escalate priorities, and finally disband once the incident is over.
- Airbnb, PagerDuty, lots of organizations use a volunteer schedule for their On-Call. It sounds absolutely crazy, I know, but think about it for a second. An IC is going to have to have gravitas, and be able to kick maybe even the CEO off the call. That’s not something you want to force people to do.
- First rule - never have a SPOF.
- Can also act as a bridge between voice and chat. Everyone else is too busy, it’s this person’s job to capture the details between the cracks.
- Aha! That’s you! OK, at least most of you. The point of ALL of this process is to give SME’s the room to solve the problem. At the end of the day, hopefully the majority of your incidents are novel - so you need SMEs focused on understanding, mitigating, and resolving them, instead of dealing with the “meta process” around it.
- Different alerts will require different levels of response, like we talked about before. A lot of organizations use some sort of graduated scale, usually based on Customer Impact. Some organizations won’t decrease their severity level, others will. But the key takeaway in defining your criteria is …
- … Whatever your criteria is, post it widely. Litigating “is this really worth it?” during an incident call is a huge waste of time and resources. Agree to it, and stick with it.
- All right, so we’ve got all that, I’m in New York, so I have to ask, how do you get to Carnegie Hall?
- Game Days, Failure Fridays, Tabletop’ing, all are valid ways of exercising. The point is to work out the kinks in your process when everything is OK, and it’s not 3AM, and revenue isn’t on the line. This give responders “muscle memory”, which becomes incredibly helpful so that you aren’t figuring things out on the fly.
- Netflix does a great job of this. So they’re known for Chaos Monkey, a tool that goes through and does controlled, plausible, real failure injection into your infrastructure. And it turns out, this is a great thing for not only checking your automation, but for testing your monitoring, alerting, and response processes all at the same time.
- In both planning and execution (which we’ll get to later) … long gone are the days of the lone sysadmin saving the day. In fact - if I’m honest - kill your hero culture. “Firefighters get all the glory, but building inspectors save all the lives.” Be a building inspector.
- “What’s with the preparing you’re always preparing!” OK, now that we have our monitoring, alerting, and process in order, and we’ve practiced it, time to wait for an incident! Don’t worry, you probably won’t have to wait too long. This in turn results in what we consider an “inner loop” of Incident Response, and it looks very similar…
- Assess & Triage. Maybe it’s automatically handled, maybe it’s suppressed, maybe it’s escalated to a human.
- OK! So, our alarms have fired! And maybe a human has looked at the alarm, and decided this was a high enough urgency that the kick off the Incident process (or it kicks off automatically). What’s first?
- Zookeeper example. This is critical, or the call will be chaos.
- State facts, separate from observations & hypothesis, separate from proposed actions. ICs can poll for status, or SMEs can push status.
- The IC should act as a coordinator. SME X proposes an idea, and IC gets consensus, then distributes the task.
- “SME X, do Y, I’ll check back in with you in Z minutes.”
- OK, so now you have an idea of what you need to do. But maybe two of the SMEs aren’t agreeing. Instead of “soooo, what should we do?”, here’s a different way of asking the same thing.
- “Do you want to be an IC?”
- Once you’ve identified the key SMEs, take steps to decrease the impact and cost of the response. Side tangent: “Scope of control”. Think in teams of 7-10. Anything bigger? Break out into sub-groups.
- At the end of your incident coordination lifecycle, there’s the aspect of capturing and learning that feeds into the next iteration.
- So then, the first part of our outer loop - prepare. This is where I’ll point out that the more effort you spend here, the less impact and easier time you’ll have on the other two items.
- It’s true. See also recent issue with GitLab - “we let that person down”.
- Here’s a fresh off the presses slide - if you check my twitter feed you’ll find a great youtube video on cognitive biases to be careful of in your postmortems. “ would likely have” (counter), “insufficient” (normative), assume the system is in perfect working order until the incident (mechanistic)
- This becomes a part of your organization’s deep history. There’s significant value to be mined from archives like this - and that’s part of why even less significant issues should go in here.
- Don’t forget to do some meta-analysis as well about your IC process. What went well? What should be changed next time? Are you getting better or worse? More folks or fewer folks?
- Here’s an example postmortem template … let’s go through a few pieces of it. Apologies for the eye chart.
- Yelp went down this path - and it’s enabled them to scale, grow along their own devops journey, and handle things far faster than before. I’m going to wrap up by giving you a challenge…
- Go, find those alarms this week. It’s only Thursday. Remember …
- … you can get back your time, and at the very least …
- … you’ll get a better night’s sleep. Thank you everyone!
- Questions?
- And I’ll leave you with that URL to our open sourced response documentation. Thank you very much!