Even test-driven development or an automated Jenkins pipeline doesn’t guarantee issue-free production operations. Nothing is immune to spike in traffic or unforeseen infrastructure issues. To increase resilience, we see a trend in applying a shift-left approach to the SRE (Site Reliability Engineering) discipline. SREs are contributing their “auto remediation as code” assets to the code repositories which get automatically built and tested in CI/CD and enable automated problem remediation in production. In this session we showcase Shift-Left SRE by leveraging Ansible on OpenShift to automate remediation of production issues based on full stack monitoring data.

1. Shift-Left SRE - Self-healing on
OpenShift with Ansible
#RedHatOSD
Jürgen Etzlstorfer, Technology Strategist
@jetzlstorfer

2. A relentless pursuit of software perfection
1600 Employees 5000 Enterprise Customers 79 of the Global 100

3. confidential
If you write applications,
they will break eventually
~ Murphy‘s law

4. On average, a single transaction uses 82 different types of technology
Browser
Multi-geo
Mobile Network
Code
Hosts
Logs
IoT
3rd parties
Services
Cloud SDN
Containers
Applications are getting more complex!

5. confidential
What if you had
something similar to
a self-healing robot?

6. confidential
But how?

7. Building auto-remediation into your pipeline
#RedHatOSD
✓ ✓ ✓ ✓
commit build test stage prod
✓

8. Building auto-remediation into your pipeline
#RedHatOSD
✓ ✓ ✓ ✓ ✗
commit build test stage prod

9. Building auto-remediation into your pipeline
#RedHatOSD
✓ ✓ ✗
commit build test stage prod
Break the
pipeline early

10. Building auto-remediation into your pipeline
#RedHatOSD
✓ ✓
commit build test stage prod
Include
remediation-as-code
…

11. Building auto-remediation into your pipeline
#RedHatOSD
✓ ✓
commit build test stage prod
Include
remediation-as-code
✓
… … … … …
✓ ✓

12. OpenShift Container Platform
Dynatrace Software Intelligence
Ansible Automation
Self-healing applications

13. Automation
via APIs
▪ Automation/Execution: perform
mitigation/remediation actions
▪ Access to all systems
▪ Monitoring: know what’s going on in your
applications
▪ End-to-end
▪ Full-stack – fully integrated in production
Self-healing building blocks
#RedHatOSD

14. FullStack-PoweredbyDynatraceOneAgent

15. Self-healing with Ansible Tower and Dynatrace
#RedHatOSD
● APIs are key to enable automation
● Ansible Tower provides rich API for managing Ansible jobs
● Playbooks can be orchestrated in workflows and job templates

16. How to enable auto-remediation
Full-stack
environment
is monitored
Anomalies
are detected
automatically
Root
cause
analysis is
performed
Problem
notification
is sent
Event is
received
Job is
triggered
Playbook is
executed
Problem is
remediated
#RedHatOSD

17. How to enable auto-remediation
Full-stack
environment
is monitored
Anomalies
are detected
automatically
Root
cause
analysis is
performed
Problem
notification
is sent
Event is
received
Job is
triggered
Playbook is
executed
Problem is
remediated
#RedHatOSD

18. Ansible Tower integration in Dynatrace
#RedHatOSD

19. Ansible Tower integration in Dynatrace
#RedHatOSD

20. Ansible Tower integration in Dynatrace
#RedHatOSD

21. confidential
What we will see in the demo
▪ TicketMonster application running on OpenShift
▪ Full-stack, end-to-end monitoring by Dynatrace
▪ Feature release via Ansible Tower
▪ Auto-remediation as code (Ansible playbooks)
#RedHatOSD

22. DEMO TIME :)

23. confidential
What we have seen in the demo – short recap
Release of a
new feature
Dynatrace
detects
increase of
failure rate
Dynatrace
fires a
problem
notification
to Ansible
Tower
Ansible
Tower kicks
off a
playbook
Check for
latest
deployment
with
remediation
scripts
Remediation
script is
executed
Problem is
remediated
#RedHatOSD

24. Self-healing in an enterprise environment
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
Escalate at
2AM?
5
1
2
3
4

25. Self-healing in an enterprise environment
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
Escalate at
2AM?
2 High Garbage Collection? Adjust/Revert Memory Settings!
5
1
2
3
4

26. Self-healing in an enterprise environment
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
3 Issue with BLUE only? Switch back to GREEN!
Escalate at
2AM?
2 High Garbage Collection? Adjust/Revert Memory Settings!
5
1
2
3
4

27. Self-healing in an enterprise environment
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
3 Issue with BLUE only? Switch back to GREEN!
Escalate at
2AM?
2 High Garbage Collection? Adjust/Revert Memory Settings!
4 Hung threads? Restart Service!
5
1
2
3
4

28. Self-healing in an enterprise environment
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
3 Issue with BLUE only? Switch back to GREEN!
Escalate at
2AM?
2 High Garbage Collection? Adjust/Revert Memory Settings!
4 Hung threads? Restart Service!
5
1
2
3
4
Update Dev
TicketsImpact Mitigated??

29. Self-healing in an enterprise environment
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
3 Issue with BLUE only? Switch back to GREEN!
Escalate at
2AM?
2 High Garbage Collection? Adjust/Revert Memory Settings!
4 Hung threads? Restart Service!
5 Still ongoing? Initiate Rollback!
5
1
2
3
4
Mark Bad
Commits
Update Dev
TicketsImpact Mitigated??

30. Self-healing in an enterprise environment
Auto Mitigate!
1 CPU Exhausted? Add a new service instance!
3 Issue with BLUE only? Switch back to GREEN!
Escalate at
2AM?
2 High Garbage Collection? Adjust/Revert Memory Settings!
4 Hung threads? Restart Service!
5 Still ongoing? Initiate Rollback!
Escalate? Still ongoing?
5
1
2
3
4
Mark Bad
Commits
Update Dev
TicketsImpact Mitigated??

31. confidential
Embed auto-remediation in your CI/CD pipeline

32. confidential
Embed auto-remediation in your CI/CD pipeline
Path to NoOps: Self-Healing, …
Shift-Right: Tags, Deploys, Events

33. confidential
Embed auto-remediation in your CI/CD pipeline
Shift-Left: Break Pipeline Earlier
Path to NoOps: Self-Healing, …
Shift-Right: Tags, Deploys, Events
Actionable Feedback Loops

34. https://www.ansible.com/blog/enable-self-healing-applications-with-ansible-and-dynatrace
https://www.dynatrace.com/news/blog/set-up-ansible-tower-with-dynatrace-to-enable-your-self-healing-applications/

35. OpenShift Dynatrace♥
“Beyond years of industry knowledge in the APM space, Dynatrace offers one of the best solutions I’ve seen
for monitoring applications running on OpenShift. What really distinguishes them from others is the use of
artificial intelligence based root-cause analysis. OpenShift is a platform to allow you to run decoupled services
and applications, which can be a monitoring nightmare, but Dynatrace’s insights makes it less scary.”
Chris Morgan, Technical Director – Red Hat OpenShift Ecosystem

36. GRAZIE PER L’ATTENZIONE
Jürgen Etzlstorfer, Technology Strategist
#RedHatOSD