Have you ever tried to debug a production outage, when your system comprises apps your team has written, third-party apps your team runs, with logs going into some system, application performance metrics going into another system, and cloud platform metrics going somewhere else? Did you find yourself switching tabs, trying to correlate metrics with logs and alerts and finding yourself in a huge tangle? It is a nightmare. In the data world, we talk about aggregating all our data so we can derive new insights quickly, but what about our operational data? Observability is your ability to be able to ask questions of your system without having to write new code, or grab new data. When you've got an observable system, it feels like you have debugging superpowers, but can be challenging to even know where to start. If you can even convince your colleagues to start, finding the right tools can be challenging. In this talk Inny and Andrew will talk about what monitoring and logging are not sufficient anymore (if they ever were), observability basics, and demo an observability platform that you can use to start your observability journey today.

1. Observability for
Everyone
Andrew Jones & Inny So
@whereismytaco @mini_inny
ThoughtWorks Australia
Kafka Summit San Fransisco 2019

2. Buzz word?

3. A system is observable if the behaviour of the
entire system can be determined by only
looking at its inputs and outputs
Kalman 1961
General theory of control system

4. Observable == logging && metrics

5. Let us tell you a story

6. The host
The app
Another host
The db

7. The host
The app
Another host
The db
Nagioshost metrics
host metrics

8. The host
The host
The host
The host
The host
The app
Another host
The db
Nagios
Logs Logs
ELK
App logs
App logs

9. Dev-ops responsibility split

10. The host
The host
The host
The host
The host
The app
Another host
The db
Nagios
Logs Logs
ELK
Statsd
App metricApp metric
App metric
App metric

11. The host
The host
The host
The host
The host
The host
The host
The app
Another host
The db
Nagios
Logs Logs
ELK
Statsd
New Relic

12. The host
The host
The host
The host
The host
The host
The host
The app
Another host
The db
Nagios
Logs Logs
Splunk
New Relic

13. One tool to rule them all

14. The host
The host
The host
The host
The host
The host
The host
The app
Another host
The db
Nagios
Logs
Logs
Splunk
New Relic
A host
ms
ms
Logs

15. k8s
k8s
The host
The host
The host
The host
The host
The host
k8s
The app
RDS
The db
Logs
Logs
Cloudwatch
New Relic
k8s
ms
ms
RDS
mdb

16. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
Cloudwatch
New
Relic
k8s
ms
ms
RDS
md
Pagerduty

17. Developer experience

20. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
Cloudwatch
New
Relic
k8s
ms
ms
RDS
md
Pagerduty
Prometheus
Sumologic

21. Dashboard culture

22. (╯°□°)╯︵ ┻━┻

23. Sound
familiar?

24. The evolution

25. The evolution
Monolithic Microservice

26. The evolution
Monolithic
Static
Infrastructure
Microservice
Dynamic
Infrastructure

27. The evolution
Monolithic
Static
Infrastructure
Logging &
Metrics
Microservice
Dynamic
Infrastructure
？

29. haveWe sufferedenough !

30. Let’s make on-call fun!

31. How do we
fix it?

32. Event-Driven Architecture

33. Event-Driven Architecture
Event stream

34. Event-Driven Architecture
Event A
APP A Event stream

35. Event-Driven Architecture
Event A
Event A
APP B
APP C
APP A Event stream

36. Logging and Metrics

37. Events

38. Customer with id c892745
login our system
at 10:15:31 on 7/7/2019
it took our system 300ms to return a 401 response

39. Event Vs Metrics
Event
{
“method”: “POST”,
“request_time_ms”’: 300,
“endpoint”: “/login’’,
“status”: 401,
“customer_id”: “c892745”,
“api_version”: 1.1.2,
“build_number”: 125,
“env”: prod,
}
Metrics
{ timer_data:
{ ‘stat.timer’:
{ mean: 250
median: 250
sum: 300
upper: 300
….
}
Tags:[
“c892745”,
“401”,
“POST”
}
}

40. Event Vs Logs
Event
{
“method”: “POST”,
“request_time_ms”’: 300,
“endpoint”: “/login’’,
“status”: 401,
“customer_id”: “c892745”,
“api_version”: 1.1.2,
“build_number”: 125,
“env”: prod,
}
Logs
2019/7/7 10:15:20 [INFO] Starting application
2019/7/7 10:15:22 [INFO] Reading Configuration
2019/7/7 10:15:23 [WARNING] undeclared region,
default to Australia
2019/7/7 10:15:25 [INFO] /login success
2019/7/7 10:15:26 [WARNING] Incorrect type
conversation, will use default
2019/7/7 10:15:27 [INFO] /getProductList/
87923407/
2019/7/7 10:15:31 [INFO] /login error for
customer c892745

41. Event = (logging + metric) * 100000

42. Event-Driven Architecture
Event A
APP A Event stream
Event A
？

43. All the events

44. What do we need?

45. We need Kafka!

46. Kafka can give us:
Evolvablity
Scalability
Reliability
Integrity

47. Alerts?
Dashboard?

48. Event-Driven Architecture
Event A
APP A Kafka

49. Event-Driven Architecture
Event A
APP A Kafka
Event A Consumer

50. Event-Driven Architecture
Event A
APP A Kafka
Event A Consumer
Alert B

51. Event-Driven Architecture
Alert B
Event A
APP A Kafka
Event A Consumer
Alerts
Alert B

52. Event-Driven Architecture
Alert B
Event A
APP A Kafka
Event A Consumer
Email
Alerts
Dashboard
…….
Alert B

53. Benefit of alerts as code

54. Benefit of alerts as code

55. Benefit of alerts as code
Its Just code!

56. Do we still need dashboard?

57. Instrument
everything
(or else)

58. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
Cloudwatch
New
Relic
k8s
ms
ms
RDS
md
Pagerduty
Prometheus
Sumologic

59. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
k8s
ms
ms
RDS
md

60. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
k8s
ms
ms
RDS
md

61. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
k8s
ms
ms
RDS
md
Kafka

62. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
k8s
ms
ms
RDS
md
KafkaEvents

63. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
k8s
ms
ms
RDS
md
Kafka
Datalake
Events
BigQuery

64. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
k8s
ms
ms
RDS
md
Kafka
Datalake
Consumer
Events
BigQuery

65. k8s
k8s
The host
The host
The hostThe host
The hostThe host
k8s
The app
RDS
The db
Logs
Logs
k8s
ms
ms
RDS
md
Pagerduty
Kafka
Datalake
Consumer
Events
BigQuery

68. 500ms request time?!

69. 500ms request time?!

70. 500ms request time?!
450ms request time?!

73. Timing
looks ok to
database!

74. Timing
looks ok to
database!

75. Timing
looks ok to
database!
New Version!

76. Everyone can be a superhero

77. Observability
for everyone

78. Andrew Jones & Inny So
@whereismytaco @mini_inny
Thank you