Presentation about Metrics Driven Development at DevOps-Finland meeting in Helsinki 10.09.2014.

1. Metrics Driven Development
How to improve visibility,
communication and feedback loop
through metrics
Erno Aapa @ernoaapa
Developer, DevOps-consultant
erno.aapa@avaus.fi
DevOps-Finland meetup
Helsinki
10.09.2014

2. ERNO AAPA
Developer / Team Leader
DevOps-consultant
Founder of DevOps-Finland
Daily work as a developer. One of Avaus
tech team leaders and consult companies
about DevOps.
On free time organize DevOps-Finland
meetings.
Who am I?
2 11/09/14

3. Measurement is one of the keys
Everyone emphasize the importance of it

4. Three Ways: Feedback
Feedback loop
4 11/09/14

5. M in C.A.L.M.S
Measure everything
Show the
improvement
5 11/09/14

6. Build-MEASURE-Learn
Measure and learn
6 11/09/14

7. … but in reality…
we need it, but we don’t have it

8. What we have
Business
Monthly / Yearly
reports
Financial reports
Google Analytics
8 11/09/14
Dev
?
Ops
CPU, Mem, IO, Disk
Zabbix/Ganglia/Nagios
QA
Test coverage reports
CI server notifications
Performance test
results

9. What that cause?
• Bad visibility
Ops-team tries to get the information from logs, load metrics, etc.
Business doesn’t have detailed visibility only high level reports
• Dev “just do my job”
They don’t feel the “heartbeat” of the service
• Invalid conception of service state
“No-one complains so it works… I guess :/ ”
• “Business bug” – feature created but not used
Service full of unused legacy features
9 11/09/14

10. Metrics Driven Development (MDD)
“…is a practice where metrics are used
to drive the entire application
development”
InfoQ / 2012

11. Principles
Define metrics before implementation
of Metrics Driven Development (MDD)
• Like in TDD implement test first, implement metric first
• For example: Include metrics to user stories
Instrumentation-as-Code
• Developers must be able to add new metric with minimal(=one line) effort
Single Source Of Truth
• Store common metrics from app, logs, monitoring agents and other tools to single place
• Platform should be so timely, comprehensive, and intuitive to use that everyone instinctively relies on it
Shared view for key metrics
• Shared view to give all same vision
• So simple that everyone can understand it
Use metrics when making decisions
• You have powerful information, use it when you’re making decisions
Maintain and follow the metrics
• Follow the metrics - now popular feature can be waste tomorrow
• Remove unneeded metrics – it’s code, it need to be maintained
11 11/09/14 Erno Aapa / 2014
Librato Blog / 2014
InfoQ / 2012

12. User story
Define what to measure and what we expect to user story
As a user I can order items in my shopping cart.
We measure visited users and count of orders.
We expect to at least 20% of users to order items in
shopping cart.
12 11/09/14
Can be
defined in
EPIC or
FEATURE
too

13. Collect required metrics
Developer adds required code to collect metrics
# Somewhere in your where login happens...
statsd.increment("users.visited”)
13 11/09/14
# ...and where you handle the orders
statsd.increment(”shopping.ordered”)
Somefile.rb
OtherFile.rb

14. Share visibility
Visualize metric and also what we expect
14 11/09/14
Visualize what we
are expecting. Easy
to everyone
understand what is
good or bad.
Use this information
when making
decisions

15. Single place of truth
Provide single place for data to easily compare and analyze
15 11/09/14
Frontend Service Monitoring

16. Shared dashboard for key metrics
Make most important metrics visible to everyone - all the time
16 11/09/14

17. In the meeting
Use the information when you’re making decisions
• What features have went to live?
• Did we achieve what wanted?
• Should we continue to improving them?
• Or should we remove them?
• What is our overall status?
• What we do next to achieve our goal?
17 11/09/14

18. PRODUCT OWNER / STAKEHOLDERS
- Get real time information from
production
- Can make data driven decisions
- Support LeanStartup way
DEVELOPER
- Get the “feeling” and visibility to
production
- Drop unused features
- Can focus on important parts
TEST MANAGER
- Get visibility to production
- Can scale tests to match with
production
SYSADMIN
- Get visibility to inside of the app
- In case of problems easier to find
the error
Everyone benefit from it
18 11/09/14
just to name few…

19. Communication
19 11/09/14
Ops
Business
Dev
Metrics
Shared view and
goals helps
communication
Benefits of MDD

20. Imaginary case
Simple Java web shop

21. My Webshop
21 11/09/14
Web
shop
UI
User orders items
through Web UI
Server Graphite Grafana
After processing the
order, server sends
data to Graphite
Use Grafana to
visualize data.

22. By developer
22 11/09/14
1. Create counter
2. Increment counter
THATS
ALL!

23. By developer
23 11/09/14
SEE
THE
RESULTS!
Type the metric
name and…

24. … when you get started
Gauges
• Queue sizes
Timers
• Query times
• Response times
Counters
• Execution counters
Histograms
• median, 75th, 90th, 95th, 98th, 99th
24 11/09/14
Measure time with
single @annotation
There is so many things what you can measure, if you
have the capability

25. Pitfalls
”Metrics are always powerful political ammunition, to be used for better or
worse”
Metrics-driven Enterprise Software Development- book
”Tried a solution where OPS and DEV were sharing one metrics server… it didn't
work”
Mantas Klasavicius / Adform
”MONITOR ALL THE THINGS! No, you should not wrap every single method call in
your application to increment a counter”
Librato blog

26. What tools to use?
To collect, store, display? SaaS maybe?

27. 27 11/09/14
TOOLS WHY
COLLECT • StatsD (multiple languages)
• Code Hale Metrics (Java)
• Easy to implement by self
• Provide ”one-line” –way to developer
collect any metrics from application
• Possible to change storage from
configuration
STORE • Graphite
• InfluxDB
• … and more
• Store time-series data
• Provides easy accessable API
• Aggrigate data on ”on the fly”
• Downsample old data
DISPLAY • Grafana
• Tesseo
• … and more
• Simple to use
• Simple and clear graphs
• Possible to create multiple dashboards
SaaS • Librato.com
• geckoboard.com
• leftronic.com
• HostedGraphite.com
• Influxdb.com
• …and many more
• Easy to get started
• Maintained
• Free to test
Give a demo and convince others!

28. Graphite vs. InfluxDB
Key / Value Send data Json
echo “dc1.server2.cpuload 5.6" | nc graphite.com
echo “app.visitors 1" | nc graphite.com 2003
28 11/09/14
"name" : “cpu_load",
"columns" : ["value", ”dc", ”server"],
"points" : [ [5.6, ”dc1", ”server2”] ]
"name" : “visitors",
"columns" : ["value”, ”browser", ”version"],
"points" : [ [1, ”Chrome” , 37.0] ]

29. Graphite vs. InfluxDB
average(dc1.*.cpuload)
sum(app.visitors)
Not possible :(
// Top 5 servers with highest load
highestAverage(*. *.load,5)
29 11/09/14
Aggregate data
select average(value) from cpu_load where dc = ‘dc1’
select sum(value) from visitors
select sum(value) from visitors
where browser = ‘Chrome’ and
version > 35
Probably not possible :(
Not full
SQL!

30. Graphite vs. InfluxDB
Summary
• Older project
• Harder to install / configure
• Over 100 aggregation functions
sum, cumulative, compare, highestAverage
• Good for collecting metrics
• Limited “WHERE” queries
30 11/09/14
• Really young project (Apr 2013)
• Easy to get started
• Only basic aggregation functions(17 total)
sum, min, max, median, percentile
• Good for collecting events
• Limited possibilities to aggregate data
Don’t pick one, use both!
(and don’t forget to check SaaS services too!)

31. What next?
That’s cannot be all…

32. What next - Annotations
What happen at 19:27? There were ad in the TV
32 11/09/14
Add annotations automatically when
something happens what can
change the service state.
Like deployments, other events, etc.

33. What next - Alerts
• Have a easy way to add new alerts
• Don’t have to be a major alerts, small “reminders” are ok too
• Notify developers by email when any page response time goes higher than 300ms
• Notify business and developers when any feature usage goes lower than 5%
33 11/09/14
Users
not
using it?
Feature “favourite list”
usage lower than 5%
It’s waste…
REMOVE IT!
Cleaner codebase
No complex UIs
Less functionality
= easier to maintain

34. Easy to get started, so why not?
Try it today, create a demo to your team, convince them
• Next time plan how you could measure it
• Add required metrics to code
• Use free SaaS services to get easily started
• Create nice dashboard, add the most
important graph and convince others!
34 11/09/14

35. Questions?
Thank you!
Erno Aapa @ernoaapa
Developer, DevOps-consultant
Erno.aapa@avaus.fi
10.09.2014