As the old adage goes, “you can’t manage what you can’t measure.” But what if the measurements are part of the problem? After all, you can move the needle on what you measure, but there may be unanticipated outcomes. When it comes to transforming how an organization delivers software, how you measure success also needs to change. Despite a marked adoption of agile development practices in the last five years, Forrester Research found that developers haven’t made much progress at delivering software faster. What’s the disconnect? Forrester has found a number of interrelated factors behind organizations that outperform in software delivery. One of them is metrics. Join this webinar with special guests Jeffrey Hammond, Forrester Research Analyst, and Patrick Feeney, Senior Director of IT Digital Products at Healthcare Services Corporation (HCSC), to learn: ● The types of metrics high-performance software teams are using ● Which metrics are counterproductive to improving software delivery ● How HCSC has used metrics to manage software delivery change ● Which roles should be using which metrics and when Speakers: Jeffrey Hammond, Vice President and Principal Analyst, Forrester Research Patrick Feeney, Senior Director, IT Digital Products, Healthcare Services Corporation

1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.
Cut the Digital Transformation
Fluff: Creating Metrics that Matter
Jeffrey Hammond, Forrester Research
Patrick Feeney, Healthcare Services Corporation
Moderator: Dormain Drewitz, Pivotal
July 18, 2019

2. Agenda notes
● Value Metrics
● The Metrics Journey
● Metrics missteps
● Culture

3. 3© 2018 FORRESTER. REPRODUCTION PROHIBITED.
How Do Your Peers Measure Software Success?
Base: 3,294,global developers
Source: Forrester Analytics Global Business Technographics Developer Survey, 2019

4. Value Metrics

5. 5© 2018 FORRESTER. REPRODUCTION PROHIBITED.
It’s tempting to measure data that’s easy to collect
Are Developers “Done on check in?”
What information can we/are we
allowed to radiate?
Do we measure projects? Products?
Services?
What cultural issues are we trying to
modify?
https://en.wikipedia.org/wiki/Build_light_indicator

6. Velocity Measures Value ?Velocity Measures Value ?

7. 7© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Making Quality “Job One” at Ultimate Software
Defects found in production
Defects found in pre-prod
* C = Δ Developer
$

8. 8© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Metrics Should Span Progress, Quality, Efficiency And Value

9. Phases of a Metrics Journey
Value
Insights
Performance
Engagement

10. Measuring what
you need to
improve
Amount of time an engineer spends writing software…
Amount of time it takes to get product into
production...

11. Measuring your “Time to Production”
https://www.youtube.com/watch?v=ZQGmtuG0Nx8 @dormaindrewitz

12. Metrics: A Journey of its Own

13. 13© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Let Get Old Skool For a Moment
https://www.tutorialspoint.com/cmmi/cmmi-maturity-levels.htm
Measure what we do...
Measure our
measures over time...

14. 14© 2018 FORRESTER. REPRODUCTION PROHIBITED.
An Example: Estimation Variance Over Time

15. BCBSIL Mobile App

16. Start Measuring One Thing...

17. 17© 2018 FORRESTER. REPRODUCTION PROHIBITED.
An Example: Zero Bug Bounce
https://www.slideshare.net/PauloHLeocadio/msf-process-model-v-31

18. 18© 2018 FORRESTER. REPRODUCTION PROHIBITED.
How To Apply Second Order Metrics To Value
𝛥 in customer satisfaction/NPS over time, or during multivariate
testing
𝛥 in revenue, operating margin, channel substitution
𝛥 in engagement frequency, product usage
𝛥 in adoption by other internal teams

19. Metrics Missteps

20. 20© 2018 FORRESTER. REPRODUCTION PROHIBITED.
A Software Development Story
You get what you measure - but not always in
the way you might expect…

21. Creating a Culture around
Metrics

22. 22© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Creating A Culture Around Metrics
Align entire teams around shared OKRs
Focus on fewer measure that matter more
Include Progress, Quality, Efficiency And Value - but keep the gross
number low
Measure at the team/squad upward
Use metrics primarily to inform and act (via information radiators)

23. Cover w/ Image
Metrics That Matter are
Visible
● Answering the question “how are we
doing”
● Small number of highly visible metrics
● Easy to understand
● Constant reinforcement from senior
leadership through middle management
● Education

24. Don’t stop at signs and
lights.
Add sirens!

25. Next steps...
● What value are you trying to measure
● Understand your Minimum Valuable Metrics (MVM)
● Starting with one thing… but iterate
● Finding the second order metrics