As far back as 1922, aeronautic experts knew how to travel at supersonic speeds, but not without the plane breaking apart and killing the pilot. Have rapid advancements in continuous integration over the past quarter-century placed software delivery in a similar dilemma? In this workshop, Dean Peters will speak to the question “How do we rapidly deliver valuable software without breaking the product and killing the company?” In showing how companies such as Netflix and Microsoft are breaking the sound barrier using feature flags to iterate safely and rapidly validate features of value, Dean will demonstrate how to power your decisions and delight your customers by aligning your product offerings on a feature-flag driven culture of experimentation.

1. using experimentation to
deliver value at
supersonic speeds
by dean peters (@deanpeters)

2. The Vision

3. Imagine working in a product culture
where conversations are fueled by
experimentation and validation.

4. Imagine a product culture not overrun by HiPPOs & RHiNOs

5. But Dean, how do we get from 0 to ProdOps in no time flat?
Are there any historic examples of big visions that can guide
us?

6. There was never a problem with a lack of vision
when it came to breaking the sound barrier

7. The problem was traveling at supersonic
speeds
without the plane breaking apart
and killing the pilot.

8. Is the our vision of rapid feature delivery
at a similar historic turning point?

9. First, a little about me ...
1995
Senior
Developer
●
2005
Director of
Product
● ●
●
2010
Product
Team Lead
● ● ● ●
McClatchy
2015
Product
Manager
● ● ● ● ●
2000
Principal
Developer
●
●
2019
Principal Product
Manager
●●●●●●●
●

10. Second, a little bit about what I’ve seen ...
1995
Every
Couple Years
●
2005
Every
Couple Weeks
● ●
●
2010
Twice
a Week
● ● ● ●
2015
Several Times
per Week
● ● ● ● ●
2000
Twice
a Year
●
●
2019
Hundreds of
Time Per Day
●●●●●●●
●

11. If we’re delivering features faster than ever before,
then why isn’t product management celebrating?
Why aren’t we having ProdOps parties like our DevOps peeps?

12. British Airways bug this past May grounds all flights from Heathrow
Is it because we’re failing fast, but
fatally?
Kindle 3.6.1 for update iOS deletes user’s books
Hulu Super Bowl LII Outage
South Carolina Lottery everyone’s a winner bug
SunCorp Bank disappearing money upgrade
Facebook & Instagram Jan & Feb’18 Outages
Uber outage leaves drivers stranded Fiat’s deadly airbag bug
Dow drops due to bug impacting Fidelity, T. Rowe Price, Bank of America, & TD Ameritrade
Twitter downed by special characters
Kraken Cryptocurrency 4 day outage
Epic Games’ Fortnite vulnerable since 1995
U.S. Customs New Year’s Day surprise outage
Knight Capital’s Feature Flag flub
Bugs cause Lockheed Martin delivers late on the F-35b
India Air 787’s grounded
Six Thousand 911 callers unable to connect last April
Apple forced to pull iOS 8 update
Bad software implementation causes U.S. National Grid Gas Co. loses $945 million
U.K. Airspace closed for a day
Amazon’s 1p pricing snafu
Starbucks cash register bug closes 60% of its stores
RBS integration w/3rd party causes 600,000 payments to fail
Bloomberg U.K. trading terminals go down
HBSC finds a bug causing 275,000 payments to fail
Prison Software approves ‘good behavior’ releases for 3,200 criminals too many
Nest Bug leaves users in the cold
Bitcoin Unlimited memory leaks
Welsh National Healthcare System unable to access patient records
TBS Bank locks out millions of users

13. The 2018 Tricentis ‘Fail Watch Report’
details 606 recent software failures
that affected over 3.6 billion people
resulting in $1.7 trillion in lost revenue.
In examining the collapse of 101 start-ups between 2017 &
2018, CBInsights cited the top 20 contributing causes,
half indicating failures in value delivery.

14. Are we simply delivering crap faster?

15. How do we align on the supersonic delivery of valuable features
without the product breaking apart
and killing the company?

16. The Strategy

17. In breaking the Sound Barrier ...
hypotheses were stated
tests were run
causality was analysed

18. State a Hypothesis
Source: xkcd.com - Creative Commons Attribution-NonCommercial 2.5 License.
A narrowly focused
if … then … feature
statement.
Whose value is measured
by impact on key
behavioral metrics.

19. Deploy Rapidly in Tiny Test Increments
Source: xkcd.com - Creative Commons Attribution-NonCommercial 2.5 License.
Trunk-based branching that
supports very small and very
automated rapid deployment.
Telemetry to capture both
technical and product metrics.

20. Learn via Causation
Source: xkcd.com - Creative Commons Attribution-NonCommercial 2.5 License.
Tests include scientific
elements of control.
Random assignment of
treatments.
Statistical significance.

21. Learn Build
Measure
Lean Build,
Measure, Learn
Model
Evolve from simply talking about Build Measure Learn

22. HypothesizeAnalyze on Causation
Better BuildsMeasure DevOps
Scientific Test Treatments
Experimentation
as a
Service
To Driving Value via a Culture of Experimentation

23. Ronny Kohavi’s observations based on experiments at
Microsoft
● 1/3 of ideas were positive ideas and statistically significant
● 1/3 of ideas were flat: no statistically significant difference
● 1/3 of ideas were negative and statistically significant
Avinash Kaushik, author of ‘Web Analytics: An Hour a Day,’
wrote in his Experimentation and Testing primer that
“80% of the time you/we are wrong about what a customer wants.”

24. The Tactics

25. Feature
Featur
e
Feature Feature
Featur
e
Featur
e
Featur
e
Featur
e
Featur
e
Sprint 1 Sprint 2 Sprint 3 Sprint 5Sprint 4
‘Lean’ into smaller feedback cycles
Waterfall
Agile
Lean w/
Feature
Flags
DeployBuildBUFD

26. How do we align on a culture of experimentation?
By measuring behaviors of value
Through the tactical implementation of feature flags.

27. A Feature Flag is Code
Source: How to implement feature flags and A|B testing; msdn blog
Feature
Flag
Segment
switch (myFeatureCondition) {
case 1:
result = myApp.NewTreatment();
break;
case 2:
result = myApp.OldTreatment();
break;
default:
result = myApp.OldTreatment();
}

28. Discussions with DevOps
● Encourage trunk-based development in tiny branches
● Allow incomplete & in-progress features to be deployed as dark code
● Protect against poorly performing features with a kill-switch
● Facilitate progressive rollouts.

29. Persuading Product & Marketing
● Supports early adopter and preview programs
● Allows visibility management via subscriptions, user rights, or sales
packages
● Empowers QA to quickly toggle a wide variety of test environments
● Supports A/B testing and generates statistically significant results

30. Some Conversational Caveats
● Feature Flags introduce Technical Debt
● Feature Flags requires a cultural shift in thinking
● Feature Flags must be short lived
● Feature Flags will get messy really fast if not well managed
● Feature Flags wile give some of your B2B customers pause

31. It still beats the alternative of ZEbRA & WoLF prioritization!
Source: Behold, the Product Management Prioritization Menagerie - DeanOnDelivery.com

32. The Example

33. A ProductCraft Use Case

34. ● We believe that we can increase reader
engagement per session by offering
‘Related Stories’ instead of ‘Trending Stories’
to users arriving at a story via a search action.
(nice hypothesis there Deano, but how do we measure it?)
Value Hypothesis

35. Pick an OOB Growth, Value, or Ops metric based on context
Sources: How to Use the Google HEART Framework to Measure and Improve Your App’s UX
DevOps Implementation Services - Veritis & AARRR vs. RARRA Sequences - AppGrowth Summit
Happiness
Engagement
Adoption
Retention
Task Success

36. Defining our example test with Google’s value HEART metric
Baseline Test
Happiness NPS NPS & Interviews
Engagement Clicks on Trending Stories Clicks on Related Stories
Adoption A/A Test A/B Test
Retention Active Visitors who Actively Search Active Visitors who Actively Search
Task Success Bounce on Trending Stories Clicked Bounce on Related Stories Clicked

37. ● We believe that we can increase reader engagement
per session by offering ‘Related Stories’ instead of
‘Trending Stories’ to users arriving at a story via a
search action.
● We will know we have succeeded if we see a lift
in click-through behaviors on related stories
offered in the sidebar.
Value Hypothesis with Success Metric

38. Run tiny experiments rapidly and incrementally
Test 1
Baseline A/A
Results on
Trending
Test 2
Replace Trending
with Related
Test 3
Measure
Relevance of
Selected Stories
Test 4
Measure Number
of Relevant
Stories

39. Visualization of our Hypothesis
Sources: How to Choose the Right Metrics for Your Experiments; Spli.io &
How to Use the Stickiness Ratio to Measure Product Health; Pendo

40. Feature Flag Powered Progressive Rollouts
Source: How to implement feature flags and A|B testing; msdn blog
Feature
Flag
Segment
5%
50%
95%

41. The Execution

42. How do we rapidly deliver valuable features
without the breaking product
and killing the company?

43. How do we move from a culture of seagull management & ticket-slinging?
Source: Behold, the Product Management Prioritization Menagerie - DeanOnDelivery.com

44. We pursue a vision of rapid value delivery
Through strategic experimentation
Tactically facilitated by feature flags
Focusing daily release activities that measure causality
Supersonic Software Delivery through Experimentation

45. Where to Start?
● Dogooding
● Proof of Concepts
● Painted Doors
● Canary Tests
● Trial Users vs. Paid
● SMB v. Enterprise
● Early Adopter Programs
You may already have some experiments in play

46. An example of some experimentation success ...
● 4 years Citrix + Pendo luv
● 10 Pendo subscriptions
● Enjoyed by both Marketing &
Product
● Used for Onboarding, User
Adoption,
Retention & Engagement, Upsell
● 22 Citrix products and services
● 500+ live guides
● 2,000+ feature tags
● Dogfooding the WorkSpace offering

47. And you’ve got friends

48. Questions?
Comments?
Conversations?
Dean Peters (@deanpeters)

49. Useful URLs
Books I referenced
● Unlearn
● Accelerate
● Lean Analytics
Valuable Vendor e-Books
● Understanding Experimentation Platforms | Split.io
● Effective Feature Management eBook | LaunchDarkly
● The Ultimate Guide For Feature Toggles | Rollout.io
Podcasts I Mentioned
● DeliverIT - Episode 91
● ProductCraft - Cast #4
● Product to Product - #NoEstimates
Shameless self-promotion
● DeanOnDelivery blog (& bookcovers)
● LinkedIn
● Twitter
● Experimental Podcast Page
(With 3 test casts posted by 28Sep19
you can enjoy & brutally critique!-)
Articles by Authors I Admire
● Ronny Kohavi (pdf)
● Avinash Kaushik (blog)
● Martin Fowler (blog)

50. www.pendo.io
And thanks to Pendo and session attendees for giving me this awesome opportunity to
share my vision on transforming product culture through experimentation!