Is Your Gatekeeper Locking Out Quality? Amy Phillips, Head of Test, Songkick Releasing software is scary. As a tester the fear of breaking things comes with the added pressure that you were responsible for the testing. Testers and people working with testers often expect that testing alone is the determiner of quality releases. When the product doesn’t meet expectations more safely checks get added to the testing and release process. Over time testers can easily become gatekeepers as they try to protect users from poor quality products. Unfortunately quality is about far more than just testing. In this experience report, Amy Phillips, with your help, will explore what quality is and look at the ways that Continuous Delivery can help deliver it. A look back over the last 6 years of Songkick development processes will show how removing gatekeepers allowed Songkick to unlock their release cycle and start shipping quality features multiple times a day. DevOps Enterprise Summit London 2016

1. Is your Gatekeeper Locking out Quality?
Amy Phillips
Songkick

2. Songkick - Track your favorite artists and never miss them live
Amy Phillips - Songkick - @itjustbroke

3. Amy Phillips - Songkick - @itjustbroke
More than 500,000 people across the
UK and Europe signed up
235,000 tickets sold
53,000 scalpers identified

4. A story about moving to
frequent releases
Amy Phillips - Songkick - @itjustbroke

5. The problem with releases
• Pressure to deliver
• Tight timeframes
• Often infrequent
Amy Phillips - Songkick - @itjustbroke

6. A typical release
Code
Release
candidate
Bug
ReportBug
Report
Bug
Report
Developer Build Manager Tester Ops Engineer
Release to
Production
Sign off
Business Rep
Amy Phillips - Songkick - @itjustbroke

7. But is this going to deliver
quality?
Amy Phillips - Songkick - @itjustbroke

8. What is Quality?
Do you consider LinkedIn to be a quality website?
Amy Phillips - Songkick - @itjustbroke

9. Some comments on LinkedIn’s quality
Hardly any problems with it, no glitches.
Nice UX and usability.
Bad UI, too many crashes and
bugs. Overloaded with features.
It meets my needs but can
be a hassle to use at times.
I have found it very buggy lately
Amy Phillips - Songkick - @itjustbroke

10. What is Quality?
“Quality is value to some person
who matters“
- Jerry Weinberg
Amy Phillips - Songkick - @itjustbroke

11. What is Testing?
Tested = Checked + Explored
“You’re not done testing until you’ve checked that
the software meets expectations and you’ve
explored whether there are additional risks.”
- Elisabeth Hendrickson
Amy Phillips - Songkick - @itjustbroke

12. The Gatekeeper process
Can I commit? Can I deploy? Can I release?
Automated
integration
test and build
Regression
test
Monitor
ReleaseCommit
Code
Amy Phillips - Songkick - @itjustbroke

13. The Gatekeeper process
Can I commit?
Can I deploy? Can I release?
Automated
integration
test and build
Regression
test
Monitor
ReleaseCommit
Code
Who
broke the
build?
When
will testing be
finished?
How
important is this
bug?
When
will this be
released?
It
doesn’t
work
Amy Phillips - Songkick - @itjustbroke

14. What are we trying to achieve with releases?
Releases when we want them.
Deliver quality products to users.
Without the unexpected surprises of downtime, bugs, unhappy users.
Amy Phillips - Songkick - @itjustbroke

15. A common problem
Releases require people from different teams to come together to
make them happen.
We release something that doesn’t meet the requirements.
Somebody wants reassurance that the release will be successful.
Technical limitations make releases more difficult.
Amy Phillips - Songkick - @itjustbroke

16. The Songkick Technology Team
Developers
Testers
Tech Ops
Designers
Product
Managers
Cross functional teams, Agile approach
Service Oriented Architecture
Ruby on rails, Go, PHP, JavaScript, Java,
C#, Swift
Amy Phillips - Songkick - @itjustbroke

17. The Songkick Gatekeeper process
Automated
integration
test and build
Manual
testing of
feature and
release ReleaseCommitCode
CI Server Test Environment
~1 hour ~2 hours
Amy Phillips - Songkick - @itjustbroke

18. Fix the right problem
Step 1 - Identify the single biggest (perceived) bottleneck. Remove it.
Step 2 - Identify the next biggest (perceived) bottleneck. Remove it.
Repeat until you run out of bottlenecks.
Amy Phillips - Songkick - @itjustbroke

19. A DevOps Approach
Automated
integration
test and build
Automated
release
testing
ReleaseCommit
Code
CI Server Test Environment
~20 minutes
Test Test Monitor
Developer
Environment
Production
Amy Phillips - Songkick - @itjustbroke

20. The result: Releases month on month
0.00
8.00
6.00
4.00
2.00
0.00
Mar Jul Nov Mar Jul
Amy Phillips - Songkick - @itjustbroke

21. In Summary
Locking down a release process and introducing sign offs doesn’t
guarantee quality.
Good quality doesn’t just mean bug-free.
Collaborative nature of DevOps can remove the need for gatekeepers
AND improve quality.
Amy Phillips - Songkick - @itjustbroke

22. Songkick is hiring
songkick.com/jobs
Amy Phillips - Songkick - @itjustbroke

23. Thanks!
Amy Phillips - Songkick - @itjustbroke