While the adoption of DevOps makes teams move faster with reduced dependency on central operations, it can constrain teams who lack the skills to self-manage the full application and infrastructure stack. The way to overcome this challenge is creating an internal platform and treating it as a world-class product offering. “Applying product management to internal platforms means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience”, via ThoughtWorks Technology Radar. In this talk, we will walk you through how Zalando adopted a customer-first mindset with regards to its developer tooling. We will show the effect on developer satisfaction when internal platforms are given the same respect as external product offerings. We will tell our story on how we moved from a classical infrastructure team to a product mindset with strong focus on building a world-class developer experience. We will share both our learnings and challenges going through this transition, and the impact it has on the daily life of our customers (developers).

1. Why we don't use
the term DevOps
2019-06-12
HENNING JACOBS
@try_except_
MOSTAFA NAGEEB
@MostafaNageeb

2. 2
ZALANDO AT A GLANCE
~ 5.4billion EUR
revenue 2018
> 250
million
visits
per
month
> 15.000
employees in
Europe
> 79%
of visits via
mobile devices
> 26
million
active customers
> 300.000
product choices
~ 2.000
brands
17
countries

3. 3
THE BIRTH OF DEVOPS - 2009
“10+ Deploys per Day: Dev and Ops Cooperation at Flickr.”
DevOps Days Belgium
https://devops.com/the-origins-of-devops-whats-in-a-name/

4. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

5. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

6. slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

7. 7
THE PHOENIX PROJECT - 2013
"The Three Ways"
1. Systems Thinking
2. Amplify Feedback Loops
3. Culture of Continual
Experimentation And Learning

8. 8
WHAT HAPPENED NEXT

9. 9
CARGO CULTING

10. 10
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

11. 11
DevOps
is normal

12. 12

13. 13
WHAT HAPPENED NEXT

14. 14
WHAT HAPPENED NEXT

15. 15
DEV + OPS ⇒ DEVOPS
Devs
"DevOps"
Team
Ops

16. 16
DEV + OPS ⇒ ENGINEERING
Devs
"DevOps"
Team
Ops
Engineering
Team

17. 17
PRODUCT DEVELOPMENT TEAMS
"Great products emerge from
empowered, fully cross-functional teams."
- John Cutler, 2019
https://amplitude.com/blog/on-being-product-led

18. 18
DEVOPS IS
It is a culture, in which people
work together to improve the
product delivery cycle.

19. 19
DEVOPS ANTI-PATTERNS
Dev and Ops Silos
DevOps Team Silo
Dev Don't Need Ops
DevOps as Tools Team
Rebranded SysAdmin
Ops Embedded in Dev Team
web.devopstopologies.com

20. 20

21. 21
TEAM INTERACTION MODES
• Collaboration: 2 teams working together
• X-as-a-Service: 1 provides, 1 consumes
• Facilitating: 1 team helps another
Product teams need a family too - DevOpsDays Portugal

22. 22
A BRIEF HISTORY OF
ZALANDO TECH

23. 23
2010
"Sysop-Test"
"QA-Test"

24. 24
DEVOPS IS ABOUT SKIN IN THE GAME
Absence of skin in the game:
People who are isolated from the
impacts of their decisions do not learn.
They remain captive to their
erroneous ideas about
how the world works.

25. 25
2013: SELF SERVICE

26. 26
2015: RADICAL AGILITY
AWS
STUPS
DOCKER
DEPLOY
SSH
ACCESS
AUDIT
REPORTS
FULL AWS
ACCESS
Teams have
admin access
& full
responsibility

27. 27
2015: ISOLATED AWS ACCOUNTS
Internet
*.abc.example.org *.xyz.example.org
Team ABC Team XYZ
EC2EC2
ELBELB
EC2

28. 28
Platform
> 1100
developers
> 200
development teams
2019

29. 29
YOU BUILD IT, YOU RUN IT
The traditional model is that you take your software to the
wall that separates development and operations, and
throw it over and then forget about it. Not at Amazon.
You build it, you run it. This brings developers into
contact with the day-to-day operation of their software. It
also brings them into day-to-day contact with the
customer.
- A Conversation with Werner Vogels, ACM Queue, 2006

30. 30
ON-CALL: YOU OWN IT, YOU RUN IT
When things are broken,
we want people with the best
context trying to fix things.
- Blake Scrivener, Netflix SRE Manager

31. 31
"SKIN IN THE GAME"

32. 32
DEVELOPER JOURNEY
Consistent story
that models
all aspects of SW dev

33. 33
Developer
Journey

34. 34
Developer
Journey
Correctness
Compliance
GDPR
Security
Cost Efficiency
24x7 On Call
Governance
Resilience
Capacity
...

35. 35
DEVELOPER PRODUCTIVITY
Code Build Test Deploy OperateSetup
Cloud Native Application Runtime

36. 36
CLOUD NATIVE
.. uses an open source software stack to deploy
applications as microservices, packaging each part into
its own container, and dynamically orchestrating those
containers to optimize resource utilization.
Cloud native technologies enable software developers to
build great products faster.
- https://www.cncf.io/

37. 37
CONTAINERS END-TO-END
Code Build Test Deploy OperateSetup
Cloud Native Application Runtime

38. 38
CONTAINERS

39. 39
CONTAINERS

41. 41
PLAN & SETUP

42. 42
Plan
Stories
Rules of Play
Tech Radar

44. 44
Setup
Application
Bootstrapping

47. 47
BUILD & TEST

48. 48
CDPGit
code
push
CONTINUOUS DELIVERY PLATFORM: BUILD

50. 50
DEPLOY

51. 51
Deploy
Kubernetes

52. 52
DEPLOYMENT CONFIGURATION
├── deploy/apply
│ ├── deployment.yaml
│ ├── credentials.yaml # Zalando IAM
│ ├── ingress.yaml
│ └── service.yaml
└── delivery.yaml # Zalando CI/CD

53. 53
INGRESS.YAML
kind: Ingress
metadata:
name: "..."
spec:
rules:
# DNS name your application should be exposed on
- host: "myapp.foo.example.org"
http:
paths:
- backend:
serviceName: "myapp"
servicePort: 80

54. 54
TEMPLATING: MUSTACHE
kind: Ingress
metadata:
name: "..."
spec:
rules:
# DNS name your application should be exposed on
- host: "{{{APPLICATION}}}.example.org"
http:
paths:
- backend:
serviceName: "{{{APPLICATION}}}"
servicePort: 80

55. 55
CONTINUOUS DELIVERY PLATFORM

56. 56
CDP: DEPLOY
"glorified kubectl apply"

57. 57
CDP: OPTIONAL APPROVAL

58. 58
STACKSET: TRAFFIC SWITCHING
github.com/zalando-incubator/stackset-controller

59. 59
STACKSET CRD
kind: StackSet
...
spec:
ingress:
hosts: ["foo.example.org"]
backendPort: 8080
stackLifecycle:
scaledownTTLSeconds: 1800
limit: 5
stackTemplate:
spec:
podTemplate:
...
github.com/zalando-incubator/stackset-controller

60. 60
TRAFFIC SWITCHING STEPS IN CDP
github.com/zalando-incubator/stackset-controller

61. 61
EMERGENCY ACCESS SERVICE
Emergency access by referencing Incident
zkubectl cluster-access request
--emergency -i INC REASON
Privileged production access via 4-eyes
zkubectl cluster-access request REASON
zkubectl cluster-access approve USERNAME

62. 62
INTEGRATIONS

63. 63
CLOUD FORMATION VIA CI/CD
├── deploy/apply
│ ├── deployment.yaml # Kubernetes
│ ├── cf-iam-role.yaml # AWS IAM Role
│ ├── cf-rds.yaml # AWS RDS Database
│ ├── kube-ingress.yaml
│ ├── kube-secret.yaml
│ └── kube-service.yaml
└── delivery.yaml # CI/CD config
"Infrastructure as Code"

64. 64
ZALANDO IAM/OAUTH VIA CRD
kind: PlatformCredentialsSet
..
spec:
application: my-app
tokens:
read-only:
privileges:
- com.zalando::foobar.read
clients:
employee:
grant: authorization-code
realm: users
redirectUri: https://example.org/auth/callback
Extensible Kubernetes API

65. 65
POSTGRES OPERATOR
Application to manage
PostgreSQL clusters on
Kubernetes
>700
clusters running
on Kubernetes
github.com/zalando/postgres-operator

66. Elasticsearch in Kubernetes
Elasticsearch
2.500 vCPUs
1 TB RAM
github.com/zalando-incubator/es-operator/

67. 67
SUMMARY
• Application Bootstrapping
• Git as source of truth and UI
• 4-eyes principle for master/production
• Extensible Kubernetes API as primary interface
• OAuth/IAM credentials
• PostgreSQL
• CloudFormation for proprietary AWS services

68. 68
Platform as a Product

69. “.. means establishing empathy with internal
consumers (read: developers) and collaborating
with them on the design. Platform product managers
establish roadmaps and ensure the platform delivers
value to the business and enhances the developer
experience.”
- ThoughtWorks Technology Radar

70. 70
PRODUCT MINDSET
... means we are developing all our products
for the best customer experience
and biggest business impact
while all actions are
aligned with our strategy.

71. 71
KNOW YOUR AUDIENCE

72. 72
● Be Prepared.
● Talk to both users and
NON users.
● Ask about problems,
not solutions.
● Open ended
questions.
TALK TO USERS

73. 73
IDENTIFY COMMON PROBLEMS
"Documentation is hard to find"
"Documentation is not comprehensive enough"
"Remove unnecessary complexity and obstacles."
"Get the documentation up to date and prepare
use cases"
"More and more clear documentation"
"More detailed docs, example repos with more
complicated deployments."

74. 74
TRANSPARENTLY COMMUNICATE
WHAT YOU ARE SOLVING

75. 75
SIGNAL: ISSUE UPVOTES

76. 76
ENGAGE CUSTOMERS EARLY ON
We are working on speeding up builds by providing basic
overlays for the most used languages/frameworks so you don't
have to download the internet every time you are building your
app.
This is the approach and the documentation (not live yet),
please add your comments to guide us in the right direction.

77. 77
SUPPORT IS A FEATURE

78. 78
DOCUMENTATION
• Restructure following
www.divio.com/en/blog/documentation/
• Concepts
• How Tos
• Tutorials
• Reference
• Global Search
• Weekly Health Check: Support → Documentation

80. 80
COMMUNICATION
Guidelines
• Target announcements
• Use the right tags
• Timing

81. 81
NEWSLETTER
"You can now.."
• You can now upload your Scala code coverage
report to SonarQube
"This enables you to.."
• This enables you to browse your code quality
results and track it over time

82. 82
“Useful information, good level of details and pleasant to
read. It's one of the few newsletters that I took time to
read entirely :)”
- a reader, July 2018

83. 83
EMPOWER YOUR TEAM

84. 84
WITH GREATER POWER COMES
GREATER RESPONSIBILITY

85. 85
DELIVERY PERFORMANCE METRICS
• Lead Time
• Release Frequency
• Time to Restore Service
• Change Fail Rate
srcco.de/posts/accelerate-software-delivery-performance.html

86. 86
DELIVERY PERFORMANCE METRICS
• Lead Time
• Release Frequency
• Time to Restore Service
• Change Fail Rate
• Commit to Prod
• Deploys/week/dev
• MTRS from incidents
• N/A

87. 87
DELIVERY PERFORMANCE METRICS
• Lead Time
• Release Frequency
• Time to Restore Service
• Change Fail Rate
• Commit to Prod
• Deploys/week/dev
• MTRS from incidents
• N/A

89. 89
DEVELOPER SATISFACTION

90. 90
INSIGHTS THAT GUIDE THE PRODUCT
“Kubernetes/CDP deploys are too
slow - ours takes over 10 minutes,
mainly due to having to download
SBT dependencies every time.”

91. 91
INSIGHTS THAT GUIDE THE PRODUCT
“CDP should somehow cache
builds or something like that. The
current approach forces everyone to
re-invent the cache wheel for their
own projects - or face a crazy slow
build.”

92. 92
INSIGHTS THAT GUIDE THE PRODUCT
“Kubernetes/CDP deploys are too
slow - ours takes over 10 minutes,
mainly due to having to download
SBT dependencies every time.”

93. 93

94. 94
TESTIMONIALS
“So, thank you, Team Automata, for listening to our
community, taking our upvotes in consideration when
developing new solutions and building every day
'the first CI that doesn't suck'.”
- a user, October 2018

95. 95
TESTIMONIALS
“Thank you very much for improving the color scheme of
the UI. It really helps me with my color blindness. It
used to be hard to distinguish between statuses
previously.”
- a user, February 2019

96. 96

97. 97
Wrap Up

98. 98
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

99. 99
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

100. 100
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

101. 101
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

102. 102
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

103. 103
DEVOPS IS
… if every person uses the same tool for the same job
… codified knowledge - everybody contributing their part to common automation
… if all people have the same privileges in their tooling
… if human error is equally possible for Dev and Ops
… replacing people interfaces by automated decisions and processes
… a result
bit.ly/5pdops

104. 104
DevOps
is normal

105. 105
DevOps
is normal

106. 106
You build it,
you run it
is normal

107. 107
WAIT A SECOND..
What about
my job title?

108. 108
"DEVOPS ENGINEER" AS A JOB TITLE

109. 109
T-SHAPED ENGINEER
"Generalizing specialist"
Breadth of Knowledge
TechnicalDepth

110. 110
DevOps Engineer
Software Engineer /
Software Developer
(T-Shaped)

111. 111
ON THE DEVOPS JOB TITLE
"Everyone’s a devops engineer
these days. I’m a devops engineer,
my wife’s a devops engineer,
even my dog’s a devops engineer"
- James Betteley, 2014
devopsnet.com/2014/08/13/the-devops-team-myth/

112. 112

113. 113
OPEN SOURCE & MORE
Kubernetes on AWS
github.com/zalando-incubator/kubernetes-on-aws
Skipper HTTP Router & Ingress controller
github.com/zalando/skipper
External DNS
github.com/kubernetes-incubator/external-dns
Postgres Operator
github.com/zalando-incubator/postgres-operator
More Zalando Tech Talks
github.com/zalando/public-presentations

114. QUESTIONS?
HENNING JACOBS
@try_except_
MOSTAFA NAGEEB
@MostafaNageeb
Illustrations by @01k