Robert Mircea & Virgil Chereches: Our Journey To Continuous Delivery at I T.A.K.E. Unconference 2015

1. Virgil Chereches & Robert Mircea
May 28th, 2015
Our Journey to
Continuous Delivery

2. 2
“How long would it take your organization to deploy a change that
involved just one single line of code?
Do you do this on a repeatable, reliable basis?”
Mary and Tom Poppendieck, Implementing Lean Software Development

3. 3
“Failure is not an option”
 Learning opportunity
 Detectable quickly
 Restore service quickly
Failure is inevitable
Orange

4. 4
IT Performance and DevOps Practices
“Our data shows that IT performance and well-
know DevOps practices, such as those that enable
continuous delivery, are predictive of
organizational performance.
As IT performance increases, profitability, market
share and productivity also increase.
IT is a competitive advantage, not just a utility, and
it’s more critical now than ever to invest in IT
performance.”
http://bit.ly/1LwY4oB

5. 5
The Need for Continuous Delivery
Agility for our Business
Eliminate wasted time due to hand offs between teams

6. 6
Enable Architectural Decomposition of Systems

7. 7
Do Continuous Delivery to be Ready for…
Microservices
Swarm

8. 8
Share the Vision of CD
Spread
the “WHY”
the “WHAT”
ensure they “GET IT”
Started organic, inspired by a few
enthusiasts.
Understand that it is about
changing the way we work

9. 9
How We Joined the Army
Dev Ops
Enthusiasts
7 x DevOps Wannabes
One project
One opportunity
R&D style
Managers jumped on
board from the start
DevOps

10. 10
What We Want to Achieve
Push to release
at Orange scale

11. 11
JBoss cluster
standalone
JBoss cluster
standalone
v1.0 v1.1
JBoss cluster
standalone
v1.1
…
VMWare VCenter Infrastructure
JBoss cluster
standalone
v1.0
ProdDev/UAT
v1.1
commit
https://www.orange.ro/myaccount

12. 12
From What We Start
Business
requests in the
hundreds/year
200+ apps in
Java, C#, C, a
few PHP. Lots of
PL/SQL
~150 people –
Dev, QA, BA,
Ops, PM
Mélange of
proprietary and
lots of custom
development apps

13. 13
What is Continuous Delivery?
Small, frequent changes delivered constantly
A method to reduce risk of failure
A way to give to our business a glimpse of the shape of product being built
A set of practices enabled by a toolset

14. 14
Many deployments eventually lead to a Product Launch
What is the difference between
a Deployment and a Product Launch?

15. 15
Changes flowing through deployment pipeline
Push the
button
“Approval”

16. 16
Orange Deployment Pipeline Architecture
(Atlassian Stash Git / SVN)
(Artifactory)
(Jenkins/Teamcity/Sonar) (Selenium)
(Enterprise Tester)
(JMeter)
Rundeck
/Puppet

17. 17
Best Practices for CD
 build application only once and use it in all environments
 keep application releasable all the time – master is releasable at all times
 hide new functionality until it is finished – use feature flags
 make small changes incrementally – commit and integrate often
 use snapshots (in Maven) with care

18. 18
Techniques for achieving
Continuous Deployment

19. 19
Blue-Green Deployments
v1.0-WL
App Server
v1.0-Jboss
www.orange.ro/myaccount-corporate
Weblogic
JBoss

20. 20
Feature flags
Integrating with
production is a test
 On/Off
 Orange IPs
 List of Users
 Time interval
 Release date
 Gradual Rollout (%)
 Blacklist/whitelist IPs
Enable/disable features
quickly (no restart)
ORO testers check feature
in live

21. 21
Prefer Non-Breaking Expansion in Database
Database migration tools
FluentMigrator (.NET)

22. 22
Code Based Migration - FluentMigrator
Run migration
.migrate --conn "server=oracle-server; database=SmsDb" -db OracleManaged -a
"..MigrationsMigrations.dll" --task migrate:up
Rollback change
.migrate --conn "server=oracle-server; database=SmsDb" -db OracleManaged -a
"..MigrationsMigrations.dll" --task migrate:down

23. 23
Automated tests AFTER deploy
Run smoke tests automatically to validate deployment success
Examples:
 make a HTTP request to homepage and look for a string
 login with a test user
 make a recharge with a test PrePay

24. 24
Metrics
Server Metrics
 CPU utilization
 disk space
Application Metrics
 JMS messages sent/received,
 memcached hits
Business Metrics
 Logins / Registration
 Orders
 Recharges
 Options activations

25. 25
Naming Metrics
Establish consistent rules in naming metrics
Env
App name
Server
name
Module
name
Message
type
Target
Action
prod.apps.maco.platini.jms.vantive.customers.request.sent

26. 26
Graphite & Grafana - real-time metrics

27. 27
Instrumenting Code for Metrics Collection
Trivial counters with Statsd (https://github.com/etsy/statsd/)
Types of counters:
 Counters
 Timers
 Gauges
 Histograms

28. 28
Centralized Logging
Apache Logs
App logs
Logstash
Lumberjack
Logstash
Log shipping
Logstash
(parse log into fields)
Elasticsearch
(index logs)
Statsd + Graphite
(metrics dashboard)
(log search & analysis)

29. 29
Infrastructure Automation

30. 30
“At an abstract level, a
deployment pipeline is an
automated manifestation of
your process for getting
software from version control
into the hands of your users.”
Deployment Pipeline

31. 31
Provisioning managers
Configuration managers
Workflow managers

32. 32
Configuration Manager Choice

33. 33
Software components
HIERA

34. 34
Coding best practices
Data/code separation
r10k
HIERA
Keep code in version control
reaktor
Unit & acceptance testing
Development workflow feature branch
workflow
puppet-rspec
server-spec

35. 35
Node breeds
Several items give
node breed:
 puppet role class
Foreman host
group (one-to-one
relationship)
 stage
(dev/prod/qa/test)
 Foreman compute
profile (compute
provider, CPUs,
RAM, Disk)

36. 36
Tag-based everything
Nodes Workflow
steps
Security

37. 37
Node herds
Pre-provisioned nodes
Shorten
deployment
time
Breed
matters
Hostname
does not
matter

38. 38
Don’t be afraid to get dirty
Rundeck API Foreman API PuppetDB API
BigIP iControl Atlassian Stash API

39. 39
Scaling Puppet & Foreman

40. 40
What’s Next
• Sponsor Rundeck
development for SCM
integration and sensitive
options encryption
• Evaluate network
orchestration
• Containers
• Acceptance testing of
puppet code with beaker
(rspec+serverspec)

41. multumim