10 Techniques for Flow & Continuous Delivery Startups are continually evangelizing DevOps to be able to reduce risk, hasten feedback and deploy 1000’s of times a day. But what about the rest of the world that comes from Waterfall, Mainframes, Long Release Cycles and Risk Aversion? Learn how one company went from 480 day lead times and 6 month releases to 3 month releases with high levels of automation and increased quality across disparate legacy environments. We will discuss how Optimizing People & Organizations, Increasing the Rate of Learning, Deploying Innovative Tools and Lean System Thinking can help large scale enterprises increase throughput while decreasing cost and risk.

1. DevOps & Lean in Legacy Environments
10 Techniques for Flow & Continuous Delivery
Scott Prugh
Chief Architect & VP NA Software Development
@ScottPrugh
Scott.Prugh@csgi.com
Copyright © 2012 CSG Systems International, Inc. and/or its 1 affiliates (“CSG International”). All rights reserved.

2. CSG in NA: Who Are We?
Customer Care and Billing Operations
• ~50M Subscribers across 120 customers
• 100k Call Center Seats
• ~6B External Transactions/month
• 40 Dev. Teams & 1000 Practitioners
• ACP: ~20 Technology Stacks: JS to HLASM
• Integrated Suite of 50+ applications
Challenges:
• Time to Market & Quality/Release Impact
• Technology Stovepipes
• Role Stovepipes
Print and Mail Factory
• ~70M statements/month
• Lean & Efficient
Challenges
• Continuous Optimization
Copyright © 2012 CSG Systems International, Inc. and/or its 2 affiliates (“CSG International”). All rights reserved.

3. Release Improvements From Lean & DevOps
Impact Score:
Critical=4
High=3
Medium=2
Low=1
Reducing the batch size from 28 to 14 weeks resulted in a 66.6%(3x) improvement
And value is delivered in half the time!
Copyright © 2012 CSG Systems International, Inc. and/or its 3 affiliates (“CSG International”). All rights reserved.

4. More Practice & Smaller Batches
“Same Code(with fixes)” deployed 2nd time…
© 2014 CSG 4 Systems International, Inc. Confidential and Proprietary

5. Goal: Improve Quality & TTM
But, how do we improve quality, reduce batch size & move towards flow
when the system is inherently structured to prevent this?
Techniques to Address
1. Accelerate Learning & Lean Thinking
2. Inverse Taylor Maneuver
3. Inverse Conway Maneuver
4. Shared Service Continuous Delivery
5. Environment Congruency & Practice
6. Application Telemetry
7. Visualize Your Work
8. Work Release & WIP Limits
9. Cadence & Synchronization
10. Reduce Batch Size
System Constraints
1. Structure
2. Stovepipes & Handoffs
3. Technology Variance
4. Defects & Quality
5. Low Automation
6. Fragility
3 year journey!
© 2014 CSG 5 Systems International, Inc. Confidential and Proprietary

6. 1. Accelerate Learning & Lean Thinking
People First: Build a culture of Learning and Self Improvement
by embracing Lean & Systems Thinking.
Find a Lean Framework Encourage Cross Training
T-Shaped
I-Shaped
E-Shaped
© 2014 CSG 6 Systems International, Inc. Confidential and Proprietary

7. 2. Inverse Taylor Maneuver
Structure & responsibility will enforce behavior and prevents learning.
Architect
Developer
Product
Owner
Developer
Developer
QA
QA
Ops
Traditional Agile
Optimize
Optimize
Optimize
Analysis Architecture Developer QA
Handoff Handoff Handoff
Which structure incents learning
and removes queues?
SM
Organize teams to optimize the entire flow of value.
QA
Optimize
© 2014 CSG 7 Systems International, Inc. Confidential and Proprietary

8. 3. Inverse Conway Maneuver
Structure will enforce technology and architecture.
Technology:
Fat CS Desktop
Technology:
Standard SOA
Architecture
Technology:
Legacy
Tuxedo MW
Technology:
Standard SOA
Architecture
Change the structure to help create the architecture you want.
© 2014 CSG 8 Systems International, Inc. Confidential and Proprietary

9. 4. Shared Service Continuous Delivery
Pave one automated high-way to production so all teams can get there
with speed and predictability.
Having many automated roads
to production is better than no
road or “dirt roads”.
But having one road encourages reuse at
scale and enforces consistency by buffering
downstream teams from variance.
© 2014 CSG 9 Systems International, Inc. Confidential and Proprietary

10. 5a. Environment Congruency & Practice
Separate deployment and operations processes via infrequent
handoffs create large batch transfers and high-failure rates.
Ops.
Optimize
Agile DBT Team(7+/-2)
Agile DBT Team(7+/-2)
Agile DBT Team(7+/-2)
Deploy Everyday
“The Practice Team”
Deploy 4x/year
“The Game Team”
Handoff 4x/year
Dev./Test Environment Production Environment
Do you play the “Game” with a different team than you “Practice” with?
© 2014 CSG 10 Systems International, Inc. Confidential and Proprietary

11. 5b. Environment Congruency & Practice
Make environments as close to production as possible and have
the same team practice daily.
Shared Operations Team(7+/-2)
Agile DBT Team(7+/-2)
Agile DBT Team(7+/-2)
Agile DBT Team(7+/-2)
Dev./Test Environment Production Environment
Deploy Everyday
“The Game Team Practices”
Deploy 4x/year
“Practice Made Perfect”
Continuous Delivery
We let our teams practice at least 70 times before “Game Day”….
© 2014 CSG 11 Systems International, Inc. Confidential and Proprietary

12. 6. Application Telemetry
Build or embed deep telemetry into your application. Telemetry allows the
teams to learn and greatly reduces TTR when there are issues.
NASA doesn’t launch a rocket without
millions of automated sensors
reporting the status of this valuable
asset.
Why don’t we take the same care with
software?
Process Process
Realtime Telemetry
Trace/Activity Collection Queues
StatHub Repository
Analytics
Reporting
© 2014 CSG 12 Systems International, Inc. Confidential and Proprietary

13. 7. Make All Work Visible
Haphazard and chaotic work intake processes invite context
switching and high WIP(Work in Process) levels.
Worker1
Worker2
Worker3
Dependency Worker4
Why am I
waiting?
Why am I
waiting?
© 2014 CSG 13 Systems International, Inc. Confidential and Proprietary

14. 8. Work Release & WIP Limits
An intake buffer and process can take many forms but allows a
quick triage, unified visibility, and WIP management.
Worker1
WIP Limit
Worker2
Work
Work
Work
Work
Work
Work
Worker3
Worker4
I’m
ready!
I’m
ready
One List of Work!
© 2014 CSG 14 Systems International, Inc. Confidential and Proprietary

15. Print Center: Work Visibility Example
Treat virtualized work with the same care that manufacturing treats physical work.
Do you know how your work comes in and is scheduled?
© 2014 CSG 15 Systems International, Inc. Confidential and Proprietary

16. Print Center: Continuous Delivery with Robots
How do you get your code in production?
© 2014 CSG 16 Systems International, Inc. Confidential and Proprietary

17. 9. Cadence & Synchronization
If you don’t force cadence and synchronization in your processes
and resources, nature will do it for you.
Program Increment Program Increment
Iterate,
Synchronize,
Release
Iterate,
Synchronize
Release
Manage new work injection!
Make unpredictable events like release planning predictable across teams.
Synchronize start and finish times to create a pull effect and prevent hidden
work from creating problems.
© 2014 CSG 17 Systems International, Inc. Confidential and Proprietary

18. 10. Reduce Batch Size
Once you have Learning, Infrastructure & Process in place:
Reduce Batch Size
PI/Release PI/Release
28 Weeks/14 Iterations 28 Weeks/14 Iterations
PI/Release PI/Release PI/Release PI/Release
14 Weeks 14 Weeks 14 Weeks 14 Weeks
Smaller/fewer things go through the system faster with less impact.
© 2014 CSG 18 Systems International, Inc. Confidential and Proprietary

19. More Practice & Smaller Batches
Where we were. Where we are. Where we want to be.
© 2014 CSG 19 Systems International, Inc. Confidential and Proprietary

20. 10 Techniques to Take Home
1. Accelerate Learning & Lean Thinking
2. Inverse Taylor Maneuver
3. Inverse Conway Maneuver
4. Shared Service Continuous Delivery
5. Environment Congruency & Practice
6. Application Telemetry
7. Visualize Your Work
8. Work Release & WIP Limits
9. Cadence & Synchronization
10. Reduce Batch Size
© 2014 CSG 20 Systems International, Inc. Confidential and Proprietary

21. Credits
 The Phoenix Project, Gene Kim, Kevin Behr, George Spofford
 Continuous Delivery, Jez Humble
 Scaled Agile Framework: www.scaledagileframework.com
 The Fifth Discipline, Peter Senge
 Martin Fowler, Technology Radar(Inverse Conway Maneuver)
 Cultural Cartography(I,T,E Shaped Resources):
http://culturecartography.wordpress.com/2012/07/26/busines
s-trend-e-shaped-people-not-t-shaped/
 Kevin Behr, Seven Traits Similar to DevOps:
http://www.kevinbehr.com/kevins-blog
© 2014 CSG 21 Systems International, Inc. Confidential and Proprietary

22. How can you help me?
 Problem Areas
• Standardizing applications at scale
- Proving the business case/savings
- Making progress quickly….
• Balancing standards with innovation
© 2014 CSG 22 Systems International, Inc. Confidential and Proprietary