Fixing Security by Fixing Software Development Using Continuous Deployment Do you have an effective release cycle? Is your process long and archaic? Long release cycle are typically based on assumptions we haven't seen since the 1980s and require very mature organizations to implement successfully. They can also disenfranchise developers from caring or even knowing about security or operational issues. Attend this session to learn more about an alternative approach to managing deployments through Continuous Deployment, otherwise known as Continuous Delivery. Find out how small, but frequent changes to the production environment can transform an organization’s development process to truly integrate security. Learn how to get started with continuous deployment and what tools and process are needed to make implementation within your organization a (security) success.

1. Fixing Security by Fixing Software Development
Using Continuous Deployment
Nick Galbreath, VP Eng IPONWEB, http://www.iponweb.com/
@ngalbreath http://www.client9.com/
May 14-15, San Francisco, California

2. Follow Along
Latest version posted online:
http://slidesha.re/144OIv3
http://www.client9.com/

3. Nick Galbreath (@ngalbreath)
! Spoken at Black Hat, DEFCON, OWASP
! Author of libinjection: advanced SQLi
detection used in > 2 WAFs, 1 Honeypot
! Book on cryptography
! but really...
! Engineering Management and Software
Development for high growth startups.
! Personal site http://www.client9.com/

4. IPONWEB
! customized online advertising
infrastructure and exchanges
! engineering offices in Moscow, with
business offices in London, New York and
Tokyo.

5. Original Abstract
Fixing Security by Fixing Software Development Using Continuous
Deployment
Do you have an effective release cycle? Is your process long and archaic? Long
release cycle are typically based on assumptions we haven't seen since the
1980s and require very mature organizations to implement successfully. They
can also disenfranchise developers from caring or even knowing about security
or operational issues. Attend this session to learn more about an alternative
approach to managing deployments through Continuous Deployment, otherwise
known as Continuous Delivery. Find out how small, but frequent changes to the
production environment can transform an organization’s development process
to truly integrate security. Learn how to get started with continuous deployment
and what tools and process are needed to make implementation within your
organization a (security) success.

6. Well that's a bold statement...
"Fixing Security by Fixing Development
Using Continuous Deployment"

7. and here's another
For web applications, our release-based
software development lifecycle is still
based on a pre-Internet model and is
harmful to organizations and
particularly harmful for security.

8. What needs fixing?
! SQLi dropped from #8 to #14 in the latest
White Hat "The State of Web Security"
report. Good news, right?
! This means SQLi is only 7% of websites.
That's 1 in 15. And this is #14
vulnerability!
! And time to fix was on average 196 days.
That's embarrassing.
Veracode claims 32% of incoming web applications have SQLi
https://info.veracode.com/state-of-software-security-report-volume5.html
https://reg.whitehatsec.com/WPstats0513

9. Even worse...
! Number 1 driver to fix security problems...
compliance.
! Number 1 reason to not fix security is...
compliance.
! Not..
! keeping our employees and customers safe
! protecting corporate interests.
! improving quality
! being good at what we do.

10. Security Products #1 .. in security bugs
VeraCode: State of
Software Security, V4
December 2011

11. Let's Just Give Up
! “You could spend all your resources chasing such
things as this,” William Ribich, the former president of
Technology Solutions Group [ QintelIQ ], said in an
interview in January. Ribich, who retired in November 2009,
shortly after the discovery of a major data theft, said he
needed to balance the uncertain risk that the hackers could
use what they stole against a growing shopping list of
security products and consulting fees.
! "You finally have to reach a point where you say
’let’s move on,’” he said.
http://www.bloomberg.com/news/2013-05-01/china-cyberspies-outwit-u-s-stealing-military-secrets.html

12. I would call that broken
! But preventing SQLi isn't a technically
hard problem.
! And most security patches are very small.
! How did we get here?

13. Software Product Model
! Code flows between functional groups
! Product Managers spec code
! Engineers write code
! QA engineers tests code
! Release engineers package code
! Operations runs code
! ... and Security does something too.

14. High Distribution Cost
! The Software Product Model is designed
for software where the cost of distribution
is high. "High" might be financial, risk,
time, resources, customer annoyance.
! Retail, physical product, CD/DVD
! Embedded of Exotic Hardware
! Safety, Medical or Defense Systems
! Operating Systems (desk or phone)
! Homework (1-time deploy)

15. Software Product Lifecycle

16. Change to Production

17. Web Applications Year 2000
! Mostly followed Software Product Model
since that's all we knew.
! High barrier to entry
! Specialized Hardware, Software and
People needed to get started.
! Lots of engineering needed to keep things
running.
! (side note: CERT/CVE started in 1999)

18. True Story #1
! "Can't push out the spelling error fix – it's
too risky"
! "That code as already been through QA,
it's locked down."
! "Product has to prioritize that change, else
we aren't touching it."

19. True Story #2
! We'll do an iteration, where we try to fix as
many things as possible.
! This won't be a scheduled iteration, it will
be done because things are so bad
! So the spelling error will get fixed...
uhh, who knows when.

20. Web Applications 2013
! Almost no barrier to entry
! Commodity hardware
! Programming not that hard
! Scaling problems can
be mostly outsourced
(mostly)

21. Cost of Distribution 2013
! Frequently no compile step
or it's very fast.
! Moving to production a few kilobytes or
megabytes of code over 1Gbps, 10Gbps
link.
! In other words... free

22. Failure is very different however
! Most web applications are data-driven.
! Frequently have social features, APIs,
user-generated content.
! Failures might be due to algorithmic
problems... but...
! Most likely to due to user input, bad data
in database or operational load.
! this means data in past can cause
problems in the future.

23. Releases and Problems
! When a web-release goes out, and has
problems....
! Next week is spent tracking down who
changed what, where.
! Re-QA
! Re-Push
! meanwhile new code is piling up.

24. When SPM meets Web Apps
! A long time between code being written
and code being released.
! Might be weeks or months
! Feedback loop between code-in-dev and
code-in-production is broken
! When security or bug reports come in, the
author is likely on a different project.

25. Hypothesis
! It is impossible to simulate the production
environment in development, either due to
operational differences or data differences.
! No amount of QA or Security Testing can
prove you don't have bugs, vulnerabilities,
or cause severe operational problems.
! You have bugs and vulnerabilities,
right now, in your application.

26. Impedance Mismatch
! Easy to write code, +
! Long release cycles +
! Security as an end-of-line or out of band
process ==
! no one cares

27. So the Answer is...
! Going slower? I'm sure your boss will love
that suggestion
! More steps and process? In other words,
slower.
! Asking for more people? Sure but good
luck hiring them. Doesn't scale.
! Asking for more products? Since the
others have worked so well.

28. Continuous Deployment
! Also known as Continuous Delivery.
! A System of Software Production
Characterized by Numerous Small
Changes the Production Environment,
initiated by the author of the change.
You change it, you push it to prod.

29. Deployment != Feature Release

30. "Writing Software"
! Software Developers think their job is
writing software.
! And so, they love to make things perfect
before anyone else sees it.
! Impolite: "data hiding"
! code is hiding on developer's computer
! or on some branch
! in other words invisible until it's ready.

31. Actually
! The software engineer's job is actually
writing running software, that works well.
! This idea is so alien, that companies have
to remind the engineers of this.

32. Rackspace Haiku
writing code is hard
if you cannot deploy
it does not matter
@paulvx from DevOpsDays Austin 2013

33. Facebook
"Move Fast and
Break Things....
Except "Push" (deployment system)
via http://mitadmissions.org/blogs/entry/
move_fast_and_break_things

34. Etsy Mantra
Just Ship

35. Today's goal
! but for today the goal is getting the
developer to care about their code
in production.
! If you don't have that, I don't think you can
really solve security problems.

36. How does this work?
! Really? Developers push their own code
out?
! How is this not a disaster.
! How is this not a security disaster?

37. The Deploy Button
! What is you had a button that said
"DEPLOY"
! That pushed to production, whatever is
current in your source control system.
! And took about a minute
! The change and who pressed the button is
logged, but that's it.

38. Part 1: Fear
! No one is going to push it ;-)
! Meanwhile code is piling up
Real example: A new hire I had at Etsy was afraid
of deploying an HTML change that they made.
"But I don't want to break the site!"

39. Part 2: First Push
! Someone brave will press the button
! And very likely the site will explode, and a
rollback will need to be done.
! They'll know since someone else will have
told them.

40. Part 3: With Graphs
! Let's get all those operational graphs out
in the open. And put them right next to
the button.

41. Part 4: Push #2
! Repush
! Site might still explode
! But the developer is aware and can
rollback.

42. Take 5: Isolation
! Hmmm, the developer notices that in the
change set, a million things are going out.
! Maybe just pushing out a smaller change
will help isolate the issue.

43. Take 6: Success!
! Yes, the developer just pushed out some
code and made the site better.
! The secret about continuous deployment
is small changes that can be easily
understood.

44. Take 7: Dark Pushes
! Now we got some bugs fixed, let's push a
feature.
! First let's push out all the supporting files.
Since they aren't being called, they do
nothing and are safe to push out.
! Now everyone can see them

45. Take 8: Getting the feature live
! Instead of "all at once", we slowly ramp up
a feature.
! if (user_id % 20 == 0):
do new feature;
! we change change the percentage easily
with another code push.
! or turn it down. Much nicer change log.
! While the site didn't explode, it's hard to
see if the feature is being used or not.

46. Take 9: Application Level Graphs
! Allow developers to instrument their code
so they can see what is happening in
production.
! Enter StatsD and other UDP-based tools
! Enter centralized logging and in-
application method to make it easy to log
problems.

47. Take 10: Communication
! So far good for one developer.
! To scale up, you'll need a system to allow
developers.
! IRC-like tools work well (e.g. "the push
channel") – skype, jabber, hangouts, etc

48. Along the way
! Expose production logs to developers
! Add in a staging-step where the code
goes to faction of the cluster, so
developers can test with real traffic
! Try to make development closer to prod.
! Make "smoke tests" to catch basic errors
! Add syntax checkers to eliminate obvious
issues.
! Use static analysis to find bugs

49. Mistakes will happen
! Do postmortem analysis
! Everyone thought they were doing the
right thing at the time.
! "How can the environment be changed to
prevent this" and build tools to enforce it.
! (Rarely can you truly change people)

50. Sounds good,
but what about...

51. That guy who pushes at 3am
! Courtesy and convention will converge
very quickly when the site goes down at
3am and the developer starts getting
calls ;-)
! Of hours pushes of course can happen,
when they notify operations.

52. What About Code Reviews?
! Yes, please do them.
! Nothing here prevents code reviews.
! In fact code reviews are easier since
! they are small
! they are in mainline not some branch

53. What about Security Reviews
! Please do them.
! Nothing here eliminates architectural
planning or review.
! This actually doesn't change the SDLC
very much.

54. What about Agile Methods
! (everyone seems to have a different idea of
what Agile is but..)
! Agile methodologies typically work to
improve the business spec / development
cycle. (are you building what the customer
wants)
! But doesn't address code deployment.
! They are complimentary practices.

55. What about Customer Service?
! "Don't they freak out with all the
changes?"
! Remember: deployment != feature release
! Most deployments do very little from the
customer point of view
! Feature releases (frequently controlled by
ramp-ups or flags) always needs to be
coordinated with product and customer
service.

56. What about Compliance? PCI?
! Let me tell you about compliance...
! mechanism not policy

57. Back to Security

58. Obvious Benefit to Security
! Security patches can go out quickly
! You know this since they are now just part
of a normal development cycle.

59. More Importantly
! That Engineer who previously didn’t push
code is now sensitized that their code has
consequences and are responsive and
empowered to fix it.
! It’s amazing how interested engineers
become in security when you find
problems with their code when they are
able to fix quickly themselves.

60. Hack The Stack
! A side effect of this you now have tools to
repurpose for security and
monitoring of production
! Note that most changes are not security
problems.

61. Logging
! Due to allow developers to see application
logging, it's now very easy to instrument
the application to log security events.
! Or add logs to times when you are under
attack.

62. Graphing
! Make dashboards of
! SQLi and XSS attacks
! Every type of log-in failure
! Core Dumps
! Database Syntax Errors

63. Static Analysis
! You now have a place to insert them.
! Work with QA group to add more code
quality tests.

64. Post-Commit Checks
! Alert on when sensitive areas of the code
are changed (auth)
! Alert on crypto usage (why is developer
using MD5.. hmmm)
! Alert your programming languages
"dangerous functions"
This allows you to engage the developer at the start of the cycle.

65. Faster is Better
! You could do most of this in a normal
release-cycle software lifecycle.
! The difference is you are finding problems
at the start instead of 10m before the
launch and telling everyone to stop.
! The feedback loop works.

66. New Roles, Less Silos
! Developers: works with operations
! QA: works on building systems for testing,
to empower others to write better tests
! Release Engineering: tools to enable code
to flow faster
! Security: in-house consultancy, secure-by-
default architecture, monitoring

67. Getting Started

68. Goal: 50% reduction in deploy times
times
! Whatever your state of deployment is, no
matter how many people are involved, no
matter how long it currently takes, make a
goal of cutting it in half.
! This is an easy sell to management just on
cost basis.
! Everything else flows from this.

69. Mechanism not Policy
! Strive for the fastest deployment mechanism for
possible
! But you define the "continuous" in Continuous
Deployment
! Yes, Etsy was 60+ deploys per day, with each having
multiple authors.
! Current gig? we have rules of no more than 3 per week
since our customer have asked for that, and only
deployed at "low-tide"

70. In Other Contexts

71. In other contexts: Operations
! How fast can you deploy OS changes to
you production environment?

72. In other contexts: IT
! How fast can you deploy desktop
changes?

73. In other contexts: software product
! here "production" might be getting code
into the main branch and running
automated build / test.
! It's the flow of code: little changes vs big.

74. In other contexts: silicon
! Continuous deployment already done for
silicon! wut?
! Only small changes, with tests are allowed
to be committed!
! Big changes are rejected.
Learned the hard way that big changes are
completely unmanageable

75. One more thing...

76. Your Attackers Use
Continuous Deployment

77. Continuous Deployment
Start with 50% Reduction
Build the Deploy Button
Nick Galbreath ★ @ngalbreath ★ nickg@client9.com ★ http://www.client9.com/
http://slidesha.re/144OIv3

78. © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it
should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO
WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Fixing Security by Fixing Software Development Using Continuous Deployment
Nick Galbreath ★ @ngalbreath ★ http://www.client9.com ★ nickg@client9.com