Principles and Practices in Continuous Deployment at Etsy
Entaggle: an Agile Software Development Case Study
1. Entaggle.com A Case Study in Agile Software Development with Rails Give Recognition Get Recognition Tag, you’re it. Elisabeth Hendrickson esh@qualitytree.com Last updated April 11, 2011 This presentation is licensed under the Creative Commons Attribution 3.0. Permission granted to distribute and excerpt. Attribute this work to Elisabeth Hendrickson, Quality Tree Software, Inc.
2. Overview First code written in October 2010. Announced to the public March 1, 2011. Automated continuous deploy in place from the beginning. Test-driven from the beginning. Built with Rails using Devise, Paperclip, Kaminari, HAML, SASS, JQueryUI with Rspec & Cucumber for testing and Fixjour for test data creation. Build w/ all tests currently takes 13 mins. …and it’s a “spare time” project.
3. Ch-ch-changes Two complete UI rewrites since the first placeholder UI put in place. (3 UI designs total since the beginning.) The core “Tagging” user flow has changed uncountable times based on feedback. New features added at least weekly. Backlog reprioritized frequently based on user feedback.
5. Reprioritizing I thought these things would be critically important: Moderation & admin features User “credibility” and Tag “prestige” I was wrong. Turns out these things were more important: More engaging UI/flow “Since you were last here…” Instrumenting & Metrics
6. Key Factors for Agility Frequent checkins Automated “checks” at both unit & acceptance levels Automated deploy to Staging Relentless attention to removing sources of friction
7. My Workflow Develop until green & clean. Then check in. Pick up the next story. Back to start. If ready, manual deploy Build kicks off. If build passes, automatic deploy to Staging Exploratory Testing Go see it in multiple browsers. My CI Box (running Hudson) PRODUCTION STAGING
9. Automated “checks” 239 Rspec examples (includes unit & integration tests) 90 Cucumber scenarios with 684 steps Over 93% code coverage All major user flows covered in Cucumber scenarios
10. Code Coverage (Rcov) WHY THE DIP? For no apparent reason, Rcov suddenly started counting coverage for standard ruby libraries. Edited the rake task to explicitly exclude them & coverage went back to normal.
11. Automated Deploy If the tests pass, Hudson CI server uses a shell command to deploy: git pull && bundle install && rake db:migrate && touch tmp/restart.txt (in the past I’ve used Capistrano; may decide I need it again. But not yet.)
12. Examples of Relentlessly Removing Sources of Friction Regularly refactoring to remove crufty code. Avoiding the generators: they spew large amounts of templated code. Investing in making deploy to staging automatic, smooth, & secure w/ RSA key authentication. Investing in making deploy to production trivially easy. Creating shell command aliases for frequent command sequences.
13. Results… Stats: 93 user stories completed including 22 user-requested enhancements 10 user reported bugs (only 1 critical in production). 41 self-reported bugs. 46 of 51 bugs closed. All 5 remaining open are self-reported. Production busted for ~3mins once on a bad deploy. Otherwise it’s been up. Almost non-existent “WTF?!?!” incident count.
14. Lessons Learned (and Reinforced) The tests are totally worth it, even when it is tempting to skip them. However, high code coverage != well tested. Exploratory Testing is essential. Building up an inventory of unused acceptance tests was an unfortunate waste. Splitting stories until they represent <= 2 days worth of work is essential for keeping momentum.
15. Appreciations… …to the following contributors: Pat Maddox, Dale Emery, Joseph Wilk, Michael Dowling, Richard Scherek, and LasseKoskela all paired with me on writing code. Andy Hohenner helped with System Administration. Yves Hannoulle has been an ardent supporter and evangelist. Marcus Smith created our cheerful “e” logo and Darren McMillan created the favicon. The Entaggle community has helped me steer the project with great suggestions. The Weekend Testers kept me busy with bug reports.