AWS Code Services
•
14 j'aime•4,240 vues
Today’s cutting edge companies have software release cycles measured in days instead of months. This agility is enabled by the DevOps practice of continuous integration and delivery, which automates building, testing, and deploying all code changes. This automation helps you catch bugs sooner and accelerates developer productivity. In this session, we’ll share the processes followed by Amazon engineers and discuss how you can bring them to your company by using a set of application lifecycle management tools from AWS: the newly announced AWS CodeBuild service, AWS CodePipeline, and AWS CodeDeploy.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à AWS Code Services
Similaire à AWS Code Services (20)
Plus de Amazon Web Services
Plus de Amazon Web Services (20)
Dernier
Dernier (20)
AWS Code Services
- 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Chuck Meyer, Security Solutions Architect Nico Vautier, Solutions Architect Manager DevOps on AWS AWS Code Services
- 2. About Me Chuck Meyer cmmeyer@amazon.com Security Solutions Architect 3.5 years at AWS Security Automation / DevSecOps Previously Senior ProServe Consultant 20+ Years in Technology
- 3. • Integration tests with other systems • Load testing • UI tests • Penetration testing Release processes have four major phases Source Build Test Production • Check-in source code such as .java files. • Peer review new code • Compile code • Unit tests • Style checkers • Code metrics • Create container images • Deployment to production environments
- 4. Release processes levels Source Build Test Production Continuous integration Continuous delivery Continuous deployment
- 5. AWS Code* Services AWS CodePipeline AWS CodeCommit AWS CodeBuildAWS CodeDeploy
- 6. AWS Code Services Source Build Test Production Software Release Phases:
- 7. AWS Code Services Source Build Test Production Software Release Phases: AWS CodeCommit
- 8. AWS Code Services Source Build Test Production Software Release Phases: AWS CodeBuild
- 9. AWS Code Services Source Build Test Production Software Release Phases: Third Party Tooling
- 10. AWS Code Services Source Build Test Production Software Release Phases: AWS CodeDeploy
- 11. AWS Code Services Source Build Test Production Software Release Phases: EC2 On-Prem AWS CodeDeploy
- 12. AWS Code Services Source Build Test Production Software Release Phases: AWS CodePipeline
- 13. AWS Code Services Source Build Test Production Third Party Tooling Software Release Phases: AWS CodeCommit AWS CodeBuild AWS CodeDeploy AWS CodePipeline
- 14. Orchestrating build and deploy with a pipeline https://www.flickr.com/photos/seattlemunicipalarchives/12504672623/
- 15. Continuous delivery service for fast and reliable application updates Model and visualize your software release process Builds, tests, and deploys your code every time there is a code change Integrates with third-party tools and AWS AWS CodePipeline
- 18. Build CodeBuild AWS CodeBuild NotifyDevelopers Lambda TestAPI Runscope Sequential actions Deploy JavaApp Elastic Beanstalk Source Source GitHub CodePipeline MyApplication
- 19. Build CodeBuild AWS CodeBuild Staging-Deploy JavaApp Elastic Beanstalk Prod-Deploy JavaApp Elastic Beanstalk QATeamReview Manual Approval Manual Approvals Review CodePipeline MyApplication
- 20. 8. Retrieve build artifact EC2 instance Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy JavaApp Elastic Beanstalk Source Artifact S3 Build Artifact S3 5. Get source artifact 1. Get Changes 6. Store build artifact 3. Poll for Job 4. Acknowledge Job 7. Put Success 9. Deploy build artifact Elastic Beanstalk Web container Java App CodePipeline MyApplication
- 21. AWS service integrations Source Invoke Logic Deploy AWS Elastic Beanstalk Amazon S3 AWS CodeDeployAWS Lambda AWS CodeCommit AWS OpsWorks AWS CloudFormation
- 22. We have a strong partner list, and it’s growing Source Build Test Deploy
- 23. Extend AWS CodePipeline Using Custom Actions Update tickets Provision resources Update dashboards Mobile testing Send notifications Security scan
- 25. Secure, scalable, and managed Git source control Use standard Git tools Scalability, availability, and durability of Amazon S3 Encryption at rest with customer-specific keys No repo size limit Post commit hooks to call out to SNS/Lambda AWS CodeCommit
- 26. Source control in the cloud Secure Fully managed High availability Store anything
- 27. AWS CodeCommit git pull/push CodeCommit Git objects in Amazon S3 Git index in Amazon DynamoDB Encryption key in AWS KMS SSH or HTTPS
- 28. $ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli Cloning into 'aws-cli'... Receiving objects: 100% (16032/16032), 5.55 MiB | 1.25 MiB/s, done. Resolving deltas: 100% (9900/9900), done. Checking connectivity... done. $ nano README.rst $ git commit -am 'updated README' [master 4fa0318] updated README 1 file changed, 1 insertion(+) $ git push Counting objects: 3, done. Delta compression using up to 4 threads. Compressing objects: 100% (3/3), done. Writing objects: 100% (3/3), 297 bytes | 0 bytes/s, done. Total 3 (delta 2), reused 0 (delta 0) remote: To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-cli 4dacd6d..4fa0318 master -> master Same Git experience
- 29. Build & test your application https://secure.flickr.com/photos/spenceyc/7481166880
- 30. Fully managed build service that compiles source code, runs tests, and produces software packages Scales continuously and processes multiple builds concurrently You can provide custom build environments suited to your needs via Docker images Only pay by the minute for the compute resources you use Launched with CodePipeline and Jenkins integration AWS CodeBuild
- 31. How does it work? 1. Downloads source code 2. Executes commands configured in the buildspec in temporary compute containers (created fresh on every build) 3. Streams the build output to the service console and CloudWatch logs 4. Uploads the generated artifact to an S3 bucket
- 32. How can I automate my release process with CodeBuild? • Integrated with AWS CodePipeline for CI/CD • Easily pluggable (API/CLI driven) • Bring your own build environments • Create Docker images containing tools you need • Open source Jenkins plugin • Use CodeBuild as the workers off of a Jenkins master
- 33. buildspec.yml Example version: 0.1 environment_variables: plaintext: JAVA_HOME: "/usr/lib/jvm/java-8-openjdk-amd64" phases: install: commands: - apt-get update -y - apt-get install -y maven pre_build: commands: - echo Nothing to do in the pre_build phase... build: commands: - echo Build started on `date` - mvn install post_build: commands: - echo Build completed on `date` artifacts: type: zip files: - target/messageUtil-1.0.jar discard-paths: yes
- 34. buildspec.yml Example version: 0.1 environment_variables: plaintext: JAVA_HOME: "/usr/lib/jvm/java-8-openjdk-amd64" phases: install: commands: - apt-get update -y - apt-get install -y maven pre_build: commands: - echo Nothing to do in the pre_build phase... build: commands: - echo Build started on `date` - mvn install post_build: commands: - echo Build completed on `date` artifacts: type: zip files: - target/messageUtil-1.0.jar discard-paths: yes • Variables to be used by phases of build • Examples for what you can do in the phases of a build: • You can install packages or run commands to prepare your environment in ”install”. • Run syntax checking, commands in “pre_build”. • Execute your build tool/command in “build” • Test your app further or ship a container image to a repository in post_build • Create and store an artifact in S3
- 35. Building Your Code “Building” code typically refers to languages that require compiled binaries: • .NET languages: C#, F#, VB.net, etc. • Java and JVM languages: Java, Scala, JRuby • Go • iOS languages: Swift, Objective-C We also refer to the process of creating Docker container images as “building” the image. EC2
- 36. No Building Required! Many languages don’t require building. These are considered interpreted languages: • PHP • Ruby • Python • Node.js You can just deploy your code! EC2
- 37. Testing Your Code Testing is both a science and an art form! Goals for testing your code: • Want to confirm desired functionality • Catch programming syntax errors • Standardize code patterns and format • Reduce bugs due to non-desired application usage and logic failures • Make applications more secure
- 38. Where to Focus Your Tests: UI Service Unit 70% 20% 10%
- 39. What service and release step corresponds with which tests? UI Service Unit Third Party Tooling AWS CodeBuild BuildTest
- 40. Pricing • Pay by the Minute • Three compute types differentiated by the amount of memory and CPU resources: • Free tier of 100 build minutes Compute instance type Memory (GB) vCPU Price per build minute ($) build.general1.small 3 2 0.005 build.general1.medium 7 4 0.010 build.general1.large 15 8 0.020 *As of January 20 2017
- 41. Demo: 1. Start with a repository (github.com/awslabs/aws- codedeploy-sample-tomcat) 2. Add buildspec.yml 3. Create CodePipeline pipeline with a Source and Build stage 4. Do a build 5. Add a deploy stage 6. Do a full execution of the pipeline
- 53. Automates code deployments to any instance Handles the complexity of updating your applications Avoid downtime during application deployment Rollback automatically if failure detected Deploy to Amazon EC2 or on-premises servers, in any language and on any operating system Integrates with third-party tools and AWS AWS CodeDeploy
- 54. AWS CodeDeploy - Components Application: What you are deploying – A container for revisions Revision: A given version of your application Instance: Target instance for deployment Deployment Groups: Group of instances – A construct for environment segregation (Dev/QA/Prod, Blue/Green, A/B, etc.) AppSpec File: Describes actions that needs to be taken pre- or post-deployment Deployment: The action of deploying a new revision onto instances
- 55. AWS CodeDeploy - Application • The Application is the highest level container • Revisions will be attached to the Application
- 56. AWS CodeDeploy - Revisions Revisions are versions of your application They can be uploaded to: • Amazon S3 • GitHub Revisions are deployed onto Deployment Groups, which are groups of instance
- 57. AWS CodeDeploy – Instance Setup • Verify your instance has an IAM instance profile and verify the permissions allows it to participate in AWS CodeDeploy deployments • Tag the instance - or make sure it’s in an Auto Scaling Group • Install the agent (can be automated) • Verify the agent is running
- 58. AWS CodeDeploy – Setup Deployment Groups Group instances by: • Auto Scaling Groups • EC2 Tags • On-Premise tags Deploy to Deployment Groups independently from each other Could be used for “DevOps” constructs: • Dev/QA/Prod/etc. • Blue/Green • A/B Testing • Etc.
- 59. AWS CodeDeploy – Lifecycle Events AfterInstall ApplicationStart ValidateService ApplicationStop BeforeInstall Agent The agent goes through a series of steps before and after deployment These steps allow you to tightly control how your application is deployed For example, you may want to stop your application cleanly
- 60. AWS CodeDeploy – AppSpec File version: 0.0 os: linux files: - source: / destination: /var/www/html/WordPress hooks: ApplicationStop: - location: helper_scripts/stop_server.sh BeforeInstall: - location: deploy_hooks/install-apache.sh - location: deploy_hooks/install-mysql.sh AfterInstall: - location: deploy_hooks /configure_app.sh timeout: 30 runas: root • AppSpec.yml sits in your application’s source directory structure • Allows you to defines the hooks you want to use.
- 61. appspec.yml Example version: 0.0 os: linux files: - source: / destination: /var/www/html permissions: - object: /var/www/html pattern: “*.html” owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh
- 62. appspec.yml Example version: 0.0 os: linux files: - source: / destination: /var/www/html permissions: - object: /var/www/html pattern: “*.html” owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh • Remove/add instance to ELB • Install dependency packages • Start Apache • Confirm successful deploy • More! • Send application files to one directory and configuration files to another • Set specific permissions on specific directories & files
- 63. AWS CodeDeploy – Deployment Configuration One-at-a-time Half-at-a-time All-at-once Custom Blue / Green Deployments supported as of January 2017
- 65. Code* Tips and Tricks • All Code* products can(and should) be provisioned and managed with AWS CloudFormation! • You could literally store the CloudFormation templates that provision your Code* resources in CodeCommit and update them via CodePipeline (It’s like Code* Inception!) • Deep integration with IAM. You can assign permissions on who can commit code, approve manual approvals, deploy to certain deployment groups and more! • Integrate with AWS Lambda to do almost anything: • CodeCommit has Repository Triggers • CodeDeploy has Event Notifications • CodePipeline has native Lambda invoke AWS CodePipeline AWS CodeCommit AWS CodeBuildAWS CodeDeploy
- 67. AWS DevOps Blog
Notes de l'éditeur
- I want to take a moment to talk about different release processes. Each team’s release process takes a different shape to accommodate the needs of each team. Nearly all release processes can be simplified down to four stages – source, build, test and production. Each phase of the process provides increase confidence that the code being made available to customers will work in the way that was intended. During the source phase, developers check changes into a source code repository. Many teams require peer feedback on code changes before shipping code into production. Some teams use code reviews to provide peer feedback on the quality of code change. Others use pair programming as a way to provide real time peer feedback. During the Build phase an application’s source code is built and the quality of the code is tested on the build machine. The most common type of quality check are automated tests that do not require a server in order to execute and can be initiated from a test harness. Some teams extend their quality tests to include code metrics and style checks. There is an opportunity for automation any time a human is needed to make a decision on the code. The goal of the test phase is to perform tests that cannot be done on during the build phase and require the software to be deployed to a production like stages. Often these tests include testing integration with other live systems, load testing, UI testing and penetration testing. At Amazon we have many different pre-production stages we deploy to. A common pattern is for engineers to deploy builds to a personal development stage where an engineer can poke and prod their software running in a mini prod like stage to check that their automated tests are working correctly. Teams deploy to pre-production stages where their application interacts with other systems to ensure that the newly changed software work in an integrated environment. Finally code gets deployed to production. Different teams have different deployment strategies though we all share a goal of reducing risk when deploying new changes and minimizing the impact if a bad change does get out to production. Each of these steps can be automated without the entire release process being automated. There are several levels of release automation that I’ll step through.
- Continuous Integration Continuous Integration is the practice of checking in your code to the continuously and verifying each change with an automated build and test process. Over the past 10 years Continuous Integration has gained popularity in the software community. In the past developers were working in isolation for an extended period of time and only attempting to merge their changes into the mainline of their code once their feature was completed. Batching up changes to merge back into the mainline made not only merging the business logic hard, but it also made merging the test logic difficult. Continuous Integration practices have made teams more productive and allowed them to develop new features faster. Continuous Integration requires teams to write automated tests which, as we learned, improve the quality of the software being released and reduce the time it takes to validate that the new version of the software is good. There are different definitions of Continuous Integration, but the one we hear from our customers is that CI stops at the build stage, so I’m going to use that definition. Continuous Delivery Continuous Delivery extends Continuous Integration to include testing out to production-like stages and running verification testing against those deployments. Continuous Delivery may extend all the way to a production deployment, but they have some form of manual intervention between a code check-in and when that code is available for customers to use. Continuous Delivery is a big step forward over Continuous Integration allowing teams to be gain a greater level of certainty that their software will work in production. Continuous Deployment Continuous Deployment extends continuous delivery and is the automated release of software to customers from check in through to production without human intervention. Many of the teams at Amazon have reached a state of continuous deployment. Continuous Deployment reduces the time for your customers to get value from the code your team has just written, with the team getting faster feedback on the changes you’ve made. This fast customer feedback loop allow you to iterate quickly, allowing you to deliver more valuable software to your customers, quicker.
- https://www.flickr.com/photos/seattlemunicipalarchives/12504672623/
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- Let’s take a look at an example Pipeline. I’ve created a simple 3 stage Pipeline to talk though my example. Source actions are special actions. They continuously poll the source providers, such as GitHub and S3, in order to detect changes. Once a change is detected, the new pipeline run is created and the new pipeline begins its run. The source actions retrieve a copy of the source information and place it into a customer owned S3 bucket. Once the source action is completed, the Source stage is marked as successful and we transition to the Build stage. In the Build Stage we have one action, Jenkins. Jenkins was integrated into CodePipeline as a CustomAction and has the same lifecycle as all custom actions. Talk through interaction Once the build action is completed, the Build stage is marked as successful and we transition to the Deploy stage The Deploy stage contains one action, an AWS Elastic Beanstalk deployment action. The Beanstalk action retrieves the build artifact from the customer’s S3 bucket and deploys it to the Elastic Beanstalk web container.
- We have partnered with popular source, build, test and deployment systems to provide out-of-the-box integrations. Jenkins, CloudBees and Solano offer CI services for build stages BlazeMeter, Apica, HP StormRunner and Runscope are load testing partners. GhostInspector is a User Interface Testing partner GitHub is a source code partner Xebia Labs is a deployment partner.
- https://secure.flickr.com/photos/spenceyc/7481166880
- The effort you put into the testing triangle should not be evenly distributed! Many experts in the industry recommend a 70,20,10 mix. (will need sources)
- The effort you put into the testing triangle should not be evenly distributed! Many experts in the industry recommend a 70,20,10 mix. (will need sources)
- https://secure.flickr.com/photos/simononly/15386966677
- https://secure.flickr.com/photos/simononly/15386966677
- https://secure.flickr.com/photos/dullhunk/202872717/