Building CI-CD Pipelines for Serverless Applications

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Building CI/CD Pipelines for
Serverless Applications
Balaji Iyer –Infrastructure Architect, AWS Professional Services

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Serverless application
SERVICES (ANYTHING)
Changes in
data state
Requests to
endpoints
Changes in
resource state
EVENT SOURCE FUNCTION
Node.js
Python
Java
C#

Understanding “CI/CD”
Source Build Test Production
Continuous integration
Continuous delivery
Continuous deployment

Common Lambda use cases
Web
Applications
• Static
websites
• Complex web
apps
• Packages for
Flask and
Express
Data
Processing
• Real time
• MapReduce
• Batch
Chatbots
• Powering
chatbot logic
Backends
• Apps &
services
• Mobile
• IoT
</></>
Amazon
Alexa
• Powering
voice-enabled
apps
• Alexa Skills
Kit
IT
Automation
• Policy engines
• Extending
AWS services
• Infrastructure
management

CI/CD for Serverless Applications
There are a number of different paradigms we need to take
into account when doing CI&CD for serverless applications:
• Lambda functions are a unit of deployment
• We’ll typically have multiple Lambda functions per
application
• Each function will have an event trigger
• Could be shared or unique to each function
• A serverless application is typically a combination of
Lambda + other AWS services

CI/CD for Serverless Applications
We’ll want to deliver our Serverless application via a
traditional development “pipeline”:
• Pipeline initiated after code is committed to a repository
• ”Built”, tested, and verified at the code level exactly once
• Aim for single artifact per deploy
• Integration tested at functional and “end to end” levels
• Deployed to independent environments for each stage of this
process
• Allow for those independent environments to be deployed exactly
the same way across infrastructure + application

Tools for CI/CD
• IDE/text editor
• source revision control
• build tool
• testing tools
• deploy tools
• infrastructure management tools
• lifecycle automation tools

Tools for CI/CD
• IDE/text editor (BYO-IDE)
• source revision control (GitHub, CodeCommit, etc)
• build tool
• testing tools
• deploy tools
• infrastructure management tools
• lifecycle automation tools

AWS CodeBuild
Fully managed build service that compiles source code,
runs tests, and produces software packages
Scales continuously and processes multiple builds
concurrently
You can provide custom build environments suited to
your needs via Docker images
Only pay by the minute for the compute resources you
use
Launched with CodePipeline and Jenkins integration
New: Can be used as a “Test” action in CodePipeline

version: 0.1
environment_variables:
plaintext:
"INPUT_FILE": "saml.yaml”
"S3_BUCKET": ""
phases:
install:
commands:
- npm install
pre_build:
commands:
- eslint *.js
build:
commands:
- npm test
post_build:
commands:
- aws cloudformation package --template $INPUT_FILE --s3-
bucket $S3_BUCKET --output-template post-saml.yaml
artifacts:
type: zip
files:
- post-saml.yaml
- beta.json
buildspec.yml Example

version: 0.1
environment_variables:
plaintext:
"INPUT_FILE": "saml.yaml”
"S3_BUCKET": ""
phases:
install:
commands:
- npm install
pre_build:
commands:
- eslint *.js
build:
commands:
- npm test
post_build:
commands:
- aws cloudformation package --template $INPUT_FILE --s3-
bucket $S3_BUCKET --output-template post-saml.yaml
artifacts:
type: zip
files:
- post-saml.yaml
- beta.json
• Variables to be used by phases of
build
• Examples for what you can do in
the phases of a build:
• You can install packages or run
commands to prepare your
environment in ”install”.
• Run syntax checking,
commands in “pre_build”.
• Execute your build
tool/command in “build”
• Test your app further or ship a
container image to a repository
in post_build
• Create and store an artifact in S3
buildspec.yml Example

Building a deployment package
Node.js & Python
• .zip file consisting of
your code and any
dependencies
• Use npm/pip to
install libraries
• All dependencies
must be at root level
Java
• Either .zip file with all
code/dependencies,
or standalone .jar
• Use Maven / Eclipse
IDE plugins
• Compiled class &
resource files at root
level, required jars in
/lib directory
C# (.NET Core)
• Either .zip file with all
code/dependencies,
or a standalone .dll
• Use NuGet /
VisualStudio plugins
• All assemblies (.dll)
at root level

Establish our testing/validation model
We want to make sure our code:
• is without syntax issues
• meets company standards for format
• compiles
• is sufficiently tested at the code level via unit tests
We want to make sure our serverless service:
• functions as it is supposed to in relation to other components
• has appropriate mechanisms to handle failures up or down stream
We want to make sure our entire application/infrastructure:
• functions end to end
• follows security best practices
• handles scaling demands

Testing tools
Code Inspection/Test Coverage:
• Landscape - https://landscape.io/ (only for Python)
• CodeClimate - https://codeclimate.com/
• Coveralls.io - https://coveralls.io/
• Codecov – https://codecov.io/
Mocking/stubbing tools:
• https://github.com/atlassian/localstack - “A fully functional local AWS cloud stack. Develop and test
your cloud apps offline!”
• Includes:
• https://github.com/spulec/moto - boto mock tool
• https://github.com/mhart/dynalite - DynamoDB testing tool
• https://github.com/mhart/kinesalite - Kinesis testing tool
• more!
API Interface/UI testing:
• Runscope - https://www.runscope.com/ - API Monitoring/Testing
• Ghost Inspector - https://ghostinspector.com/ - Web interface testing

AWS Serverless Application Model (SAM)
CloudFormation extension optimized for
serverless
New serverless resource types: functions, APIs,
and tables
Supports anything CloudFormation supports
Open specification (Apache 2.0)
https://github.com/awslabs/serverless-application-model

SAM template
AWSTemplateFormatVersion: '2010-09-09’
Transform: AWS::Serverless-2016-10-31
Resources:
GetHtmlFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: s3://sam-demo-bucket/todo_list.zip
Handler: index.gethtml
Runtime: nodejs4.3
Policies: AmazonDynamoDBReadOnlyAccess
Events:
GetHtml:
Type: Api
Properties:
Path: /{proxy+}
Method: ANY
ListTable:
Type: AWS::Serverless::SimpleTable

AWSTemplateFormatVersion: '2010-09-09’
Transform: AWS::Serverless-2016-10-31
Resources:
GetHtmlFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: s3://sam-demo-bucket/todo_list.zip
Handler: index.gethtml
Runtime: nodejs4.3
Policies: AmazonDynamoDBReadOnlyAccess
Events:
GetHtml:
Type: Api
Properties:
Path: /{proxy+}
Method: ANY
ListTable:
Type: AWS::Serverless::SimpleTable
Tells CloudFormation this is a SAM
template it needs to “transform”
Creates a Lambda function with the
referenced managed IAM policy,
runtime, code at the referenced zip
location, and handler as defined.
Also creates an API Gateway and
takes care of all
mapping/permissions necessary
Creates a DynamoDB table with 5
Read & Write units
SAM template

SAM template
From: https://github.com/awslabs/aws-serverless-samfarm/blob/master/api/saml.yaml
<-THIS
BECOMES THIS->

AWS commands – Package & Deploy
Package
•Creates a deployment package (.zip file)
•Uploads deployment package to an Amazon S3 bucket
•Adds a CodeUri property with S3 URI
Deploy
•Calls CloudFormation ‘CreateChangeSet’ API
•Calls CloudFormation ‘ExecuteChangeSet’ API

SAM Template Capabilities
• Can mix in other non-SAM CloudFormation resources in the
same template
• i.e. S3, Kinesis, Step Functions
• Supports use of Parameters, Mappings, Outputs, etc
• Supports Intrinsic Functions
• Can use ImportValue
(exceptions for RestApiId, Policies, StageName attributes)
• YAML or JSON

Template File
Defining Stack
Source
Control
Dev
Test
Prod
Use the version
control system of
your choice to
store and track
changes to this
template
Build out multiple
environments, such
as for Development,
Test, Production and
even DR using the
same template,
even across
accounts
Many Environments from One Template

Configure multiple environments
A good developer knows they need different environments for building,
testing, and running their application!
Why?
• Avoid overlapping usage of resources
• Safely test new code without impacting your customers
• Safely test infrastructure changes
How?
• AWS Account strategies
• Using Infrastructure as Code tools
• Using Variables unique to each environment
• Automating application delivery/testing

Two popular AWS account strategies:
Same account, different stacks:
+ Easier management of
resources
+ Easier visibility via
management/monitoring tools
- Can be harder to create
permission/access separation
Better for smaller teams/individuals
Configure multiple environments
Multiple accounts:
+ Assured separation of permissions
and access
+ Resource limits per account to
control usage
- Overhead of managing multiple
accounts and controls between them
Better for larger teams/companies
!! Check out AWS Organizations

Lambda Environment Variables
• Key-value pairs that you can dynamically pass to your function
• Available via standard environment variable APIs such as
process.env for Node.js or os.environ for Python
• Can optionally be encrypted via KMS
– Allows you to specify in IAM what roles have access to the keys to
decrypt the information
• Useful for creating environments per stage (i.e. dev, testing,
production)

API Gateway Stage Variables
• Stage variables act like environment variables
• Use stage variables to store configuration values
• Stage variables are available in the $context object
• Values are accessible from most fields in API Gateway
• Lambda function ARN
• HTTP endpoint
• Custom authorizer function name
• Parameter mappings

Lambda and API Gateway Variables + SAM
Parameters:
MyEnvironment:
Type: String
Default: testing
AllowedValues:
- testing
- staging
- prod
Description: Environment of this stack of
resources
SpecialFeature1:
Type: String
Default: false
AllowedValues:
- true
- false
Description: Enable new SpecialFeature1
…
#Lambda
MyFunction:
Type: 'AWS::Serverless::Function'
Properties:
…
Environment:
Variables:
ENVIRONMENT: !Ref: MyEnvironment
Spec_Feature1: !Ref: SpecialFeature1
…
#API Gateway
MyApiGatewayApi:
Type: AWS::Serverless::Api
Properties:
…
Variables:
ENVIRONMENT: !Ref: MyEnvironment
SPEC_Feature1: !Ref: SpecialFeature1
…

AWS Systems Manager – Parameter Store
• Centralized store to manage your
configuration data
• plain-text or encrypted with KMS
• Can be secured with IAM
• Calls recorded in CloudTrail
• Available via API/SDK
• Useful for: centralized environment
variables, secrets control, feature
flags
• from __future__ import print_function
• import json
• import boto3
• ssm = boto3.client('ssm', 'us-east-1')
•
• def get_parameters():
• response = ssm.get_parameters(
• Names=['LambdaSecureString'],Wit
hDecryption=True
• )
• for parameter in
response['Parameters']:
• return parameter['Value']
•
• def lambda_handler(event, context):
• value = get_parameters()
• print("value1 = " + value)
• return value # Echo back the first
key value

Continuous delivery service for fast and reliable
application updates
Model and visualize your software release process
Builds, tests, and deploys your code every time
there is a code change
Integrates with third-party tools and AWS
AWS CodePipeline

Delivery via CodePipeline
Pipeline flow:
1. Commit your code to a source code repository
2. Package/Test in CodeBuild
3. Use CloudFormation actions in CodePipeline to
create or update stacks via SAM templates
Optional: Make use of ChangeSets
4. Make use of specific stage/environment
parameter files to pass in Lambda variables
5. Test our application between stages/environments
Optional: Make use of Manual Approvals

An example minimal Developer’s pipeline:
MyBranch-Source
Source
CodeCommit
MyApplication
Build
test-build-source
CodeBuild
MyDev-Deploy
create-changeset
AWS CloudFormation
execute-changeset
AWS CloudFormation
Run-stubs
AWS Lambda
This pipeline:
• Three Stages
• Builds code artifact
• One Development environment
• Uses SAM/CloudFormation to
deploy artifact and other AWS
resources
• Has Lambda custom actions for
running my own testing functions

Via referenced parameter file:
CodePipeline + CloudFormation Parameters
Via Parameter Overrides:

Via Parameter Overrides:Pros:
• Allows Developers to update and
provide parameters via file in the code
repository
• Easier to change and iterate via
deployment
Cons:
• Potentially harder to control
security/confidential information passed
in

Via Parameter Overrides:
Pros:
• Tighter control over parameters
passed in
• Can restrict access to information
based on visibility to CodePipeline
and CloudFormation
Cons:
• Modification requires a change to the
pipeline and a re-execution
• Harder to track the changes to these
values unless you are tracking them
via CloudFormation to manage the
pipeline(as an example)

Source
Source
CodeCommit
MyApplication
An example minimal Production pipeline:
Build
test-build-source
CodeBuild
Deploy Testing
create-changeset
AWS
CloudFormation
execute-changeset
AWS
CloudFormation
Run-stubs
AWS Lambda
Deploy Staging
create-changeset
AWS
CloudFormation
execute-changeset
AWS
CloudFormation
Run-API-test
Runscope
QA-Sign-off
Manual Approval
Review
Deploy Prod
create-changeset
AWS
CloudFormation
execute-changeset
AWS
CloudFormation
Post-Deploy-Slack
AWS Lambda
This pipeline:
• Five Stages
• Builds code artifact
• Three deployed to “Environments”
• Uses SAM/CloudFormation to
deploy artifact and other AWS
resources
• Has Lambda custom actions for
running my own testing functions
• Integrates with a 3rd party
tool/service
• Has a manual approval before
deploying to production

Introducing AWS CodeStar

aws.amazon.com/serverless

Additional Resources
Serverless Application Model (SAM) - https://github.com/awslabs/serverless-
application-model
Learn more:
AWS Lambda: https://aws.amazon.com/lambda
Amazon API Gateway: https://aws.amazon.com/api-gateway
Products that helped us today:
CloudFormation: https://aws.amazon.com/cloudformation
CodePipeline: https://aws.amazon.com/codepipeline
CodeBuild: https://aws.amaz.com/codebuild

?
https://secure.flickr.com/photos/dullhunk/202872717/

Building CI-CD Pipelines for Serverless Applications

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Building CI-CD Pipelines for Serverless Applications

Similaire à Building CI-CD Pipelines for Serverless Applications (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

Building CI-CD Pipelines for Serverless Applications

Notes de l'éditeur