AWS Serverless Video Workflows

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Building Serverless Video Workflows
U s m a n S h a k e e l – W W T e c h L e a d f o r M & E a t A W S
T o m N i g h t i n g a l e – S o l u t i o n s A r c h i t e c t
N o v e m b e r 2 8 , 2 0 1 7
C T D 4 0 5

er The Top Video is complex
Agenda
• Introduction/Voice of the Customer
• Relevant AWS services
• Work Backwards from VOC
• Evolution of the Solution
• Version 1 / MVP
• Version 2
• Future Plans
Serverless video workflows

Introduction and VOC

“EXPLOSION OF CONTENT AND DEVICES”
 Age of content abundance:
Everyone is creating video
 Devices everywhere: Today, an
estimated 1.6 billion people
worldwide watch online video on
connected devices
 OTT services: At end of 2016, more
than 49 million homes accessed at
least one OTT service
Annual Expenditure on Programming ($B)

“DIRECT/RELEVANT TO
CONSUMER”
Pay TV Net Changes by Provider
Source:UBS June 2017
Ad Revenues
“Over 75% of what people watch come from
recommendations”
Source: TheGuardian May 7, 2016
Source: Time April 5, 2016
Source: Netflix blog

Personalisation
DVR / PVR
Asset
Management
Encryption Metadata
API’s
Packaging and
Transmux
Transcoding
Ingest /
Acquisition
Delivery
Subtitling /
Audio
Description
FingerprintingQuality Control
Checks
Content Plane
Expiration
Policy
Watermarking
CRM
Billing
Systems
Paywall
Viewership
Analytics
Policy
Management
Ad
Integration
Content
Catalogue
API
Endpoint
Payment
Gateway
Rights
Management
Business Plane
Device
Register
Subscriber
Management
Media
Origin
Content
Delivery
Web Apps
Load
Balancing
CMS
DRM
Licensing
Device
Apps
DeliveryPlane
Systems
Registry
Alerting
Orchestration
Service
Manager
Logging
Workflow
Automation
Monitoring
SystemsPlane
Storage Database
er The Top Video is complex
ComputeNetwork
BUT – “OTT IS COMPLEX”

Video workflow – Content plane
Ingest/ Acquire
Store /
Manage
Process Origin / Deliver Analyse
Store Once
Deliver Anywhere
Business Plane

AWS stack for media-streaming workflow
AWS
Direct Connect
AWS Import/
Export
AWS Storage
Gateway
Ingest/Create
Amazon S3/Glacier
Store
Amazon EBS/ EFS Amazon
CloudSearchAmazon SQS
Amazon
VPC
Amazon
RDS
Lambda
Process
Amazon EC2 CloudFront
Route 53
Deliver
AWS WAF
Devices
AWS Partner Solutions
Live
File Based
Display
Store Once
Deliver Anywhere
Source
AWS Snowball
S3 Transfer
Acceleration
AWS Elemental
Live
AWS Elemental MediaStore Amazon
DynamoDB
AWS Elemental
MediaLive
AWS Elemental
MediaPackage
AWS Elemental
MediaConvert
AWS Elemental
MediaTailor
Amazon
MachineLearning
Amazon
Rekognition
Lambda at
the Edge

I NT R ODUC I NG AWS ELEMENT AL MEDI AC ONV ER T
 Access to professional-grade video features and quality
 No software or hardware infrastructure to manage
 Automatically scales in response to variations in
incoming video volume
 Ability to manage capacity and control order in
which jobs are processed
 API Interface to integrate via Lambda/Serverless functions in a complex
workflow

AWS ELEMENTAL MEDIACONVERT BASICS
NOTIFICATIONS AND METRICS
DOWNSTREAM PROCESSING
JOB 1
INPUT 1
QUEUE(S)
INPUT FILE 1
INPUT
CONFIGURATION
INPUT MEDIAASSET(S) TRANSCODEDMEDIA
Manage available parallel processing capacity between rush
jobs and low priority jobs
JOB TEMPLATE
Amazon CLOUDWATCH
HTTP/HTTPS
Email
SMS
Application
LAMDA
SQS
INPUT n
INPUT FILE n
INPUT
CONFIGURATION
JOB TEMPLATE
HLS PACKAGE
PRESENT 1
PRESENT 2
PRESENT n
DASHPACKAGE
PRESENT 1
PRESENT 2
PRESENT n
TAGSAND
METADATA
OTHER JOB
CONFIG.
Job
 Primary unit of work; Specifies
input and output
Output Preset
 Settings to create a single output
Job Template
 Collection of commonly used job
settings
 Useful when processing a collection
of inputs to produce a fixed set of outputs
Queue
 All jobs are submitted to a queue
 Allows user to separate or group jobs
for processing

Video workflow – Content plane
Ingest/ Acquire
Store /
Manage
Process Origin / Deliver Analyse
Store Once
Deliver Anywhere
Business Plane
AWS Elemental
MediaStore
AWS Elemental
MediaLive
AWS Elemental
MediaPackage
AWS Elemental
MediaConvert
AWS Elemental
MediaTailor

Working Backwards from the VOC

https://aws.amazon.com/answers/
• Best practices
• Prescriptive design patterns
• Ready-made solutions
• Strategic guidance
AWS Answers

Live Streaming Solution

Live streaming solution
AWS provides a live streaming video solution that combines AWS Elemental Cloud, a service that enables
customers to rapidly deploy multiscreen offerings for live and on-demand content, with Elastic Load
Balancing, Auto Scaling, and Amazon CloudFront to build a highly resilient and scalable architecture that
delivers your live content worldwide.

Live streaming solution – Evolution
AWS Elemental Media Live and Media Package offered as a managed service
AWS Elemental
MediaLive
AWS Elemental
MediaPackage
AWS Elemental
MediaTailor
Multiple DevicesLive Stream
Amazon CloudWatch
Amazon
CloudFront

File-Based Streaming Solution

Strategic service development
Lift & Shift
• Build infrastructure
• Move platform
• Automate
Optimise
• Transition to services
• Integrate monitoring
• Abstract data
Transform
• Decouple application
• Build resilience
• Infrastructure as code

Each progressive step was better
Physical Servers
Datacenters
Virtual Servers
Datacenters
• Higher utilization
• Faster provisioning speed
• Improved uptime
• Disaster recovery
• Hardware independence
• Trade CAPEX for OPEX
• More scale
• Elastic resources
• Faster speed and agility
• Reduced maintenance
• Better availability and
fault tolerance
Virtual servers
in the cloud

But there are still limitations
Physical Servers
Datacenters
Virtual Servers
Datacenters
• Trade CAPEX for OPEX
• More scale
• Elastic resources
• Faster speed and agility
• Reduced maintenance
• Better availability and
fault tolerance
• Still need to administer
virtual servers
• Still need to manage
capacity and utilization
• Still need to size
workloads
• Still need to manage
availability, fault tolerance
• Still relatively expensive
to run intermittent jobs
Virtual servers
in the cloud

BEFORE WE DIVE IN…

Event-driven computing
Function as a Service
Serverless
Low-touch, high-developer
productivity for Video workflows

Serveless means…
No servers to provision
or manage
Scales with usage
Never pay for idle
Availability and fault
tolerance built in

Coordination by method call
λ λ
λ
λ
λ
λ
λ

Coordination by function chaining
λ
λ
λ
λ
λλ
λ

Coordination by database
λλ
λ
λ
λλ
λ

Coordination by queues
λ
λ
λ
λ
λλ
λ

Application life cycle – AWS Step Functions
AWS STEP FUNCTIONS
Visualize in the
console
Define in JSON Monitor
executions

Video on Demand (VOD) on AWS
Ingest Step
Functions
Publish Step
Functions
Source
Video
Source
Amazon S3
Process Step
Functions
Workflow
MP4
Amazon S3
HLS
Amazon S3
Amazon
CloudFront
Amazon
DynamoDB
Amazon
SNS
AWS
Lambda
Amazon
Glacier
AWS Elemental
MediaConvert

Ingest Step Function
Check source video and watermark file
Run MediaInfo process – validate this is a valid media
file
Create entry in DynamoDB table with GUID
Notify external systems – email
Execute Process Step Functions with GUID

Process Step Function
Validate encoding profiles against source height
Submit MediaConvert Encode jobs based on profile
Update DynamoDB with job IDs and workflow status

Publish Step Function
Validate files in Amazon S3 and generate CloudFront
URLs
Tag source objects to enable Amazon S3 lifecycle
policy
Update DynamoDB with the output details
Publish – send Amazon SNS notification with workflow
summary

Amazon CloudFront
global content delivery network
100 (PoPs), 88 CloudFront edge locations, 11 regional edge caches, 50 cities, 23 countries
Regional Edge Cache Locations
• Oregon
• Ohio
• N. Virginia
• London
• Frankfurt
• Sao Paulo
• Mumbai
• Singapore
• Seoul
• Tokyo
• Sydney

Optimizing content delivery
Optimize
caching
Whitelist
query strings
and cookies
Reserved
capacity pricing
Origin response headers
CloudFront customization

Securing content delivery
Securing
content
Geo restriction
Securing connection – HTTPS
Access control
Securing the
origin

CloudFront reporting

Analysing CloudFront access logs
Amazon
CloudFront
S3 bucket S3 bucket
Amazon
QuickSight
#Version: 1.0 #Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status
cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol
cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version
2014-05-23 01:13:11 FRA2 182 192.0.2.10 GET d111111abcdef8.cloudfront.net /view/my/file.html 200
www.displaymyfiles.com……
Amazon
Redshift
W3C Log
Processor
AWS Glue
Amazon
Athena

Evolution of the Solution

Development Milestones for VOD in 2017
AGENDA
Version 1
• MVP
Version 2
• Decoupling the Workflow
• Error Handling
• MediaConvert & Metadata
https://aws.amazon.com/answers/media-entertainment/video-on-
demand-on-aws/

• Single input, multiple outputs
• 7 Encoding Profiles (MP4, HLS)
• Encoding decision
• CloudFront for HLS Playback
• SNS Notification
• Capture the workflow details in
DynamoDB
A p p l e T e c h N o t e s T N 2 2 2 4
Version 1 MVP Objective
demand-on-aws/

Version 1 Architecture Overview
demand-on-aws/

Demonstration
Version 1
demand-on-aws/

Version 1 Ingest Step Functions
demand-on-aws/

Version 2 Ingest Step Functions
demand-on-aws/

Version 2 Process Step Functions
demand-on-aws/

Version 1 Error Handling
Amazon
CloudWatch
AWS Step
Functions
Amazon SNS
Errors
demand-on-aws/
Amazon Elastic
Transcoder

Version 2 Error Handling
Amazon SNS
Publish
Amazon
CloudWatch
Amazon
DynamoDB
AWS Lambda
Workflow
functions
Amazon SNS
Errors
AWS Lambda
Error Handler
AWS Step
Functions
demand-on-aws/
AWS Elemental
MediaConvert
Amazon Elastic
Transcoder

C l o u d F o r m a t i o n p a r a m e t e r sA m a z o n S 3 e v e n t t r i g g e r
Version 2 Encoding Options Video Only
demand-on-aws/

I n g e s t & M e t a d a t aA m a z o n S 3 e v e n t t r i g g e r
Version 2 Encoding Options Metadata
demand-on-aws/

Version 2: MediaConvert Encoding
demand-on-aws/
MediaConvert
Regional/Account
API Endpoint

Demonstration
MediaConvert &
Metadata
demand-on-aws/

Version 2 Architecture
Workflow
MP4
Output
HLS/DASH
Output
Amazon
CloudFront
DynamoDB
CloudWatch
Event
Rules
Publish
SNS
Processing Step
Functions
Ingest Step
Functions
Lambda Ingest
Execute
Workflow
Lambda
Functions
Glacier
Archive
Source
S3 Bucket
Video
Watermark
Errors
SNS
Lambda
Error Handler
demand-on-aws/
Metadata
AWS Elemental
Magikarp AWS Elemental
MediaConvert
Publish Step
Functions

Future Plans

Third-party integration
Ingest Step
Functions
Publish Step
Functions
Source
Video
Source
Amazon S3
Process Step
Functions
Workflow
MP4
Amazon S3
HLS
Amazon S3
Amazon
CloudFront
Amazon
DynamoDB
Amazon
SNS
AWS
Lambda
Amazon
Glacier
demand-on-aws/
WS Elemental
MediaConvertAWS Elemental
MediaConvert

THANK YOU!

AWS Serverless Video Workflows

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à AWS Serverless Video Workflows

Similaire à AWS Serverless Video Workflows (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

AWS Serverless Video Workflows