Amazon Rekognition is a deep learning-based image and video analysis service that enables developers to integrate easy-to-use APIs into their applications. This session will walk through the Amazon Rekognition features, including object and scene detection, text-in-image extraction, celebrity recognition, content moderation, and more. Developers can quickly get started with this fully-managed service and start building computer vision applications such as a searchable media library, automated content moderation, an image-based alert system, and more. Level: 200-300 Speaker: Binoy Das - Partner Solutions Architect, AWS

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Build Computer Vision Applications with Amazon Rekognition
Binoy Das
Partner Solutions Architect, AWS

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
The ML Journey: Unlocking Opportunities
1. Automation
– Optimize existing processes
– Maintain (or improve) quality & reduce cost
2. Enrichment
– Improve content and add features to products & services
– Increase engagement & attract new viewers
3. Innovation
– Create entirely new customer experiences
– Leverage industry knowledge and AI expertise for entirely new
categories and business models

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Rekognition Image
Facial Analysis Face Recognition
Text in Image
Deep learning-based image analysis service
Unsafe Image Detection
Object & Scene Detection
Celebrity Recognition

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Rekognition Video
Object & Activity Detection Face Detection &
Recognition
Real-time Live Stream
Deep learning-based video analysis service
Unsafe Video DetectionCelebrity Recognition
Pathing

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Why use Amazon Rekognition?
• Object & Scene Detection
Photo-sharing apps can power smart searches
and quickly find cherished memories, such as
weddings, hiking, or sunsets
• Facial Analysis
Retail businesses can understand the
demographics and sentiment of in-store customers
• Face Comparison
Hotels & hospitality businesses can provide
personalized service for guests and VIPs
• Facial Recognition
Provide secondary authentication for existing
applications

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Object & Scene Detection
Object and scene detection makes it easy for you to add features that search, filter,
and curate large image libraries.
Identify objects and scenes and provide confidence scores
DetectLabels
Flower
Arrangement
Chair
Coffee Table
Living Room Indoors
Furniture
Cushion
Vase
Maple
Villa
Plant
Garden
Water
Swimming Pool
Tree
Potted Plant
Backyard
Patio

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Facial Analysis
Analyze facial characteristics in multiple dimensions
DetectFaces
Image Quality
Facial Landmarks
Demographic Data Emotion Expressed
General Attributes
Facial Pose
Brightness 23.6%
Sharpness 99.9%
EyeLeft,EyeRight,Nose
RightPupil,LeftPupil
MouthRight,LeftEyeBrowUp
Bounding Box...
Age Range 29-45
Gender:Male 96.5%
Happy 83.8%
Surprised 0.65%
Smile:True 23.6%
EyesOpen:True 99.8%
Beard:True 99.5%
Mustache:True 99.9%...
Pitch 1.446
Roll 5.725
Yaw 4.383

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Face Comparison
Measure the likelihood that faces are of the same person
Similarity 93% Similarity 0%
CompareFaces

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Facial Recognition
Find similar faces in a large collection of images
SearchFacesByImage
Search
Index
Collection

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Celebrity Recognition & Image Moderation
Detect explicit and suggestive contentRecognize thousands of famous individuals
DetectModerationLabelsRecognizeCelebrities

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Interfacing with Rekognition
• S3 input for API calls - max image size of 15MB
• 5MB limit for non-S3 (Base64 encoded) API calls
• Minimum image resolution (x or y) of 80 pixels
• Image data supported in PNG or JPG format
• Max number of faces in a single face collection is 1 million
• The max matching faces the search API returns is 4096
• Size of face should occupy 5%+ of image for detection
• Collections are for faces
Optimizing your input & requests for best performance
…
Use Amazon CloudWatch to observe & issue alerts on Rekognition metrics

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
https://console.aws.amazon.com/rekognition/home

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
Rekognition APIs – Overview
Rekognition’s computer vision API operations can be grouped into
Non-storage API operations, and Storage-based API operations
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
Non-storage API Operations
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
Storage-based API Operations
ListFaces

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Rekognition APIs – Advanced Usage
Decision trees and processing pipelines
Why?
• Many use cases require more than a single
operation to arrive at actionable data
How?
• S3 event notifications, Lambda, Step Functions
• DynamoDB for persistent pipeline storage
• Augmenting results with 3rd Party AI/ML
• OpenCV, MXNet, etc. on EC2 Spot, ECS, AI/ML AMI
Sample Use Cases
• Secondary authentication for mobile apps
• Moderate user-generated content
IndexFaces
DetectLabels
“person”

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
What Can You Do with Amazon Rekognition?
• Search for people, objects, scenes, and concepts across millions of
images
• Filter inappropriate or specific content
• Redact identities from images of faces
• Verify identities by matching against reference faces
• Recognize individuals by matching faces to a collection
• Analyze user traffic hotspots and journey paths by demographics and
sentiment

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Social Platforms

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Analyze User Generated Content
Person 99.2%
Dog 95.7%
Person 99.2%
Snowboarding 98.1%
No Moderation labels
detected

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
2. Submit
Image
4. DetectFaces 7. DetectText
1. Upload
3. Store image Lambda Step Functions
5.DetectLabels 6.DetectModerationLabels
8. Store Metadata
& analysis
DynamoDB
ElasticSearch
Analyze Images

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Before After
2.6x more insights, on 30% more posts
Visual Search for Influencer Marketing

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Visual Search for Influencer Marketing
Solution Architecture

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Example: Serverless Image Processing Workflow
https://github.com/aws-samples/lambda-refarch-imagerecognition/

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Demo
• Analyze images/user generated content

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Face-Based User Verification
Authenticated User
Image Capture Application
Amazon S3
Compare Faces
If the similarity score exceeds the selected
threshold (such as 99%), the application
returns a green status. If not, an alert is
issued to security staff.
The application captures a live
image of each employee as they
scan their access card
Rekognition compares the live image
and the badge image – and returns
a similarity score
The application retrieves the
user’s badge from S3
Confirm user identities by comparing their live image with a reference image

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Collections & Access Patterns
Logging - visitor logs, digital libraries
• Easily find specific images from a digital library
• Find certain images by using a reference image
Social Tagging - photo storage and sharing
• One collection per application user
• Automated friend tagging
Person Verification - employee gate check
• One collection for each person to be verified
• Detection of stolen/shared IDs

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Demo
• Face Based Verification

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Fighting Human Trafficking & Rescuing Victims
• Uses Amazon Rekognition to power their Traffic
Jam FaceSearch tool
• “Without Traffic Jam, investigators are left to manually sift
through thousands of online ads. This means they sit at their
computer, with a picture of the victim taped to their screen, and
compare every photo they see online in the hope that they
might find a match. Using AI technology, like Amazon
Rekognition, this critical task can now be done with more
accuracy and within seconds as compared to days, which is so
important in cases where detectives have limited time to find
the victim before he or she is moved to the next city.”
• - Emily Kennedy, President & Co-Founder, Marinus Analytics

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Fighting Child Sexual Abuse & Trafficking
• Using AWS services, such as Amazon Rekognition,
Thorn has seen a 65% reduction in investigation time
• “Amazon Rekognition has been an incredible partner to Thorn by
helping us leverage their image and video analysis solutions in
our mission to defend children from sexual abuse. Abusers have
hijacked the most advanced technology to exploit children—
selling children online for sex, circulating abuse images and
videos, and engaging in live-streaming abuse. AWS has chosen
to be a part of the solution—partnering to leverage their solutions
to help find exploited children faster and stop abuse.
Collaborating with leading technology companies like AWS is
critical to building the tools we need to find these children faster,
and end the spread of child sexual abuse material.”
• - Julie Cordua, CEO, Thorn

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
"Persons": [
{
"Timestamp": number,
"Person":
{
"Index": number,
"BoundingBox":
{
"Width": number,
"Top": number,
"Height": number,
"Left": number
},
"Face":
{
"BoundingBox": { ... },
"Landmarks": { ... },
"Pose": { ... },
"Quality": { ... },
"Confidence": number
}
},
...
GetPathing
StartPathing
Pathing

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Sports Highlights and Pathing
Amazon S3 Face collectionRekognition
1. Store images of recognized people 2. Index individuals into a collection
R e c o g n i t i o n a n d p a t h i n g o f o n - s c r e e n p e r s o n s
Amazon S3 Rekognition Video
3. Store video files
4. Pathing of on screen
and recognize individual
faces
Videos AWS Lambda
7. Summarize Info
on screen detail
DynamoDB
API Gateway
6. Retrieve additional
person’s info
5. Face Match,
analysis and Label
Detection

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pathing Demo

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Metadata Generation

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Royal Wedding

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Saved 22 days of manual image and video tagging by using Amazon Rekognition
Data-Driven Creative Asset Production

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Data-Driven Creative Asset Production
Solution Architecture

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Use-Case: C-SPAN
• Over 200,000 hours of content
• Only half of content is manually tagged
• How can we enrich our metadata in AWS?
• How can we unleash the value of content we already own
once in AWS?
Background
• Large scale video library
• High accuracy required
• Limited budget
• Ability to extract from video
• Keep up with daily increase in content
Challenge
• Solution developed within three weeks
• Live video frame based analysis
• Established, searchable baseline archive
• All content is now tagged and indexed
• Over 99,000 faced indexed and searchable
• Saved ~9,000 hours a year in manual curation costs
Results

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Machine Learning Services
Language
Comprehend, Transcribe
Vision
Rekognition Image & Video
Media Analysis Solution starter kits
Go To aws.amazon.com > answers > media-entertainment > Media-Analysis-Solution
Automated Metadata Generation
Label & Face Detection
Celebrity Detection
Face Search
Person in Picture Tracking
Subtitling
Context
Key Entity & Phrase Detection
Other Services
e.g. Translate Future starter kits
Media Analysis Solution Starter Kit
Automatically provision the services necessary for building common media use cases on AWS

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Benefits of using the solution starter kits
Accelerate Revenue
Get started with your project quickly and
start monetizing your content
One-Click Launch
Launch a fully-tested solution starter kit in
your AWS account with one click
Well-architected
Rely on the architecture as it reflects AWS
best practices
Agility
Save the resources of developing a
solution from the beginning
Expand and Add
Use the framework and available guides to expand the
solution starter kit based on your needs
Please contact your AWS sales contact or kuthan@amazon.com with any questions
The media analysis starter kit provides customers a pre-built solution for common media use cases, which
includes a step-by-step implementation guide for automated deployment

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Media Analysis Solution architecture
Amazon
Rekognition
Amazon
Rekognitionfor
video
Amazon
Transcribe*
Amazon S3
AWSLambda
Search
AmazonAPI
Gateway
Amazon
ElasticSearch
Amazon
CognitoUser
Pool
Amazon
Comprehend
AWSLambda
AIAnalytics SearchDataInteraction
.jpg,
.png
.mp4,
.mov
.mp4,
.mp3,
.wav,
.flac
transcript.j
son
meta
data
meta
data
meta
data
meta
data
AWSStep
Functions
AWSAmplify

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Demo
• Video Meta-Data Generation

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Media Analysis Solution interface
Go To aws.amazon.com > answers > media-entertainment > Media-Analysis-Solution
Below the interface how it will looks deployed in AWS account

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Developer Resources for Amazon Rekognition
Homepage: https://aws.amazon.com/rekognition/
AWS Machine Learning Blog – Amazon Rekognition:
https://aws.amazon.com/blogs/ai/tag/amazon-rekognition/
Serverless Image Recognition processing backend:
https://github.com/awslabs/lambda-refarch-imagerecognition
Reviewing User Generated Content Demo Code:
https://github.com/aws-samples/amazon-rekognition-policing-user-content

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Amazon Rekognition Customers

43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS