Best Practices for Database Migration to the Cloud: Improve Application Performance & Accelerate Data Insights

P U B L I C S E C T O R
S U M M I T
Washington DC

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C TO R
S U M M I T
Best Practices for Database Migration to
the Cloud: Improve Application
Performance & Accelerate Data Insights
3 0 9 0 6 4
Tony Nguyen
Professional Services - Data and Analytics
AWS

S U M M I T
Related Breakouts
295344 - Reducing IT Costs By Migrating Oracle Database to AWS for Florida
Retirement Services
Walter Kelleher, Director of Educational Services, Office of Defined Contribution Programs, State
Board of Administration of Florida
309061 - Learn How to Become Migration Ready to Accelerate and Optimize Your
Cloud Adoption
Sam Smith, Senior IT Project Leader, The Wharton School at the University of Pennsylvania
295361 - The Power of Cloud for Operational Transformation: Follow
Healthcare.gov's Journey to the Cloud
Marc Richardson (CMS) Director of Marketplace Plan Management Group

S U M M I T
What to Expect
Understand
the value
proposition for
databases in the
cloud and the AWS
database portfolio
1
Get a sense
of the volume
and scale at which
customers are using
various AWS
database services
2
Learn
the key
considerations for
cloud database
selection, how to
assess your specific
requirements, and
map them to the right
database
3
Decide and
Execute
how to get started
for the future with
some best practices
4

S U M M I T
What’s so special about databases on AWS?

S U M M I T
Power, HVAC, net
Rack & stack
Server maintenance
OS patches
DB s/w patches
Database backups
Scaling
High availability
DB s/w installs
OS installation
App optimization
If you host your databases on-premises
you

S U M M I T
Power, HVAC, net
Rack & stack
Server maintenance
OS patches
DB s/w patches
Database backups
Scaling
High availability
DB s/w installs
OS installation
App optimization
If you host your databases in Amazon Elastic
Compute Cloud (Amazon EC2)
you

S U M M I T
Power, HVAC, net
Rack & stack
Server maintenance
OS patches
DB s/w patches
Database backups
App optimization
High availability
DB s/w installs
OS installation
Scaling
If you host your databases on managed
services on AWS
you

S U M M I T
Which database is most appropriate for my workload?

S U M M I T
Relational Key-value Document In-memory Graph
Referential
integrity, ACID
transactions,
schema-
on-write
Low-latency,
key lookups
with high
throughput and
fast ingestion
of data
Indexing and
storing
documents with
support for
query on any
attribute
Microseconds
latency, key-
based queries,
and specialized
data structures
Creating and
navigating
data relations
easily and quickly
Lift and shift,
EMR, CRM,
finance
Real-time bidding,
shopping cart, social
Content
management,
personalization,
mobile
Leaderboards,
real-time
analytics, caching
Fraud detection,
social networking,
recommendation
engine
Search
Indexing and
searching
semistructured
logs and data
Product catalog,
help and FAQs,
full text
Time-series Ledger
Collect, store, and
process data
sequenced by
time
IoT applications,
event tracking
Complete,
immutable, and
verifiable history
of all changes to
application data
Systems
of record, supply
chain, health
care,
registrations,
financial
Data categories and common use cases

S U M M I T
Amazon
ElastiCache
AWS: Purpose-built databases
Relational Key-value Document In-memory Graph Search
Amazon
DynamoDB
Amazon NeptuneAmazon Relational
Database Service
(Amazon RDS)
Aurora CommercialCommunity
Amazon
Elasticsearch
Service
Amazon
DocumentDB
Time-series Ledger
Amazon Timestream Amazon
Quantum Ledger
Database

S U M M I T
A Short Break from Generalities
Relational Non-Relational
NoSQL SQL
Schema Schema-free
Unstructured Structured

S U M M I T
Looking at the Specifics
Purpose of a database Your application needs

S U M M I T
Database Workloads
Data Considerations
Shape Size Compute

S U M M I T
Shape
Purpose-Built For Optimized for When you need to Example Workload
Row Store Operate on a record or group of records Payroll
Column Store Aggregations, scans, and joins Analytics
Key-Value Store Query by key with high throughput & fast ingestion Tracking devices
Document Store Index & store documents for query on any property Patient data
Graph Store Persist and retrieve relationships Recommendations
Time-Series Store Store and process data sequence Process Engine telemetry
Unstructured Store Get and put of objects Store user reviews

S U M M I T
Size
Considerations Example Workload
Size at limit – bounded or unbounded
Number of employees – bounded
Number of sensors – unbounded
Working set size & caching
10-years of sales data but only the last 12-months is queried
Session data for users of a streaming service
Retrieval size
Get one row
Get one thousand rows
Partitionable or monolithic
Storage and processing of car location data is partitionable
Company payroll data has no natural partition boundary

S U M M I T
Compute
Considerations Example Workload
Compute functions
Sum of sales for the last 12-months
Get & Put data
Throughput
Million users browsing a product catalogue every second
50 doctors looking at 300 patient records per day
Latency
Get the location of a car in 5 milliseconds
Get the min, max & average deal size for the last 12-months in 5
seconds
Change rate
Inventory counts are frequently updated
Sales records are never updated
Rate of ingestion
Location telemetry from cars added to the database every minute
New employees records being added to the database

S U M M I T
Data Characteristics: Temperature
Hot Warm Cold
Volume MB–GB GB–TB PB–EB
Item size B–KB KB–MB KB–TB
Latency ms ms, sec min, hrs
Durability Low–high High Very high
Request rate Very high High Low
Cost/GB $$-$ $-¢¢ ¢

S U M M I T
Data Characteristics

S U M M I T
My [insert your favorite DB] works for
ever ything
General purpose Special purpose
One size fits all Efficiency at scale

S U M M I T
But Which Database to Use When?
Decision points and considerations

S U M M I T
But Which Databases to Use
When?
Why pick just one?

S U M M I T
Our Strategy

S U M M I T
Amazon
ElastiCache
AWS: Purpose-built databases
Relational Key-value Document In-memory Graph Search
Amazon
DynamoDB
Amazon
Neptune
Amazon
RDS
Aurora CommercialCommunity
Amazon
Elasticsearch
Service
Amazon
DocumentDB
Time-series Ledger
Amazon
Timestream
Amazon
Quantum
Ledger
Database

S U M M I T
Which database(s) should I choose?

S U M M I T
Decision Matrix
Service Shape Size Workload Performance Durability Expertise Legacy Business
Needs
Ops.
Load
Platform
Integration
Amazon
Aurora
MySQL
Structured,
semistruct.
Mid TB
Range
K/V lookups,
transactional,
light analytics
High
throughput,
low latency
High Relational,
MySQL,
SQL Server
User defined
code, COTS
Database
Freedom
Low to
moderate
Serverless,
IAM, AWS
Lambda, Auto
Scaling,
Amazon Simple
Storage Service
(Amazon S3)
Amazon
Aurora
PostgreSQL
Structured,
semistruct.
Mid TB
Range
Transactional,
light analytics
High
throughput,
low latency
High Relational,
PostgreSQ
L, Oracle
User defined
code, COTS
Database
Freedom
Low to
moderate
In the works
Amazon
RDS DB
Engines
Structured,
semistruct.
Low TB
Range
Transactional,
light analytics
Mid-to-high
throughput,
low latency
User
controlled
Engine
specific
Engine req.,
COTS
Right
sizing
Moderate Log streaming
Amazon
DynamoDB
Semistruct. High TB
Range
K/V lookups,
NoSQL, OLTP,
document store
Ultra-high
throughput,
low latency
High NoSQL Zero
downtime,
Ultra-high
scale
Low Serverless,
IAM, Lambda,
DAX, Auto
Scaling, Kinesis
Streams
Amazon
Neptune
Graph-
structured
Mid TB
Range
Graph, highly
connected
data, transact.
High
throughput,
low latency
High Graph,
Gremlin,
SPARQL
Database
Freedom
Low IAM, Amazon
S3

S U M M I T
Decision Matrix (cont.)
Service Shape Size Workload Performance Durability Expertise Legacy Business
Needs
Ops. Load Platform
Integration
Amazon
ElastiCache
Semistruct.,
Unstruct.
Low TB
Range
In-memory
caching,
K/V lookups,
NoSQL
High
throughput,
ultra-low
latency
Low (In-
memory,
auto
failover to
read replica
Caching,
NoSQL
Response
latency, DB
cost opt.
Low Scalable
clusters
Amazon
Redshift
Structured,
semistruct.
PB
Range
Optimized
analytics
Mid-to-high
latency
High DW, data
science
User
defined
code,
COTS
Cost opt,
performance,
reporting
Moderate IAM,
Amazon S3
Amazon
Athena
Structured,
semistruct.
PB
Range
Flexible
analytics
High latency High
(Amazon
S3)
Data lakes,
data science
Flexibility,
easy query
against data
lake
Low IAM,
Amazon S3
Amazon
EMR
Structured,
Semistruct,
unstruct.
PB
Range
Flexible
analytics
Low-to-high
latency
User-
controlled
Data lakes,
data science,
DW
Tooling &
versioning
Cost opt.,
Flexibility,
analytics
Moderate IAM,
Amazon S3,
EC2 Spot
Amazon
EC2
* * * Low-to-high
latency
User-
controlled
* * * High *

S U M M I T
Well Modelled DataData Exploration
Non-SQL Analytics
Real-Time analytics
Managed Storage Delivery
Amazon
Elasticsearch Service
Kinesis Analytics
Athena Amazon
Redshift
Amazon S3
You don’t have to pick just one – choose the
best tool for the job

S U M M I T
Pattern: Multi-DB analytics with Amazon
Aurora & Amazon Redshift

S U M M I T
Can I actually migrate my existing
database?

S U M M I T
• Quickly rule out strict non-candidates.
• Is it a COTS application without support for your target database?
• Does the application have RDBMS-specific modules produced by a 4GL or
application framework tool (e.g., Oracle APEX, Report Writer, Oracle Forms, etc) or
pre-compiler (e.g., Pro*C, OCI, Pro*COBOL)?
• Can it reasonably be refactored / ported?
• Does the application have significant amounts of anonymous PL/SQL?
• Is the initial cost of refactoring worth the benefits gained from moving to the target
database?
• Think license costs, productivity gains, architectural flexibility…
• Are there workload needs blocking me from taking advantage of Amazon RDS or other
managed services?
• No show-stoppers? Go ahead!
Does my application / workload support a
migration?

S U M M I T
An example evaluation considering Oracle to Aurora
Interested in “Database Freedom”?
Stay on Oracle Database
Workload Needs?
 64 TB
 Storage(> 80K IOPS, > ~1300 MB/s)
Migration Options
DMS, RMAN, expdp/impdp, GG, etc
RDS for Oracle
• Fully-managed DBaaS
• Patching
• Backups
• Availability Options
• Multi-AZ
EC2
A VM Instance with a
customer-maintained OS
image and Oracle Database
software
• Instance types
• EBS storage options
• Hi-AV, DR Strategies
Qualify the App
Is it COTS or otherwise
copyrighted code
Is the App Portable?
(Limited: Oracle Forms, Report
Writer reports, anonymous
PL/SQL, etc)
Re-Engineer
(e.g., Forms conversion Tools,
SCT, DMS, Complete
Rewrite,Consulting)
Porting
• SCT, DMS
• Manual Effort
• Partial Rewrite
• Consulting
Aurora
Fully-managed enterprise-
grade RDBMS service based
on open source databases
(e.g., Postgres, MySQL)
Yes
Yes
No
No
Yes
No
Yes, RDS! No, not RDS

S U M M I T
What tools are available to help me
with my migration?

S U M M I T
AWS migration options
AWS Database Migration
service (AWS DMS)
Database migration and
replication managed
service
Benefit
• Easily and securely
migrate and/or replicate
your databases and
data warehouses to
AWS
• Migrate between
different database
engines
• Low cost and global
availability
AWS Schema
Conversion Tool (AWS
SCT)
Development tooling
to convert schemas
between databases
and data warehouses
Benefit
• Automates schema
conversion including
database structure and
code
• Minimizes manual effort
of performing a schema
re-write
• Allows conversion from
commercial to open
source platforms
Native tooling
Migration options
included with the
engine
Benefit
• Leverage a familiar
environment
• Full support for native
features
Open source
Ora2Pg and others
Benefit
• Tailored solution for a
specific problem
• Multiple tools can offer a
more complete solution
than any one product
• Free
Commercial
Attunity, Golden Gate
and others
Benefit
• Commercial solutions
are available for a wide
range of migration
sources and targets
including legacy
mainframe systems

S U M M I T
AWS Database Migration Service

S U M M I T
>110,000 databases migrated with DMS

S U M M I T
The US Department of Veterans Affairs (VA) processes hundreds of
thousands of Veterans Appeals each year.
"Our appeals processing system, VACOLS, includes 20 million records
stored in an Oracle 11g database. The system is more than 20 years old
and is in the process of being modernized. During this time, we need to
ensure that the data is securely replicated into the cloud for safekeeping.
We're using AWS DMS to replicate the database into an RDS Oracle
database in AWS GovCloud, in a Multi-AZ deployment. This setup
ensures that VACOLS data is preserved, secured, and highly available
in the cloud, which is a serious win for VA and for our Veterans, who rely
on us for the safeguarding of their information."
Alan Ning, Site Reliability Engineer, U.S. Digital Service
DMS Customer Experience
"Our appeals processing system, VACOLS, includes 20 million records stored in an Oracle 11g database. The system is more than 20 years old and is in the process of being modernized. During this time, we need to ensure that the data is securely replicated in
"Our appeals processing system, VACOLS, includes 20 million records stored in an Oracle 11g database. The system is more than 20 years old and is in the process of being modernized. During this time, we need to ensure that the data is securely replicated

S U M M I T
AWS Schema Conversion Tool
• Assessment report
• Project interface
• Code browser
• Automates many conversion tasks
Packages
Stored procedures
Functions
Triggers
User defined types
Schemas
Tables
Indexes
Views
Sequences
Synonyms
Tab with the assessment report
Manual conversion tips
Side by side code view

S U M M I T
SQL scripts
Packages
Stored procedures
Functions
Triggers
User defined types
Schemas
Tables
Indexes
Views
Sequences
Synonyms
AWS SCT does a great
job of converting your
schema and code
objects
Users, roles, grants
https://aws.amazon.com/blogs/database/use-sql-to-map-users-roles-and-grants-from-oracle-to-postgresql/

S U M M I T
SCT migration assessment report
Connect SCT to “source” &
“target”
Run “assessment report”
Read “executive summary”
& follow detailed
instructions

S U M M I T
Convert schema & migrate data
Step 1: Convert or copy your schema
Source DB or DW AWS SCT Destination DB or DW
Step 2: Move your data
Source DB or DW Destination DB or DW
AWS DMS
Copy or convert
Data
schema

S U M M I T
“
HowAmazon is Achieving Database Freedom Using
AWS
Amazon is on the verge of concluding
an enterprise-level, multiyear initiative
to move >50PB of the company's data
from ~5000 Oracle databases onto
AWS, and had urgent reasons to make
it work including complicated and error
prone administration, provisioning, and
capacity planning. The company's steep
growth trajectory required more and
more Oracle database shards.
Business as usual on Oracle would
have cost 10% more per year.
• Cut database operating costs by
50%
• Eliminated most database-
administration and hardware-
management overhead
• Lowered latency of most critical
services by 40%
SolutionsChallenge Benefits
Company: Amazon
Industry: Multiple
Country: US and Worldwide
Employees: 560,000
Website: www.amazon.com
About Amazon
Amazon is guided by four principles:
customer obsession rather than
competitor focus, passion for
invention, commitment to operational
excellence, and long-term thinking.
Customer reviews, 1-Click shopping,
personalized recommendations,
Prime, Fulfillment by Amazon, AWS,
Kindle Direct Publishing, Kindle, Fire
tablets, Fire TV, Amazon Echo, and
Alexa are some of the products and
services pioneered by Amazon.
• Amazon DynamoDB
• Amazon Aurora
• Amazon Simple Storage Service
• Amazon Relational Database
Service
• AWS Database Migration
Service
• AWS Schema Conversion Tool
If a company like Amazon can move so many
databases used by so many decentralized, globally
distributed teams from Oracle to AWS, it's really within
the reach of almost any enterprise.
- Thomas Park, Senior Manager of Solution Architecture for Consumer
Business Data Technologies, Amazon
“

S U M M I T
Under the hood – Analytics at Amazon
Amazon builds and operates thousands of micro services to serve millions of customers.
More than 1,800 teams publish datasets to Amazon’s analytics infrastructure, including
more than 50 petabytes of data and 75,000 tables, processing 600,000 user analytics jobs
each day, while more than 3,300 consumer teams analyze this data. The on-premises
Oracle infrastructure was monolithic, transformations of tables >100 million rows
consistently failed, limiting the ability to generate insights and leverage machine learning.
SCT was used to automatically convert and validate 80 percent of the 200,000 queries from
Oracle SQL to Amazon Redshift SQL, saving more than 1,000 person-months of manual
effort.
The new analytics infrastructure has one data lake with more than 200 petabytes of data—
almost four times the size of the previous Oracle data warehouse.

S U M M I T
Migration Best Practices

S U M M I T
Lift and shift
Move to AWS with as few changes as possible (i.e., no RDS/Aurora)
Leverage Amazon EC2 and Amazon S3, with plan for possible changes at
later date
Homogeneous migration
Keep existing database engine
For example, Oracle on-prem to AWS Oracle (RDS or EC2)
Amazon EC2 Oracle to Amazon RDS Oracle
Heterogeneous migration
Migrate to a managed or open source engine
For example, Oracle-to-Amazon Aurora, RDS PostgreSQL, or RDS MySQL
Migrate traditional RDBMS or commercial data warehouse to Redshift
Common Migration Terms and Types

Determine
Migration Path
Lift and Shift
Automate
RE-HOST
Lift and Reshape
RE-PLATFORM
Drop & Shop
REPLACE
Re-architect/
Decoupling apps
REFACTOR
Purchase
COTS/SaaS &
licensing
MODIFY UNDERLYING
INFRASTRUCTURE
Manual Install
& Setup Integration
Redesign
Application/
Infrastructure
Architecture
App Code
Development
Full ALM /
SDLC
Integration
Use Migration Tools
Assess/Prioritize
Applications
Discover
RETAIN RETIRE
Not Moving Decommission
Validation Transition Production
Manual Install Manual Config Manual Deploy
MANUAL
4 R’s of Migration

S U M M I T
Migration assessment
Keep the following assessments separate:
Technical
Processes
Personnel & tools
Create an “inventory” for the database migration, including features:

S U M M I T
• You can use the WQF for classifying workloads to determine the ease of migration, staff-hour consumption,
and to which appropriate AWS Service to migrate to
• There are two main types of database workloads:
• Online Transaction Processing (OLTP): many small/fast transactions and queries via indexes that
return few rows
• Data Warehouse (DW): interactive and long-running queries that scan millions of rows and batched
updates
• WQF classifies OLTP and DW workloads into five categories:
Workload qualification framework (WQF)
Category 1 ODBC/JBDC workloads Lightweight applications with no PL/SQL
Category 2 Light, proprietary feature workloads Custom web application with some PL/SQL
Category 3 Heavy, proprietary feature workloads Custom web application with heavy PL/SQL
Category 4 Engine-specific workloads Remedy or other proprietary applications
Category 5 Non-portable, high-risk, or lift-and-shift Legacy software
Other way to classify the database workload can be:

S U M M I T
Estimating “level of effort” for DB migration

S U M M I T
Classify OLTP and OLAP workloads using AWS
SCT with the Workload Qualification
Framework module

S U M M I T
Master plan

S U M M I T
What are other ways AWS can help
me accelerate my migration?

S U M M I T
Database Migration Resources
Migration Methodology
AWS Tools and
Specialist Teams
AWS Partners
AWS Investment AWS Training AWS Professional
Services

S U M M I T
Programs
Discover Database Freedom with AWS
Innovation
• Tools: Database Migration Service (DMS) and Schema Conversion Tool (SCT)
• AWS native managed database services
• Optimized and new Amazon EC2 and Amazon RDS instance types
• ProServe, Partners, Service Teams
• Workload Qualification Framework
• Patterns and Recommendations
Expertise
• AWS Professional Services, partners, product teams
• Migration Playbooks
• Patterns and Recommendations
• Proof-of-concepts
• Workshops
• Incentives & credits
Database Freedom is an AWS Database Migration initiative focused on accelerating enterprise migrations from
commercial database engines (Oracle and SQL Server) to AWS native database services.

S U M M I T
Legacy databases to AWS Migration
Playbooks• Topic-by-topic overview of how to migrate databases and
data warehouses to AWS services
• Covers all proprietary features and the different database
objects
• Migration best practices
• Oracle to Aurora PostgreSQL – available
• SQL Server to Aurora MySQL – available
• SQL Server to Aurora PostgreSQL – available
SCT DMS Playbook
Schema Data Best Practices

S U M M I T
AWS database partners
Migration SI Partners License Advisory Partners

S U M M I T
In Closing
AWS offers a myriad of services designed to help you solve your toughest migration problems
at scale – no need to just pick one service
When selecting a database, consider the dimensions and pick the best match for each
component of your workloads. Prefer purpose-built tools over all-in-one solutions!

S U M M I T
In Closing - Migration Best Practices
• Don’t assume everything will just work after running the tools once
• Understand your source, target, and data
• Plan your migrations just like a new development project, use the
Workload Qualification Framework (WQF) to help scope
• Be prepared to test your migration strategy and iterate
• Define a comprehensive set of KPIs!
• Use migration partners and migration playbooks
• Ask for help from AWS
• Provide feedback on your experience so we can improve our tools
and services

S U M M I T
Resources
Database Migrations
https://aws.amazon.com/solutions/database-migrations/
Database Freedom
https://aws.amazon.com/solutions/databasemigrations/database-
freedom/
AWS Database Migration Service
https://aws.amazon.com/dms/resources/

S U M M I T
Q&A

S U M M I T

Thank you!
S U M M I T
Tony Nguyen
Professional Services - Data and Analytics
AWS

Best Practices for Database Migration to the Cloud: Improve Application Performance & Accelerate Data Insights

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Best Practices for Database Migration to the Cloud: Improve Application Performance & Accelerate Data Insights

Similaire à Best Practices for Database Migration to the Cloud: Improve Application Performance & Accelerate Data Insights (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

Best Practices for Database Migration to the Cloud: Improve Application Performance & Accelerate Data Insights