SlideShare une entreprise Scribd logo
1  sur  8
Page 1 of 8
Comparing Redis with Relational
Hands-on Lab
1. Create an Amazon EC2 t2.micro instance with Amazon Linux
 See documentation at
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html
 Create and use a security group that allows inbound TCP access
using SSH (port 22) and MYSQL (port 3306) plus a custom rule to
allow access from Redis (port 6379)
You might get an automated warning that your EC2 instance is
“open to the world”, because we’re not limiting the source range
for SSH. This is expected. In a production system, you’ll want to
provide a limited IP range for allowed SSH access. For this lab,
disregard the warning.
 We’ll be accessing both MySQL and Redis from this EC2 instance.
Once the instance is created, go back to the security group and
update the “Source” for both services to use the IP address of your
EC2 instance.
2. Access the linux console. See documentation at
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html
Use ssh for Linux or Mac; use PuTTY for Windows
Example:
ssh –i ~/Downloads/key.pem ec2-user@ec2-01-02-03-99.us-west-2.compute.amazonaws.com
[ec2-user@ip-192-168-0-1 ~]$
 Install MySql and Redis development clients
$ sudo yum install mysql-devel
$ sudo yum install gcc
$ sudo pip install MySQL-python
$ sudo pip install redis
Page 2 of 8
 Install Redis command line interface
$ sudo yum --enablerepo=epel install redis
3. Create an Amazon RDS MySQL db.t2.micro instance
 See documentation at
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.Cre
atingConnecting.MySQL.html
 Name the instance sql-lab. Be sure to choose MySQL (not Aurora
and not MariaDB) and the db.t2.micro instance size
 Choose a username and a password (and don’t forget them!)
 Do not create a database (we will do that later)
 Once the instance is created, find your mysql node name
i. On the AWS Console, choose Services, then RDS
ii. On the RDS dashboard, choose DB Instances
Page 3 of 8
iii. Select the ▶ next to your DB Instance
iv. Note your endpoint name. You will need it later!
When using the endpoint name, you usually should not use
the port extension (:3306), just the name.
 Verify you can access the mysql> console from your EC2 instance
$ mysql –h <mysql node name> -u <user name> -p
Example:
$ mysql -h sql-lab.cxpjiluqq0c9.us-west-2.rds.amazonaws.com -u awsuser -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 9999
Server version: 5.6.27-log MySQL Community Server (GPL)
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
mysql>
To exit from the mysql> prompt, use CTRL-D
4. Create an Amazon ElastiCache cache.t2.micro instance with Redis
 See documentation at
Page 4 of 8
http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/GettingStarted.ht
ml
 Name the instance redis-lab. Be sure to choose the cache.t2.micro
instance size
 Disable replication (just a single node)
 Once the instance is created, find your ElastiCache for Redis node
name
i. On the AWS Console, choose Services, then ElastiCache
ii. On the ElastiCache dashboard, choose Cache Clusters
Page 5 of 8
iii. Select the ▶ next to your Cache Cluster name, then click on
the Nodes
iv. Note your endpoint name. You will need it later!
 Verify you can access the redis> console from your EC2 instance
$ redis-cli –h <redis node name>
Example:
$ $ redis-cli -h redis-lab.qwe34ytz.0001.usw2.cache.amazonaws.com
redis-lab.sjyq2g.0001.usw2.cache.amazonaws.com:6379>
To exit from the redis> prompt, use CTRL-D
Page 6 of 8
5. Download and Prepare Landsat scenes
 See documentation at
https://aws.amazon.com/public-data-sets/landsat/
 From your EC2 instance, download the Landsat scenes
$ wget http://landsat-pds.s3.amazonaws.com/scene_list.gz
 Unzip the scene list
$ gunzip scene_list.gz
 Trim the list to the last 250,000 scenes
$ cp scene_list scene_list.orig
$ tail -n 250000 scene_list.orig > scene_list
6. Load to MySQL
 Log into the mysql> console
 Create a landsat database
mysql> CREATE DATABASE landsat;
mysql> USE landsat;
 Create the scene_list table
mysql> CREATE TABLE scene_list (entityId VARCHAR(64), acquisitionDate
DATETIME,cloudCover DECIMAL(5,2),processingLevel VARCHAR(8),path
INT,row INT,min_lat DECIMAL(8,5),min_lon DECIMAL(8,5),max_lat
DECIMAL(8,5),max_lon DECIMAL(8,5),download_url VARCHAR(128));
 Load the landsat data
mysql> LOAD DATA LOCAL INFILE 'scene_list' INTO TABLE scene_list FIELDS
TERMINATED BY ',';
Page 7 of 8
7. Load to Redis
 Create the file sql2redis.py, containing the following code. Be sure to
replace items in red with your own endpoints, user name and
password.
#!/usr/bin/python
import redis
import MySQLdb
from collections import Counter
r = redis.StrictRedis('<your redis node>',port=6379,db=0)
database = MySQLdb.connect("<your MySQL node>","<your username>","<your
password>","landsat")
cursor = database.cursor()
select = 'SELECT entityId, UNIX_TIMESTAMP(acquisitionDate), cloudCover,
processingLevel, path, row, min_lat, min_lon, max_lat, max_lon, download_url FROM
scene_list'
cursor.execute(select)
data = cursor.fetchall()
for row in data:
r.hmset(row[0],{'acquisitionDate':row[1],'cloudCover':row[2],'processingLevel':row[
3],'path':row[4],'row':row[5],'min_lat':row[6],'min_lon':row[7],'max_lat':row[8],'max_
lon':row[9],'download_url':row[10]})
r.zadd('cCov',row[2],row[0])
r.zadd('acqDate',row[1],row[0])
cursor.close()
database.close()
 From the linux console, run the python script to cache data in Redis
$ python sql2redis.py
o this will take a few minutes to run. To check progress, log into
your Redis node and run
redis> DBSIZE
Page 8 of 8
8. Run SQL query
 Log in to your MySQL node and run a query
mysql> SELECT DISTINCT(a.entityId) AS Id, a.cloudCover
FROM scene_list a
INNER JOIN (
SELECT entityId, acquisitionDate
FROM scene_list
WHERE acquisitionDate > (
SELECT MAX(acquisitionDate)
FROM scene_list
WHERE acquisitionDate < CURDATE() - INTERVAL 1 YEAR
)
) b ON a.entityId = b.entityId AND a.acquisitionDate = b.acquisitionDate
WHERE cloudCover < 50
ORDER BY Id;
 This generates a list of all the satellite images during the last year
which have less than 50% cloud cover
 Note how long it takes to get an answer
9. Run Redis query
 Log in to your Redis node and run
redis> zunionstore temp:cCov 1 cCov
redis> zremrangebyscore temp:cCov 50 inf
redis> zunionstore temp:acqDate 1 acqDate
redis> zremrangebyscore temp:acqDate 0 1484611200
redis> zinterstore out 2 temp:cCov temp:acqDate WEIGHTS 1 0
 Again, this generates a list of all the satellite images during the last
year which have less than 50% cloud cover
 Note how long it takes to get an answer

Contenu connexe

Tendances

AutoScaling and Drupal
AutoScaling and DrupalAutoScaling and Drupal
AutoScaling and DrupalPromet Source
 
Taking advantage of the Amazon Web Services (AWS) Family
Taking advantage of the Amazon Web Services (AWS) FamilyTaking advantage of the Amazon Web Services (AWS) Family
Taking advantage of the Amazon Web Services (AWS) FamilyBen Hall
 
Introduction To Terraform
Introduction To TerraformIntroduction To Terraform
Introduction To TerraformSasitha Iresh
 
How to create aws s3 bucket using terraform
How to create aws s3 bucket using terraformHow to create aws s3 bucket using terraform
How to create aws s3 bucket using terraformKaty Slemon
 
Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more
Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and moreScaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more
Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and moreDropsolid
 
AWS Sydney Summit 2013 - Your First Week with Amazon EC2
AWS Sydney Summit 2013 - Your First Week with Amazon EC2AWS Sydney Summit 2013 - Your First Week with Amazon EC2
AWS Sydney Summit 2013 - Your First Week with Amazon EC2Amazon Web Services
 
Lab 1: Introduction to Amazon EC2 and MPI
Lab 1: Introduction to Amazon EC2 and MPILab 1: Introduction to Amazon EC2 and MPI
Lab 1: Introduction to Amazon EC2 and MPIZubair Nabi
 
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google CloudDrupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google CloudDropsolid
 
[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...
[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...
[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...Amazon Web Services Korea
 
Developing Terraform Modules at Scale - HashiTalks 2021
Developing Terraform Modules at Scale - HashiTalks 2021Developing Terraform Modules at Scale - HashiTalks 2021
Developing Terraform Modules at Scale - HashiTalks 2021TomStraub5
 
(DEV301) Automating AWS with the AWS CLI
(DEV301) Automating AWS with the AWS CLI(DEV301) Automating AWS with the AWS CLI
(DEV301) Automating AWS with the AWS CLIAmazon Web Services
 
Moving Drupal to the Cloud
Moving Drupal to the CloudMoving Drupal to the Cloud
Moving Drupal to the CloudAri Davidow
 
(APP306) Using AWS CloudFormation for Deployment and Management at Scale | AW...
(APP306) Using AWS CloudFormation for Deployment and Management at Scale | AW...(APP306) Using AWS CloudFormation for Deployment and Management at Scale | AW...
(APP306) Using AWS CloudFormation for Deployment and Management at Scale | AW...Amazon Web Services
 
Amazon Web Services EC2 Basics
Amazon Web Services EC2 BasicsAmazon Web Services EC2 Basics
Amazon Web Services EC2 BasicsOnur ŞALK
 
Fargate 를 이용한 ECS with VPC 1부
Fargate 를 이용한 ECS with VPC 1부Fargate 를 이용한 ECS with VPC 1부
Fargate 를 이용한 ECS with VPC 1부Hyun-Mook Choi
 
How to scheduled jobs in a cloudera cluster without oozie
How to scheduled jobs in a cloudera cluster without oozieHow to scheduled jobs in a cloudera cluster without oozie
How to scheduled jobs in a cloudera cluster without oozieTiago Simões
 

Tendances (20)

AutoScaling and Drupal
AutoScaling and DrupalAutoScaling and Drupal
AutoScaling and Drupal
 
Taking advantage of the Amazon Web Services (AWS) Family
Taking advantage of the Amazon Web Services (AWS) FamilyTaking advantage of the Amazon Web Services (AWS) Family
Taking advantage of the Amazon Web Services (AWS) Family
 
Introduction To Terraform
Introduction To TerraformIntroduction To Terraform
Introduction To Terraform
 
How to create aws s3 bucket using terraform
How to create aws s3 bucket using terraformHow to create aws s3 bucket using terraform
How to create aws s3 bucket using terraform
 
Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more
Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and moreScaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more
Scaling Drupal in AWS Using AutoScaling, Cloudformation, RDS and more
 
AWS Sydney Summit 2013 - Your First Week with Amazon EC2
AWS Sydney Summit 2013 - Your First Week with Amazon EC2AWS Sydney Summit 2013 - Your First Week with Amazon EC2
AWS Sydney Summit 2013 - Your First Week with Amazon EC2
 
Lab 1: Introduction to Amazon EC2 and MPI
Lab 1: Introduction to Amazon EC2 and MPILab 1: Introduction to Amazon EC2 and MPI
Lab 1: Introduction to Amazon EC2 and MPI
 
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google CloudDrupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
 
[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...
[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...
[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...
 
Programming Amazon Web Services
Programming Amazon Web ServicesProgramming Amazon Web Services
Programming Amazon Web Services
 
Oracle on AWS RDS Migration - 성기명
Oracle on AWS RDS Migration - 성기명Oracle on AWS RDS Migration - 성기명
Oracle on AWS RDS Migration - 성기명
 
Developing Terraform Modules at Scale - HashiTalks 2021
Developing Terraform Modules at Scale - HashiTalks 2021Developing Terraform Modules at Scale - HashiTalks 2021
Developing Terraform Modules at Scale - HashiTalks 2021
 
Deep Dive on Amazon EC2
Deep Dive on Amazon EC2Deep Dive on Amazon EC2
Deep Dive on Amazon EC2
 
(DEV301) Automating AWS with the AWS CLI
(DEV301) Automating AWS with the AWS CLI(DEV301) Automating AWS with the AWS CLI
(DEV301) Automating AWS with the AWS CLI
 
Moving Drupal to the Cloud
Moving Drupal to the CloudMoving Drupal to the Cloud
Moving Drupal to the Cloud
 
Azure powershell management
Azure powershell managementAzure powershell management
Azure powershell management
 
(APP306) Using AWS CloudFormation for Deployment and Management at Scale | AW...
(APP306) Using AWS CloudFormation for Deployment and Management at Scale | AW...(APP306) Using AWS CloudFormation for Deployment and Management at Scale | AW...
(APP306) Using AWS CloudFormation for Deployment and Management at Scale | AW...
 
Amazon Web Services EC2 Basics
Amazon Web Services EC2 BasicsAmazon Web Services EC2 Basics
Amazon Web Services EC2 Basics
 
Fargate 를 이용한 ECS with VPC 1부
Fargate 를 이용한 ECS with VPC 1부Fargate 를 이용한 ECS with VPC 1부
Fargate 를 이용한 ECS with VPC 1부
 
How to scheduled jobs in a cloudera cluster without oozie
How to scheduled jobs in a cloudera cluster without oozieHow to scheduled jobs in a cloudera cluster without oozie
How to scheduled jobs in a cloudera cluster without oozie
 

Similaire à Hands-on Lab: Comparing Redis with Relational

Lab Manual Combaring Redis with Relational
Lab Manual Combaring Redis with RelationalLab Manual Combaring Redis with Relational
Lab Manual Combaring Redis with RelationalAmazon Web Services
 
Lab Manual reModernize - Updating and Consolidating MySQL
Lab Manual reModernize - Updating and Consolidating MySQLLab Manual reModernize - Updating and Consolidating MySQL
Lab Manual reModernize - Updating and Consolidating MySQLAmazon Web Services
 
Lab Manual Managed Database Basics
Lab Manual Managed Database BasicsLab Manual Managed Database Basics
Lab Manual Managed Database BasicsAmazon Web Services
 
Working in the multi-cloud with libcloud
Working in the multi-cloud with libcloudWorking in the multi-cloud with libcloud
Working in the multi-cloud with libcloudGrig Gheorghiu
 
대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016
대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016
대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016Amazon Web Services Korea
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesYevgeniy Brikman
 
An intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSAn intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSYevgeniy Brikman
 
KDD 2016 Streaming Analytics Tutorial
KDD 2016 Streaming Analytics TutorialKDD 2016 Streaming Analytics Tutorial
KDD 2016 Streaming Analytics TutorialNeera Agarwal
 
Gianluca Arbezzano Wordpress: gestione delle installazioni e scalabilità con ...
Gianluca Arbezzano Wordpress: gestione delle installazioni e scalabilità con ...Gianluca Arbezzano Wordpress: gestione delle installazioni e scalabilità con ...
Gianluca Arbezzano Wordpress: gestione delle installazioni e scalabilità con ...Codemotion
 
Amazon Web Services and Docker: from developing to production
Amazon Web Services and Docker: from developing to productionAmazon Web Services and Docker: from developing to production
Amazon Web Services and Docker: from developing to productionPaolo latella
 
Higher order infrastructure: from Docker basics to cluster management - Nicol...
Higher order infrastructure: from Docker basics to cluster management - Nicol...Higher order infrastructure: from Docker basics to cluster management - Nicol...
Higher order infrastructure: from Docker basics to cluster management - Nicol...Codemotion
 
Hopping in clouds - phpuk 17
Hopping in clouds - phpuk 17Hopping in clouds - phpuk 17
Hopping in clouds - phpuk 17Michele Orselli
 
Artem Zhurbila - docker clusters (solit 2015)
Artem Zhurbila - docker clusters (solit 2015)Artem Zhurbila - docker clusters (solit 2015)
Artem Zhurbila - docker clusters (solit 2015)Artem Zhurbila
 
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS LambdaImmutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS LambdaAOE
 
Docker Security workshop slides
Docker Security workshop slidesDocker Security workshop slides
Docker Security workshop slidesDocker, Inc.
 
R server and spark
R server and sparkR server and spark
R server and sparkBAINIDA
 
Manage cloud infrastructures using Zend Framework 2 (and ZF1)
Manage cloud infrastructures using Zend Framework 2 (and ZF1)Manage cloud infrastructures using Zend Framework 2 (and ZF1)
Manage cloud infrastructures using Zend Framework 2 (and ZF1)Enrico Zimuel
 
Bare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and ChefBare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and ChefMatt Ray
 

Similaire à Hands-on Lab: Comparing Redis with Relational (20)

Lab Manual Combaring Redis with Relational
Lab Manual Combaring Redis with RelationalLab Manual Combaring Redis with Relational
Lab Manual Combaring Redis with Relational
 
Lab Manual reModernize - Updating and Consolidating MySQL
Lab Manual reModernize - Updating and Consolidating MySQLLab Manual reModernize - Updating and Consolidating MySQL
Lab Manual reModernize - Updating and Consolidating MySQL
 
Lab Manual Managed Database Basics
Lab Manual Managed Database BasicsLab Manual Managed Database Basics
Lab Manual Managed Database Basics
 
Working in the multi-cloud with libcloud
Working in the multi-cloud with libcloudWorking in the multi-cloud with libcloud
Working in the multi-cloud with libcloud
 
대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016
대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016
대용량 데이타 쉽고 빠르게 분석하기 :: 김일호 솔루션즈 아키텍트 :: Gaming on AWS 2016
 
Reusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modulesReusable, composable, battle-tested Terraform modules
Reusable, composable, battle-tested Terraform modules
 
An intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECSAn intro to Docker, Terraform, and Amazon ECS
An intro to Docker, Terraform, and Amazon ECS
 
Book
BookBook
Book
 
KDD 2016 Streaming Analytics Tutorial
KDD 2016 Streaming Analytics TutorialKDD 2016 Streaming Analytics Tutorial
KDD 2016 Streaming Analytics Tutorial
 
Gianluca Arbezzano Wordpress: gestione delle installazioni e scalabilità con ...
Gianluca Arbezzano Wordpress: gestione delle installazioni e scalabilità con ...Gianluca Arbezzano Wordpress: gestione delle installazioni e scalabilità con ...
Gianluca Arbezzano Wordpress: gestione delle installazioni e scalabilità con ...
 
Amazon Web Services and Docker: from developing to production
Amazon Web Services and Docker: from developing to productionAmazon Web Services and Docker: from developing to production
Amazon Web Services and Docker: from developing to production
 
Higher order infrastructure: from Docker basics to cluster management - Nicol...
Higher order infrastructure: from Docker basics to cluster management - Nicol...Higher order infrastructure: from Docker basics to cluster management - Nicol...
Higher order infrastructure: from Docker basics to cluster management - Nicol...
 
Hopping in clouds - phpuk 17
Hopping in clouds - phpuk 17Hopping in clouds - phpuk 17
Hopping in clouds - phpuk 17
 
Artem Zhurbila - docker clusters (solit 2015)
Artem Zhurbila - docker clusters (solit 2015)Artem Zhurbila - docker clusters (solit 2015)
Artem Zhurbila - docker clusters (solit 2015)
 
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS LambdaImmutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
 
Docker Security workshop slides
Docker Security workshop slidesDocker Security workshop slides
Docker Security workshop slides
 
R server and spark
R server and sparkR server and spark
R server and spark
 
Manage cloud infrastructures using Zend Framework 2 (and ZF1)
Manage cloud infrastructures using Zend Framework 2 (and ZF1)Manage cloud infrastructures using Zend Framework 2 (and ZF1)
Manage cloud infrastructures using Zend Framework 2 (and ZF1)
 
Bare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and ChefBare Metal to OpenStack with Razor and Chef
Bare Metal to OpenStack with Razor and Chef
 
Terraform Cosmos DB
Terraform Cosmos DBTerraform Cosmos DB
Terraform Cosmos DB
 

Plus de Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

Plus de Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Hands-on Lab: Comparing Redis with Relational

  • 1. Page 1 of 8 Comparing Redis with Relational Hands-on Lab 1. Create an Amazon EC2 t2.micro instance with Amazon Linux  See documentation at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html  Create and use a security group that allows inbound TCP access using SSH (port 22) and MYSQL (port 3306) plus a custom rule to allow access from Redis (port 6379) You might get an automated warning that your EC2 instance is “open to the world”, because we’re not limiting the source range for SSH. This is expected. In a production system, you’ll want to provide a limited IP range for allowed SSH access. For this lab, disregard the warning.  We’ll be accessing both MySQL and Redis from this EC2 instance. Once the instance is created, go back to the security group and update the “Source” for both services to use the IP address of your EC2 instance. 2. Access the linux console. See documentation at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstances.html Use ssh for Linux or Mac; use PuTTY for Windows Example: ssh –i ~/Downloads/key.pem ec2-user@ec2-01-02-03-99.us-west-2.compute.amazonaws.com [ec2-user@ip-192-168-0-1 ~]$  Install MySql and Redis development clients $ sudo yum install mysql-devel $ sudo yum install gcc $ sudo pip install MySQL-python $ sudo pip install redis
  • 2. Page 2 of 8  Install Redis command line interface $ sudo yum --enablerepo=epel install redis 3. Create an Amazon RDS MySQL db.t2.micro instance  See documentation at http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.Cre atingConnecting.MySQL.html  Name the instance sql-lab. Be sure to choose MySQL (not Aurora and not MariaDB) and the db.t2.micro instance size  Choose a username and a password (and don’t forget them!)  Do not create a database (we will do that later)  Once the instance is created, find your mysql node name i. On the AWS Console, choose Services, then RDS ii. On the RDS dashboard, choose DB Instances
  • 3. Page 3 of 8 iii. Select the ▶ next to your DB Instance iv. Note your endpoint name. You will need it later! When using the endpoint name, you usually should not use the port extension (:3306), just the name.  Verify you can access the mysql> console from your EC2 instance $ mysql –h <mysql node name> -u <user name> -p Example: $ mysql -h sql-lab.cxpjiluqq0c9.us-west-2.rds.amazonaws.com -u awsuser -p Enter password: Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 9999 Server version: 5.6.27-log MySQL Community Server (GPL) Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. mysql> To exit from the mysql> prompt, use CTRL-D 4. Create an Amazon ElastiCache cache.t2.micro instance with Redis  See documentation at
  • 4. Page 4 of 8 http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/GettingStarted.ht ml  Name the instance redis-lab. Be sure to choose the cache.t2.micro instance size  Disable replication (just a single node)  Once the instance is created, find your ElastiCache for Redis node name i. On the AWS Console, choose Services, then ElastiCache ii. On the ElastiCache dashboard, choose Cache Clusters
  • 5. Page 5 of 8 iii. Select the ▶ next to your Cache Cluster name, then click on the Nodes iv. Note your endpoint name. You will need it later!  Verify you can access the redis> console from your EC2 instance $ redis-cli –h <redis node name> Example: $ $ redis-cli -h redis-lab.qwe34ytz.0001.usw2.cache.amazonaws.com redis-lab.sjyq2g.0001.usw2.cache.amazonaws.com:6379> To exit from the redis> prompt, use CTRL-D
  • 6. Page 6 of 8 5. Download and Prepare Landsat scenes  See documentation at https://aws.amazon.com/public-data-sets/landsat/  From your EC2 instance, download the Landsat scenes $ wget http://landsat-pds.s3.amazonaws.com/scene_list.gz  Unzip the scene list $ gunzip scene_list.gz  Trim the list to the last 250,000 scenes $ cp scene_list scene_list.orig $ tail -n 250000 scene_list.orig > scene_list 6. Load to MySQL  Log into the mysql> console  Create a landsat database mysql> CREATE DATABASE landsat; mysql> USE landsat;  Create the scene_list table mysql> CREATE TABLE scene_list (entityId VARCHAR(64), acquisitionDate DATETIME,cloudCover DECIMAL(5,2),processingLevel VARCHAR(8),path INT,row INT,min_lat DECIMAL(8,5),min_lon DECIMAL(8,5),max_lat DECIMAL(8,5),max_lon DECIMAL(8,5),download_url VARCHAR(128));  Load the landsat data mysql> LOAD DATA LOCAL INFILE 'scene_list' INTO TABLE scene_list FIELDS TERMINATED BY ',';
  • 7. Page 7 of 8 7. Load to Redis  Create the file sql2redis.py, containing the following code. Be sure to replace items in red with your own endpoints, user name and password. #!/usr/bin/python import redis import MySQLdb from collections import Counter r = redis.StrictRedis('<your redis node>',port=6379,db=0) database = MySQLdb.connect("<your MySQL node>","<your username>","<your password>","landsat") cursor = database.cursor() select = 'SELECT entityId, UNIX_TIMESTAMP(acquisitionDate), cloudCover, processingLevel, path, row, min_lat, min_lon, max_lat, max_lon, download_url FROM scene_list' cursor.execute(select) data = cursor.fetchall() for row in data: r.hmset(row[0],{'acquisitionDate':row[1],'cloudCover':row[2],'processingLevel':row[ 3],'path':row[4],'row':row[5],'min_lat':row[6],'min_lon':row[7],'max_lat':row[8],'max_ lon':row[9],'download_url':row[10]}) r.zadd('cCov',row[2],row[0]) r.zadd('acqDate',row[1],row[0]) cursor.close() database.close()  From the linux console, run the python script to cache data in Redis $ python sql2redis.py o this will take a few minutes to run. To check progress, log into your Redis node and run redis> DBSIZE
  • 8. Page 8 of 8 8. Run SQL query  Log in to your MySQL node and run a query mysql> SELECT DISTINCT(a.entityId) AS Id, a.cloudCover FROM scene_list a INNER JOIN ( SELECT entityId, acquisitionDate FROM scene_list WHERE acquisitionDate > ( SELECT MAX(acquisitionDate) FROM scene_list WHERE acquisitionDate < CURDATE() - INTERVAL 1 YEAR ) ) b ON a.entityId = b.entityId AND a.acquisitionDate = b.acquisitionDate WHERE cloudCover < 50 ORDER BY Id;  This generates a list of all the satellite images during the last year which have less than 50% cloud cover  Note how long it takes to get an answer 9. Run Redis query  Log in to your Redis node and run redis> zunionstore temp:cCov 1 cCov redis> zremrangebyscore temp:cCov 50 inf redis> zunionstore temp:acqDate 1 acqDate redis> zremrangebyscore temp:acqDate 0 1484611200 redis> zinterstore out 2 temp:cCov temp:acqDate WEIGHTS 1 0  Again, this generates a list of all the satellite images during the last year which have less than 50% cloud cover  Note how long it takes to get an answer