Talk about the experience we had building an application on Amazon EC2. Given at the PHPBenelux meeting on April 28th, 2010.

1. Developing Social Games
in the Cloud
Pieter De Schepper
Jurriaan Persyn

2. Tags
Cloud Computing
IaaS (Infrastructure as a Service)
PaaS (Platform as a Service)
AWS (Amazon Web Services)
Amazon EC2
Social Applications
Multi Platform
Game Development

3. Who we are
Pieter De Schepper Jurriaan Persyn
previously working on Netlog previously part of the front-end
APIs, OpenSocial & other development team focusing on
integrations in Netlog user features
Netlog team member for 3 Netlog team member for 4
years years

4. Gatcha! wants to bring people
together through gaming wherever
they are.

5. Gatcha! builds and distributes
games to
social networks and web-portals.

6. Who we are
For Game Developers For Site Owners
We offer: We offer:
• Reach & distribution • Quality gaming content
• Social Enrichment • Social engagement
• Features • Monetization
• High-scores
• Achievements
• Challenges
• ...
• Monetization
• Localization

7. Every single day:
over 1M game plays
over 2200 days of play time

8. Where we come from
Grown and successful as one of
Netlogʼs features
Since Jan 2010 a dedicated team of 8
with a more stand-alone approach to
go cross-platform.

10. UC Berkeley RAD Lab Definition
Huge Resources
The illusion of infinite computing resources available on demand,
thereby eliminating the need for users to plan far ahead for
provisioning resources.
No Commitment
The elimination of an up front commitment by users, thereby
allowing companies to start small and increase resources only
when there is an increase in their needs.
Pay By The Drink
The ability to pay for use of computing resources on a short-term
basis as needed (e.g., processors by hour and storage by the
day) and release them as needed.

11. Types
Infrastructure as a Service (IaaS)
Amazon EC2, GoGrid, Eucalyptus, ...
Platform as a Service (PaaS)
Google AppEngine, Windows Azure, Amazon Simple DB, ...
Software as a Service (SaaS)
Gmail, Google Documents, ...

12. Why IaaS?
You donʼt want to worry ...
• about building an expensive datacenter
• about the location of your datacenter
• about finding the right ITS team & field engineers
• about broken discs
• about hardware getting out of fashion
• ...
(minimal CAPEX, Capital Expenditures)

13. But?
We have a dedicated team
We have a datacenter
We have negotiated contracts
So, why?

14. Our main goal ...
• instant scaling
• the promise of hardware on demand
• handling peaks in traffic
• handling heavy single tasks
• cost optimization

15. Why Amazon?
Simple Storage Service (S3)
since March 2006
Elastic Cloud Computing (EC2)
since August 2006
Almost immediately de facto standard
for cloud computing
Others: Rackspace, GoGrid

16. Amazon Web Services
IaaS
S3: Simple Storage Service
EC2: Elastic Compute Cloud
EBS: Elastic Block Storage
PaaS
SQS: Simple Queue System
RDS: Relational Database Service
SDB: SimpleDB
Cloudfront: Content Delivery

17. What is Amazon EC2?
Amazon EC2
is a virtual computer grid
provisioning virtual environments
on Amazon infrastructure

18. What is Amazon EC2? (contʼd)
Elastic capacity
easy scaling within minutes
Full Control
root-level system access
Flexible
multiple instance types & sizes
Can be used in combination with other services
e.g. S3, SimpleDB, SQS, ...
Secure
network security model

19. Amazon Concepts
AMI - Amazon Machine Images
packaged-up environment for setting up and
booting your instance
Availability Region
geographically dispersed in separate geographic
areas or countries
Availability Zone
distinct datacenter locations that are engineered
to be insulated from failures
External IPʼs
mapped via NAT on Amazon routers
1 external IP -> 1 internal IP (max 2 IPs / instance)
Security
firewall security groups

20. Amazon Concepts (contʼd)
CU - Computing Unit
1 CU = 1 GHz 2007 opteron/xeon processor
each instance has a certain number of CUʼs
IAM - Identity Authentication Management
AWS Credentials & X.509 Certificates
Load Balancing
AWS auto-scaling
API
Web API, Command-Line Tools
Management
AWS Management Console
Firefox Elasticfox plug-in
Security
firewall security groups

21. Storage Solutions
EC2 Instance Default Local Storage
Ephemeral virtual disks that are integral part of EC2 VM
instance
Range from 160GB to 1.7 TB total space
Only use this for temporary storage

22. Storage Solutions (contʼd)
EBS - Elastic Block Share
Persistent disk volumes that can be mounted on your instances
1 GB to 1 TB per volume, default quota of 20 volumes
You can create any filesystem on EBS volumes
Easy to backup with snapshots
Usage:
• Create volume (same availability region)
• Attach to instance
• Create filesystem
• Mount filesystem in your operating system

23. Storage Solutions (contʼd)
S3 File Storage
Reliable, Web URL accessible file-based storage
Doesnʼt really store files, but objects, buckets and keys
Max 5GB per bucket, unlimited amount of buckets
Use:
• Backups
• Storing Bundled AMIʼs

24. </hype>

25. <?php
// Getting Started
// with Amazon EC2
echo "Hello Cloud!";
?>

27. Amazon EC2 sign up
Contact information
Billing information
credit card
Pricing information
since August 2006

29. The first Instance
Choosing an AMI
from a list of Amazon AMI
user uploaded AMIʼs
(be careful of which AMIʼs you trust)
typical LAMP setups available
Gearman AMIʼs, memcached AMIʼs, MySQL AMIʼs, ...
Creating a key-pair
= security credentials
public/private keypair for authentication
Creating a security group
= firewall rules
Access instance
ssh -i keypair.pem root@ec2-
<...>.compute-1.amazonaws.com

30. More tools
RESTful / SOAP API
several php API clients available
some maintained by Amazon
Command Line Tools
launch/terminate/... from CLI
ElasticFox
FF plugin
similar to AWS console

31. Gatcha EC2 Setup
Management Servers Application servers
Inventory Management Web Servers
Configuration Management Databases
Monitoring Service Memcache Servers
DNS Service Gearman
Load Balancing Service Smartfox
...
Development server

32. Gatcha EC2 Setup (contʼd)
Inventory Management
Which servers are running?
What is their role?
What is their instance id?
Read:
SQLite database and API with list of instances
and their core meta-data

33. Gatcha EC2 Setup (contʼd)
Configuration Management
via puppet:
“Open Source Configuration Management Tool”
Custom declarative language to describe system configuration
“Templates” to describe parts of the system
users / services / packages
On deploy
Puppet puts managed systems into declared state
Read:
Puppet installs right version of Apache, MySQL.
Configures users, access, ...

34. Gatcha EC2 Setup (contʼd)
Monitoring
via Zabbix:
“Open Source Distributed Monitoring Solution”
(Amazon offers “CloudWatch”)
Read:
Periodic polling of all instances and get health &
performance parameters. Trigger alerts when
needed.

35. Gatcha EC2 Setup (contʼd)
Internal DNS Server
via Bind9:
“Berkeley Internet Name Domain DNS Server”
Read:
Instances talk to each other via hosts names.
hostname to ip translation via internal DNS
server

36. Gatcha EC2 Setup (contʼd)
Load Balancing
via Amazon Elastic Load Balancing
= Amazon Service
Amazon provides failover
Public via Elastic IP
Distributing incoming traffic between all instances in same
availability zone
Performs health checks (polling of instances)
Also possible via own instances, eg. HaProxy AMIʼs
Read:
Responsible for distributing user requests to
array of web servers

37. Gatcha EC2 Setup (contʼd)
Applications Servers
Servers that need CPU & Servers that need
Memory persistent storage
web servers databases
gearman order services: Amazon SDB, Amazon
other services: Amazon SQS RDS
smartfox
memcached
other services: gear6
2 AMIʼs:
instance store AMI
EBS AMI

38. (Auto) Scaling
Typical set up of new web server
custom AMI w/ Debian distribution
boot script
sets up SSH keys
configures hostname
updates internal DNS server
further provisioning via Puppet
ultimately
health checks to Zabbix
contact AmazonLoadBalancer API and start serving requests

39. (Auto) Scaling
Zabbix sees high load on web servers?
Triggers bash script to create new web
launch new instance on Amazon
add to Inventory Management
boot scripts and puppet bring it into production
Zabbix sees low load on web servers?
Triggers bash script to terminate instance

40. Backups
Backups
via:
“Open Source Network Backup Solution”
Read:
Separate service that polls Inventory
Management for list of database.
Create EBS snapshots for volumes of those DBs

41. ... but ...

42. Gotchaʼs

43. Gotchaʼs
Default storage = instance storage (ephemeral)
If instance goes down -> storage is gone
Persistent storage: use EBS bases AMIʼs
No external static IP available for instances
Use Elastic IP (only when necessary)
No guarantee that your instance will run forever
No need to worry about hardware, but... what is
Amazon doing, how, when, why?
Donʼt trust any AMI youʼll stumble upon

44. Some obstacles
Database server crashes
Eventually found out it was related to kernel.
No possibility to create own kernels.
The most recent kernel on Amazon was 3 years
old
Experience with Amazon Support wasnʼt that
smooth.

45. Some obstacles (contʼd)
Custom VPN
Still working out the best way to create and
manage the VPN over both Netlog & Gatcha
servers.
Amazon servers are a maze of networks. Difficult
to control in custom VPN.
Amazon offers Virtual Private Cloud
Currently only in US-EAST

46. Extra services
RightScale
“Cloud Computing Management Platform”
A management tool, with different features atop of AWS, enabling
you to take full control of all concepts of your clouds
Also provides support and consultancy
You pay for what you get (pretty expensive)
Power users:
• Zynga (FarmVille, Maffia wars, Zynga Poker, ...)
• Playfish (Hotel City, Word Challenge, ...)

47. Extra services (contʼd)
Scalr
Cheaper alternative to RightScale
Part Open Source
Focus
Easy scaling: scaling your database, application servers and
load balancers
Monitoring: automatically replaces servers when one fails
Backups: makes automated backups to Amazon EBS

48. Impact on application
Focus on application needs, not on hardware
specs.
What do I need today?
Upgrade hardware when itʼs needed.
(e.g.. start with normal instance for you db, upgrade to large, extra-
large when needed.)

49. Impact on application (contʼd)
Dynamic Configuration
e.g.. New Memcached server?
Add it to your configuration files
(Use consistent hashing algorithms for Memcached)

50. Impact on application (contʼd)
Stand alone system
Try to identify components in your application
that could stand alone.
Loosely-coupled systems are easier to scale.

51. Impact on application (contʼd)
Auto Scaling not simple for database servers
Adding MySQL slaves can be automated
for scaling READS
Scaling databases is still the hardest part
AWS only helps us in scaling infrastructure.
Scalability is more than that.

52. Impact on application (contʼd)
“Everything fails, all the time.”
Avoid single points of failure.
Design for failure.

53. Conclusion
Set-up
Initial set-up is very simple and easy.
Enough solutions, partners and documentation
to help you out.
For custom stuff, it gets harder.
Integration with existing monitoring / backup /
configuration management not that simple.
You depend on Amazon.
Thereʼs always the unexpected stuff.

54. Conclusion
The money?
+ No expensive set-up costs.
No long term commitments to make.
No planning for hardware.
Impact of performance improvements (cpu/
bandwidth) has direct relation to money
saved.
- Consultancy and Cloud manager
tools are expensive.

55. Conclusion
Pricing Scenario per month
Set up:
10 web & memcached servers
2 gearman job processors
2 databases
1TB provisioned storage
3TB data transfer from public ips
(these are not the Gatcha! figures)

56. Conclusion
Amazon Elastic Compute Cloud - EU (Ireland) Region
Amazon EC2 running Linux/UNIX
$0.38 per Large Instance (m1.large) instance-hour (or partial hour)
e.g.. 10 web, memcached, load balancer, ... servers
10 * 24 * 31 = 7400 hours
$2812
$0.76 per Extra Large Instance (m1.xlarge) instance-hour (or partial
hour)
e.g.. 2 database servers
2 * 24 * 31 = 1488 hours
$1130
$0.76 per High-CPU Extra Large Instance (c1.xlarge) instance-hour (or
partial hour)
e.g.. 2 gearman worker server
2 * 24 * 31 = 1488 hours
$1130

57. Conclusion
Amazon EC2 EBS
$0.11 per GB-month of provisioned storage
e.g.. 1TB-Mo
$110
$0.11 per 1 million I/O requests
e.g.. 2OOM IOs
$22
$0.15 per GB-Month of snapshot data stored
e.g.. 10GB-Mo
$15
$0.010 per 10,000 gets (when loading a snapshot)
e.g.. 150k requests
$0.15
$0.010 per 1,000 puts (when saving a snapshot)
e.g.. 30k requests
$0.30

58. Conclusion
Elastic IP Addresses
$0.01 per non-attached Elastic IP address per complete hour
e.g.. 1 ip you want to keep in use
1 * 24 * 31 = 744 hours
$8
Elastic Load Balancing
$0.028 per LoadBalancer-hour (or partial hour)
e.g.. 1 load balancer
1 * 24 * 31 = 744 hours
$21
$0.008 per GB Data Processed by the LoadBalancer
insignificant

59. Conclusion
AWS Data Transfer (excluding Amazon CloudFront) (EU (Ireland) Region)
$0.150 per GB - up to 10 TB / month data transfer out
e.g.. 200GB
$30
$0.000 per GB - first 1 GB / month data transfer out
$0
$0.000 per GB - data transfer in (free Until June 30, 2010)
$0
$0.010 per GB - regional data transfer - in/out/between EC2 Avail Zones or
when using public/elastic IP addresses or ELB
e.g.. 3TG
$30

60. Total:
$5308

61. Conclusion
The money? (contʼd)
If you have a fairly static set up, own hardware
will be cheaper.
Not being able to quickly scale has itʼs price
too.
Ideally: use hybrid set up where Amazon helps
in handling peak traffic or unexpected traffic.
(Typical situation: 1/3 of servers only in use for 1/5
of the time, to be able to handle peak.)

62. Conclusion
Get:
Flexible resource handling
Minimal capital expenses
Need:
Efficiency needs automation
Effectiveness needs experience

65. Conclusion
AWS allows you to focus on
product
and
users
Amount of hardware available buys you time.
You can temporarily solve problems with
“throwing hardware at the problem”

66. VELO PERS
FO R DE

67. A ME High-score Handling
R G
YOU Tournaments
A ME Challenge builder
IA LG Achievements
S OC
OP
AT
Got an idea for a game? Great!

68. Gatcha For Game Developers
Game tracking
Start game and end game calls results in accurate gameplay
tracking and allows us to show who is playing the game at any
given moment, compute popularity, target games.
High-scores
You push your high-score to our API, we do the hard work of
creating different types of leader boards and rankings.
Achievements
Pushing achievements reached in your game, just takes one API
call, no configuration needed.

69. Gatcha For Game Developers
Multiplayer Games
We run SmartFox servers that enable you to build real-time
multiplayer games, with e.g.. in game chat
coming:
Challenges & Tournaments
Allow your game players to challenge each other, or build
challenges & contests yourself.

70. Gatcha For Game Developers
How to integrate?
Flash Games
We offer wrapper for AS3 and AS2 games with full
implementation of our API
Unity3D Games
OpenSocial Games
Talk to the supported containers via the Gatcha OpenSocial
Extension
Other Games
Simple iframe implementation. PHP Client API available for the
Gatcha API
Start developing in our sandbox.

74. Job openings
Weʼre searching for great developers!
PHP Talents
Working on integrations and the gaming platform
Flash Developers
Working on Flash Games and the gaming platform
Design Artists
Designing games and integrations

75. Want to know more?
Ask.
pieter@netlog.com
jurriaan@netlog.com
toon@netlog.com
www.gatcha.com

76. sources
developer.amazonwebservices.com
hyperstratus.com
rightscale.com
eucalyptus.com
jochen-lillich.de
gatcha.com
pictures
Iko - flickr.com/photos/iko/106957481
Marco Fulle - antwrp.gsfc.nasa.gov