Autoscaling Spark for Fun and Profit 11th Spark Meetup London

1. Autoscaling Spark for Fun and
Profit
Rafal Kwasny
11th Spark London Meetup
2015-11-26
1

2. Who am I
•DevOPS
•Build a few platforms in my life
•mostly adtech, in-game analytics for Sony
Playstation
•Currently advising Investment Banks
•CTO Entropy Investments
2

3. How do you run spark?
•Who runs on AWS?
•Who uses EMR?
3

4. So how to use autoscaling on AWS?
4

5. Overview
•typical architecture for AWS
•How autoscaling works
•Scripts to make your life easier
5

6. Typical architecture for AWS
6

7. Typical architecture for AWS
7
Generate some data

8. Typical architecture for AWS
8
Store it in S3

9. Typical architecture for AWS
9
or store it in a message queue

10. Typical architecture for AWS
10
Use your favourite tool for ETL

11. Typical architecture for AWS
11
Ship it back to S3

12. Typical architecture for AWS
12
Or send it somewhere

13. Typical architecture for AWS
13
- EMR
- spark-ec2
- build cluster from scratch

14. Map-reduce is about quickly writing very inefficient
code and then running it at massive scale
(C) Someone
14

15. Problem
•EC2 is a pay-for-what-you-use model
•You just have to decide how much resources
you want to use before starting a cluster
15

16. Problem
Most common problems while running on EC2
Scaling up
•My team needs a new cluster, how big it
should be?
Scaling down
•Did I shut down the DEV cluster before leaving
the office on Friday evening?
16

17. How to automate scaling?
17

18. Types of scaling
Vertical scaling -
„Let’s get a bigger box”
•Change instance type
•Change EBS parameters
18
Horizontal scaling -
„Just add more nodes”

19. Autoscaling
•Automatic resizing based on demand
•Define minimum/maximum instance count
•Define when scaling should occur
•Use metrics
•Run your jobs and don’t worry about
infrastructure
19

20. Architecture with autoscaling
20

21. Using RAM/local SSDs for caching
Only saving output into S3

22. Fault recovery

23. Autoscaling components
•AMI - machine image with installed spark
•Launch configuration - defines:
•AMI
•instance type
•instance storage
•public IP
•security groups
23

24. Autoscaling components
•Autoscaling group
•launch configuration
•availability zones
•VPC details
•min/max servers
•when to scale
•metrics/health checks
24

25. Putting it all together
Then you can run your job
25

26. Complicated?
•AWS provides a lot of services
26

27. spark-cloud
• Better scripts to start spark clusters on EC2
• Alpha version
• https://github.com/entropyltd/spark-cloud
27

38. What’s inside spark-cloud
Building AMI’s through packer
Packer is a tool for creating machine and
container images for multiple platforms from a
single source configuration.
Supports AWS, DigitalOcean, Docker,
OpenStack, Parallels, QEMU, VirtualBox,
VMware
38

39. Current functionality
•Start cluster
•Shutdown cluster
•But more to come :)
39

40. Spot instances
•Spot instances
40

41. Spot instances
–On-Demand:
$1.400
–Spot: $0.15
–89% cheaper
41

42. Summary
•Spark and EC2 is a very common combination
•Because it makes your life easier
•And cheaper
•spark-cloud script will help you
•You can just worry about writing good Spark
code!
42

43. Thank You
rafal@entropy.be
43

44. 44

45. Amazon S3 Tips
•Don’t use s3n://
•Use s3a:// with hadoop 2.6
–Parallel rename, especially important for committing output
–Supports IAM authentication
–no „xyz_$folder$" files
–input seek
–multipart upload ( no 5GB limit )
–Error recovery and retry
More info https://issues.apache.org/jira/browse/HADOOP-10400
45

46. Why not EMR?
•Why pay for EMR? It costs more than a spot
instance
•vendor lock-in and proprietary libraries
•netlib-java
46