This session will cover a series of use cases where you can store your data cheaply in files and analyze the data with Apache Spark, as well as use cases where you want to store your data into a different data source to access with Spark DataFrames. Here’s an example outline of some of the topics that will be covered in the talk:
Use cases to store in file systems for use with Apache Spark:
- Analyzing a large set of data files.
- Doing ETL of a large amount of data.
- Applying Machine Learning & Data Science to a large dataset.
- Connecting BI/Visualization tools to Apache Spark to analyze large datasets internally.
10. Just in Time Data Warehouse w/ Spark
and more…
HDFS
11. Separate Compute vs. Storage
11
Benefits:
• No need to import your data into Spark to begin
processing.
• Dynamically Scale Spark clusters to match compute
vs. storage needs.
• Choose the best data storage with different
performance characteristics for your use case.
12. 12
Know when to use other data stores
besides file systems
Today’s Goal
14. Good: General Purpose Processing
Types of Data Sets to Store in FileSystems:
• Archival Data
• Unstructured Data
• Social Media and other web datasets
• Backup copies of data stores
14
15. Types of workloads
• Batch Workloads
• Ad Hoc Analysis
– Best Practice: Use in memory caching
• Multi-step Pipelines
• Iterative Workloads
15
Good: General Purpose Processing
18. Bad: Random Access
sqlContext.sql(
“select * from my_large_table where id=2I34823”)
Will this command run in Spark?
Yes, but it’s not very efficient — Spark may have
to go through all your files to find your row.
18
19. Bad: Random Access
Solution: If you frequently randomlyaccess your
data, use a database.
• For traditional SQL databases, create an index
on your key column.
• Key-Value NOSQL stores retrieves the value
of a key efficiently out of the box.
19
20. Bad: Frequent Inserts
sqlContext.sql(“insert into TABLE myTable
select fields from my2ndTable”)
Each insert creates a new file:
• Inserts are reasonably fast.
• But querying will be slow…
20
21. Bad: Frequent Inserts
Solution:
• Option 1: Use a database to support the inserts.
• Option 2: Routinely compact your Spark SQL table files.
21
22. Good: Data Transformation/ETL
Use Spark to splice and dice your data files any way:
File storage is cheap:
Not an “Anti-pattern” to duplicately store your
data.
22
23. Bad: Frequent/Incremental Updates
Update statements — not supported yet.
Why not?
• Random Access: Locatetherow(s) in the files.
• Delete &Insert: Delete the old row and insert a new one.
• Update: Fileformats aren’t optimized for updating rows.
Solution:Manydatabasessupport efficient update operations.
23
24. Use Case: Up-to-date, liveviews of your SQL tables.
Tip: Use ClusterBy for fast joins or Bucketing with 2.0.
Bad: Frequent/Incremental Updates
24
Incremental
SQL Query
Database
Snapshot
+
25. Good: Connecting BI Tools
Tip: Cache your tables for optimal performance.
25
HDFS
29. Good: Machine Learning & Data Science
UseMLlib, GraphXandSparkpackagesformachine
learninganddatascience.
Benefits:
• Built in distributedalgorithms.
• In memorycapabilitiesfor iterativeworkloads.
• All in one solution:Data cleansing,featurization,
training, testing, serving,etc.
29
30. Bad: Searching Content w/ load
sqlContext.sql(“select * from mytable
where name like '%xyz%'”)
Spark will go through each row to find results.
30
32. Good: Periodic Scheduled Jobs
Schedule your workloads to run on a regular basis:
• Launch a dedicated cluster for important workloads.
• Output your results as reports or store to a
files/database.
• Poor Man’s Streaming: Spark is fast, so push the
interval to be frequent.
32
33. Bad: Low Latency Stream
Processing
Spark Streaming can detect new files dropped into a
folder to process, but there is a delay to build up a
whole file’s worth of data.
Solution: Send data to message queues not files.
33