A comparison of different tools for change management in big data systems.

1. BIG DATA’S JOURNEY TO ACID
Owen O’Malley
owen@cloudera.com
October 2019
@owen_omalley

2. WHY IS ACID IMPORTANT?

3. © 2019 Cloudera, Inc. All rights reserved. 3
BIG DATA HAS A LOT OF CONCURRENCY
• Your data changes continually.
• Daily, hourly, or faster
• Ad hoc solutions require a lot of work
• Producers and consumers must agree
• Distributed systems have lots of actors
• And no global clock

4. © 2019 Cloudera, Inc. All rights reserved. 4
USE CASES
• Updating dimension tables
• Changing a user’s address
• Deleting old records
• GDPR user removal
• Update/restate large fact tables
• Fix problems after they are in the warehouse
• Streaming data ingest
• NOT OLTP

5. THE SYSTEMS

6. © 2019 Cloudera, Inc. All rights reserved. 6
APACHE HADOOP MAP/REDUCE
• Only supporting adding new directories
• Provided isolation via the output committer.
• Task isolation
• Job isolation
• Used HDFS atomic renames
• Used _SUCCESS_ file to mark available directories

7. © 2019 Cloudera, Inc. All rights reserved. 7
APACHE HBASE
• Provided point lookup and edits
• Read & Write performance – low latency, low throughput
• Row level atomicity
• Tephra provided transactions, but lacks adoption
• Write-Ahead Log (WAL)
• Regular compactions

8. © 2019 Cloudera, Inc. All rights reserved. 8
TRADITIONAL APACHE HIVE
• Provided Hive Meta-Store (HMS) to track tables
• Provided structure for table layout
• Value partitioning
• Only add or remove partition operations were atomic
• Only add partition was isolated
• Provided simplistic locking

9. © 2019 Cloudera, Inc. All rights reserved. 9
APACHE HIVE ACID
• Supports streaming writes
• Integrated with SQL data manipulation commands
• Insert, delete, update, merge
• Snapshot isolation
• Read & Write performance: high throughput, high latency
• Lockless compaction
• Writes delta directories
• Assumes HDFS consistent directory listings

10. © 2019 Cloudera, Inc. All rights reserved. 10
APACHE HUDI
• Designed for streaming data
• Row level updates
• WAL & compaction
• Assumes HDFS
• Provides three reading levels:
• Compacted
• Compacted + deltas
• Deltas

11. © 2019 Cloudera, Inc. All rights reserved. 11
APACHE ICEBERG
• Designed to support data in object stores (eg. S3)
• Avoids inconsistent & slow directory listing
• Tracks tables and partitions to file level
• Supports column min, max, and count per file
• Snapshot isolation
• Writers automatically retry on conflict
• Manifest files use copy on write
• Supports time travel and rollback

12. © 2019 Cloudera, Inc. All rights reserved. 12
DATABRICKS DELTA
• Open-source, but closed governance
• Ignoring the proprietary version
• Designed for object stores
• Avoids inconsistent & slow directory listings
• Snapshot isolation
• Add, replace, remove data files

13. CONCLUSIONS

14. © 2019 Cloudera, Inc. All rights reserved. 14
CONCLUSIONS
• GDPR is huge and leading to redesign of data warehouse
• Support for object stores like S3 is critical
• Streaming ingest and processing is growing quickly
• This area is under active development
Will change over the next 6 months
Hive ACID is adding Presto & Impala support.
Iceberg is adding delta files and Hive support

15. © 2019 Cloudera, Inc. All rights reserved. 15
OVERVIEW OF HIGH THROUGHPUT SYSTEMS
SQL data
data ops
Open Write Amp
Amp
Object Store
Store
Stream ingest
ingest
Engines
Hive ACID Yes Govern Low Poor Good RW: Hive;
R: Spark, Impala
Hudi No Govern Low Poor Good RW: Spark;
R: Hive, Presto
Iceberg No Govern High Good Poor RW: Spark, Presto;
R: Pig
Delta No Source High Good Poor RW: Spark;
R: Presto

16. THANK YOU
Owen O’Malley
owen@cloudera.com
@owen_omalley