The document discusses the Lambda architecture, its advantages and disadvantages, and how Kudu can serve as an alternative. The Lambda architecture marries batch and real-time processing by using separate batch, speed, and serving layers. While it provides scalability, maintaining two code bases is complex. Kudu can fill the gap by enabling both fast analytics on frequently updated data through its ability to support updates, scans and lookups simultaneously. Examples of how Kudu has been used by Xiaomi to simplify their analytics pipeline and reduce latency are provided. The document cautions against premature optimization and advocates optimizing only as needed.

1. 1© Cloudera, Inc. All rights reserved.
Michael Crutcher
Director, Product Management - Storage
Lambda Architecture

2. 2© Cloudera, Inc. All rights reserved.
Agenda
• Big Data Challenges
• What is Lambda?
• Lambda Advantages and Disadvantages
• Kudu as a Lambda alternative

3. 3© Cloudera, Inc. All rights reserved.
Big Data Challenges

4. 4© Cloudera, Inc. All rights reserved.
“Something interesting is happening”
The world’s largest
taxi company owns
ZERO vehicles.
The world’s largest
accommodation provider
owns ZERO real estate.
The world’s most
popular media owner
creates ZERO content.
The world’s leading
music platform owns
no music.

5. 5© Cloudera, Inc. All rights reserved.
Data is now a strategic asset
Instrumentation
Consumerization
Experimentation
Today, everything that can be
measured will be measured.
Today, data IS the
application.
Today, becoming data-driven
is a business imperative.

6. 6© Cloudera, Inc. All rights reserved.
“It will soon be technically
feasible & affordable to
record & store everything…”
— New York Times
“Digital technologies will, in
the near future, accomplish
many tasks once considered
uniquely human.”
.
— Second Machine Age
Data is abundant,
diverse & shared freely
As is how we store,
process and analyze it
Streaming Machine Learning BI
ETL Modeling

7. 7© Cloudera, Inc. All rights reserved.
The new analytics paradigm
Understand
why it
happened
Change
what
happens
next
Determine
what
happened
Make it
happen
consistently

8. 8© Cloudera, Inc. All rights reserved.
So Why Big Data?
What does the reporting look
like at your business today?
What if it could happen in half
the time, or half that time?
What data are you looking at?
What data do you want to know
about your customers? How
can you best use external data?
Too often data is archived,
combined, or simplified to save
space and strain on systems.
Once data is combined we loose
the ability to dig deeper.
Better Business Forecasting Better Views of CustomersFull Fidelity Data Access

9. 9© Cloudera, Inc. All rights reserved.
What is Lambda architecture?

10. 10© Cloudera, Inc. All rights reserved.
What is Lambda Architecture?
Batch Layer
Serving Layer
Speed Layer
New Data
Data Lake
(HDFS)
Precompute
Views
Stream or
Micro Batch
Increment
Views
Data
Application
“Real-time” Increment
Batch Recompute
Merge
Hadoop
Storm/Spark
HBase
Impala

11. 11© Cloudera, Inc. All rights reserved.
Batch Layer
• Manages the master data set, an immutable, append-only set of raw data
• Pre-computes views of the data
• “Traditionally” this has been in HDFS and processed with Map/Reduce
• There has already been some shift to cloud based object storage and processing
in other frameworks like Spark

12. 12© Cloudera, Inc. All rights reserved.
Speed Layer
• This layer ingests streaming data or micro-batches
• Spark and Storm are traditionally used
• In some cases micro-batches are directly ingested into NoSQL data stores like
HBase
• This data is periodically expunged
• In many “Lambda-like” architectures I’ve seen, this layer is used to provide an
“active partition” that provides a limited window of mutability

13. 13© Cloudera, Inc. All rights reserved.
Serving Layer
• As you might guess from the name, this is the layer that serves data
• It would be unusual for raw data to be served directly
• This could be an application written directly against a data store like HBase
• It could be a SQL engine on top of a file system, Impala + Parquet is an example

14. 14© Cloudera, Inc. All rights reserved.
What is a Kappa Architecture?
Batch Layer
Serving Layer
Speed Layer
New Data
Data Lake
(HDFS)
Precompute
Views
Stream or
Micro Batch
Increment
Views
Data
Application
“Real-time” Increment
Batch Recompute
Merge
Hadoop
Storm/Spark
HBase
Impala

15. 15© Cloudera, Inc. All rights reserved.
Everything Has a New Name
Batch Layer
Serving Layer
Speed Layer
New Data
Data Lake
(HDFS)
Precompute
Views
Stream or
Micro Batch
Increment
Views
Data
Application
“Real-time” Increment
Batch Recompute
Merge
System of Record (OLTP)
Operational Data Store Derived Tables (EDW)
In-Memory Database
Star/Snowflake, Cubes,
or In-Memory Tables

16. 16© Cloudera, Inc. All rights reserved.
The Log as Storage
• Lambda and Kappa architectures are both predicated on immutable source data
• Data can be modeled as a series of events recorded at specific points in time
about entities
• Updates are modeled as new events and the current or historic value associated
with an entity can be reconstructed through the collected events
• Kappa calls this ordered set of events “a log”, it’s safe to say they didn’t invent
this term
A B C D B E F A G B
1 10
Ordered over Time

17. 17© Cloudera, Inc. All rights reserved.
Is Raw Data the Right Logical Model?
• It’s possible to derive many higher level logical abstractions from raw data
• As an example, I could construct a customer account balance from raw account
activity data
• This doesn’t mean it’s a good idea
A B C B A C B A A C
t0 t12Account Activity
+$10 +$20 +$15 -$10 +$35 -$5 +$25 +$15 -$20 +$10
Easy:
What was the last account event for Customer C?
Harder:
What is the account balance for Customer A at t12?

18. 18© Cloudera, Inc. All rights reserved.
There are Only Two Hard CS Problems
1) Cache invalidation
2) Naming things
-- Phil Karlton

19. 19© Cloudera, Inc. All rights reserved.
Data Engineering has one hard problem
• When should I denormalize to maximize performance?
• When should I normalize to minimize maintenance problems?
Denormalize Everything!
Normalize Everything!
I wish things were faster!
I wish things were easier
to maintain!

20. 20© Cloudera, Inc. All rights reserved.
Lambda Advantages and
Disadvantages

21. 21© Cloudera, Inc. All rights reserved.
Lambda Advantages
• Marries diverse strengths of existing open source software into a unified
architecture
• Provides scalability via the batch layer
• Provides real time performance via the speed layer

22. 22© Cloudera, Inc. All rights reserved.
Lambda Disadvantages
• Complexity
• Many moving parts
• Restatement is difficult
• Two code bases must be kept in sync
• Proper failure handling is complex

23. 23© Cloudera, Inc. All rights reserved.
Lambda Complexity
Batch Layer
Serving Layer
Speed Layer
New Data
Data Lake
(HDFS)
Precompute
Views
Stream or
Micro Batch
Increment
Views
Data
Application
“Real-time” Increment
Batch Recompute
Merge
Hadoop
Storm/Spark
HBase
Impala
Code must be kept in sync
Restatement is difficult

24. 24© Cloudera, Inc. All rights reserved.
Lambda Complexity
Batch Layer
Serving Layer
Speed Layer
New Data
Data Lake
(HDFS)
Precompute
Views
Stream or
Micro Batch
Increment
Views
Data
Application
“Real-time” Increment
Batch Recompute
Merge
Hadoop
Storm/Spark
HBase
Impala
Hmm… this data looks fishy
Problem Here?
Here?
Here?
Here?
Here?
Here?
Here?

25. 25© Cloudera, Inc. All rights reserved.
The Log as Storage
• The idea of representing data as immutable log information is not new and is not
without tradeoffs:
• Space amplification: how many bytes of data are stored, relative to how many
logical bytes the database contains
• Write amplification: how many bytes of data are written by the database
compared to the number of bytes changed by the user
• Read amplification: how many bytes the database has to physically read to
return values to the user compared to the bytes returned
• Complexity: am I solving a CS problem or a customer problem?
• These are not simple issues and there’s no straightforward “right” answer

26. 26© Cloudera, Inc. All rights reserved.
Premature Optimization
Programmers waste enormous amounts of time thinking about, or worrying about,
the speed of noncritical parts of their programs, and these attempts at efficiency
actually have a strong negative impact when debugging and maintenance are
considered. We should forget about small efficiencies, say about 97% of the time:
premature optimization is the root of all evil.
--Donald Knuth

27. 27© Cloudera, Inc. All rights reserved.
Gap Filling vs. Optimization
• Some Lambda implementations are deployed on big data systems that don’t require
significant optimization to deliver desired SLAs
• Often, Lambda architectures are used to fill the very stark difference in workload
processing capabilities of technologies that are used typically used for the batch (long
scan) and fast layers (quick point lookups)
• Anecdotally, Lambda architectures seem to be deployed much more often with current
generation open source technology than they were with legacy commercial offerings
• Part of this is because of data volume, variety, and velocity caused by our increasingly
data driven world, but I think part of this is also because legacy technologies haven’t had
as stark of a difference in what workloads they’re optimal for
• Are you deploying a Lambda architecture because you need to squeeze out all of the
performance possible, or because you have a mixed workload that can’t be deployed on
one single storage technology?

28. 28© Cloudera, Inc. All rights reserved.
Gap Filling v2: Lack of Mutability
• Some Lambda implementations aim to fill the gap of
the lack of mutability in HDFS
• Raw, master data should be immutable, but in the real
world raw data could potentially need to be adjusted
• Sensors could have been miscalibrated, data may have
been incorrectly entered, raw data might be an
approximation before finalization, etc.
• Derived aggregations might more efficiently modified
in place, vs. recalculated from raw data, recalculating
all of history is often not practically possible
Incoming Data
(Messaging
System)
New Partition
Most Recent Partition
Historic Data
HBase
Parquet
File
• Wait for running operations to complete
• Define new Impala partition referencing the
newly written Parquet file
Reporting
Request
Impala on HDFS

29. 29© Cloudera, Inc. All rights reserved.
Kudu as a Lambda alternative

30. 30© Cloudera, Inc. All rights reserved.
HDFS
Fast Scans, Analytics
and Processing of
Stored Data
Fast On-Line
Updates &
Data Serving
Arbitrary Storage
(Active Archive)
Fast Analytics
(on fast-changing or
frequently-updated data)
Kudu: Fast Analytics on Fast-Changing Data
New storage engine enables new Hadoop use cases
Unchanging
Fast Changing
Frequent Updates
HBase
Append-Only
Real-Time
Kudu Kudu fills the Gap
Modern analytic
applications often
require complex data
flow & difficult
integration work to
move data between
HBase & HDFS
Analytic
Gap
Pace of Analysis
PaceofData

31. 31© Cloudera, Inc. All rights reserved.
Kudu Increases the Value of Time Series Data
Time Series
Inserts, updates, scans, lookups
Workload
Examples
Stream market data, fraud detection &
prevention, risk monitoring
Time series data is most valuable if you can
analyze it to change outcomes in real time.
Kudu simulateneously enables:
• Time series data inserted/updated as it arrives
• Analytic scans to find trends on fresh time series data
• Lookups to quickly visit the point in time where an
event occurred for further investigation

32. 32© Cloudera, Inc. All rights reserved.
Kudu can help spot problems before they
happen. Real-time data inserts with the ability to
analyze trends identifies potential problems.
Kudu identifies trouble through:
• Extreme scale, allowing better historic trend analysis
• Fast inserts to enable an up-to-date view of your
business
• Fast scans identify/flag undesired states for remedy
Kudu Keeps Your Business Operational
Machine Data
Analytics
Inserts, scans, lookups
Workload
Examples
Network threat detection, IoT, predictive
maintenance and failure detection

33. 33© Cloudera, Inc. All rights reserved.
More Versatility in Online Reporting
Online
Reporting
Inserts, updates, scans, lookups
Workload
Examples
“Active” Reporting
Online reporting has traditionally been limited by
data volume and analytic capabilitiy, keeping
only recent data designed for granular queries.
Kudu adds online reporting versatility through:
• Fast inserts and updates to keep data fresh
• Fast lookups and analytic scans in one data store

34. 34© Cloudera, Inc. All rights reserved.
Xiaomi use case
• World’s 4th largest smart-phone maker (most popular in China)
• Gather important RPC tracing events from mobile app and backend service.
• Service monitoring & troubleshooting tool.
High write throughput
• >5 Billion records/day and growing
Query latest data and quick response
• Identify and resolve issues quickly
Can search for individual records
• Easy for troubleshooting

35. 35© Cloudera, Inc. All rights reserved.
Xiaomi big data analytics pipeline
Large ETL pipeline delays
● High data visibility latency
(from 1 hour up to 1 day)
● Data format conversion woes
Ordering issues
● Log arrival (storage) not
exactly in correct order
● Must read 2 – 3 days of data
to get all of the data points
for a single day

36. 36© Cloudera, Inc. All rights reserved.
Xiaomi big data analytics pipeline
Simplified with Kudu
Low latency ETL pipeline
● ~10s data latency
● For apps that need to avoid
direct backpressure or need
ETL for record enrichment
Direct zero-latency path
● For apps that can tolerate
backpressure and can use the
NoSQL APIs
● Apps that don’t need ETL
enrichment for storage /
retrieval
OLAP scan
Side table lookup
Result store

37. 37© Cloudera, Inc. All rights reserved.
Conclusions
• Lambda has a real place in big data architectures
• Optimize as needed, but beware of the cost of premature optimization
• Kudu is designed to be a simple solution for when you need a data store that’s
updatable and provides “good enough” performance for analytic and real time
workloads simultaneously