This document discusses big data and Hadoop. It defines big data as extremely large data sets that are difficult to process using traditional databases. Three key drivers of big data are identified as volume, variety and velocity of data. Hadoop is introduced as an open source framework for storing and processing big data across multiple machines in parallel. Examples of big data pioneers using Hadoop like Yahoo, Facebook and LinkedIn are provided. Potential uses of big data in the financial services industry are also briefly outlined.

1. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
[[The Wikibon Project]]
Big Data and Hadoop: Key Drivers,
Ecosystem and Use Cases
November 2011

2. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
What is Big Data?
2
Big Data n Data sets whose size, type
and/or speed make them impractical
to process and analyze with traditional
database technologies and related data
management tools.

3. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Why is Big Data Important?
3
Big Data is the new de.initive source
of competitive advantage across
industries …
… For those organizations that
embrace Big Data, the possibilities
for innovation, improved agility, and
increased pro.itability are nearly
endless.

4. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Three Key Big Data Drivers
4
1. Volume, Variety, Velocity
2. Hardware Commoditization
3. Cloud Computing

5. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Characteristics of Big Data
5

6. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Sources of Big Data
6

7. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Hadoop
7
Open source framework for processing, storing
and analyzing Big Data.
Fundamental concept: Rather than banging
away at one, huge block of data with a single
machine, Hadoop breaks up Big Data into
multiple parts so each part can be processed
and analyzed in parallel.

8. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Hadoop: The Pros and Cons
8
First the pros … Hadoop is a time- and
cost-effective approach to store,
process and analyze large volumes of
unstructured data allowing for new and
unprecedented types of analytics.
Now the cons … Hadoop is complex and
difficult to deploy and manage; there’s a
dearth of Hadoop-savvy engineers and
Data Scientists on the job market; the
risk of forking and vendor lock-in
remains.

9. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Hadoop: The Pros and Cons cont.
9
More pros … Many bright minds contributing to
Hadoop resulting in rapid development and an
ecosystem of vendors emerging to make Hadoop
enterprise-ready.

10. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
The Big Data Ecosystem
10

11. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Big Data Pioneers
11
• Largest Hadoop instance
on the planet … 40,000
nodes handling 200+ PB
of data.
• Used to support research
for ad systems and Web
search.
• Match ads with users,
detect spam in Yahoo!
Mail, pick relevant top
stories.

12. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Big Data Pioneers cont.
12
• Two major clusters processing and
storing over 30 PB of data.
• Uses HDFS to store copies of
internal log and dimension data.
• Developed Hive to
perform large-scale
analytics on user data.
• Using HBase to store,
manage and retrieve
Facebook Messenger
data.

13. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Big Data Pioneers cont.
13
• Uses Hadoop to support “People You May Know” feature.
• Tailors its search engine to return most relevant results
for recruiters, employers and job seekers.
• Created a visualization tool to allow users to explore their
professional network to discover hidden patterns.

14. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Big Data in Financial Services
14
• Over 30,000 databases and 15,000 applications
spread across 7 business units.
• Using Hadoop as the basis of its Common Data
Platform.
• Looking to establish 360 degree view of customer
for upsell and cross-sell opportunities.

15. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Big Data in Financial Services cont.
15
• Risk management and analysis to understand
financial exposure.
• Detecting fraudulent transactions and potentially
criminal activity.
• Conduct sentiment analysis on social media data.

16. © Wikibon 2008© Wikibon 2011 | Confidential www.wikibon.org
Thank You
16
Jeffrey F. Kelly
Principal Research Contributor
The Wikibon Project
jeff.kelly@wikibon.org
@jeffreyfkelly
www.wikibon.org
www.siliconangle.com