1. Use case study of Hive/Hadoop Eva Tse,Jerome Boulon

2. What are we trying to achieve? Scalable log analysis to gain business insights: Logs for website streaming (phase 1) All logs from web (phase 2) Output required: Engineers access: Ad-hoc query and reporting BI access: Flat files to be loaded into BI system for cross-functional reporting.

3. Some Metrics Parsing 0.6 TB logs per day Running 50+ persistent nodes

4. Architecture Overview Web App Phase 1 Phase 2 Phase 2 Chukwa Collector Log copy deamon Hive & Hadoop (for query) Hive MetaStore S3 HDFS / S3 Hive & Hadoop running on the cloud

6. Data sent to a remote collector using Thrift

7. Collector write to localFS/S3n/HDFS compressed

9. Workflow to Hive (phase 2) Continuous log collection via Chukwa Generic and continuous parse/merge/load to ‘real-time’ Hive warehouse merge at hourly boundary and load to public Hive warehouse. SLA is 2 Hr on merged data. Daily/Hourly job: For summary. For publishing data to BI for reporting.

10. Today’s Hive usage at Netflix Streaming summary data: CDN performance # of streams/day # of errors/session Test cell analysis Ad-hoc query for further analysis like: Raw log inspection Detailed inspection of one stream session Simple summary (e.g., percentile, count, max, min, bucketing) for operational metrics

11. Challenges Hive UI (for query building) Multi-DB support (Hive-675) and user access management Hive query on subset of partition files for handling late files (Hive-837 or Hive-951) Merging small files (can’t use hive.merge.mapfiles)