Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter
•
2 j'aime•507 vues
Tajo is an advanced open source data warehouse system on Hadoop. Tajo has rapidly evolved over couple of years. In this talk, I will present how Tajo has been improved for years. In particular, this talk will introduce new features of the most recent major release Tajo 0.10: Hbase storage support, thin JDBC driver, direct JSON support, and better Amazon EMR support. Then, I will present the upcoming features that currently Tajo community is doing: Multi-tenant scheduler, allowing multiple users to submit multiple queries into one cluster; nested schema support, allowing users to directly handle complex data types without flattening; more advanced SQL features like WITH clause, window frame, and subqueries.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter
Similaire à Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter (20)
Plus de Data Con LA
Plus de Data Con LA (20)
Dernier
Dernier (20)
Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter
- 1. What’s New Tajo 0.10 and its Beyond Big Data Day LA 2015 Hyunsik Choi, Gruter Inc. (hschoi @ gruter.com)
- 2. Agenda • Tajo Overview • Milestones and 0.10 Features • What’s Next
- 3. About Me • Hyunsik Choi (pronounced “Hyeon-‐shickCheh”) • PhD (Computer Science & Engineering) • Director of Research, Gruter Inc. • Open-‐source Involvement • Full-‐time contributor to Apache Tajo (2013.6 ~ ) • Apache Tajo PMC member and committer (2013.3 ~ ) • Apache Giraph PMC member and committer (2011. 8 ~ ) • Contact Info • Email: hyunsik@apache.org • Linkedin: http://linkedin.com/in/hyunsikchoi/
- 4. Tajo: A Data Warehouse System • Apache Top-‐level project • Distributed and scalable data warehouse system on Hadoop • Low latency, and long running batch queries in a single system • Features • ANSI SQL compliance • Mature SQL features: Joins, Group by, Order by, Aggregation and Window function • Partitioned table support • Java/Python UDF support • JDBC driver and Java-‐based asynchronous API • Read/Write support of CSV, JSON, RCFile, SequenceFile, Parquet, ORC
- 5. Master
- 7. Query
- 9. UI Slave
- 11. Query
- 14. Query
- 17. a
- 18. query Manage
- 20. a
- 21. query send
- 22. tasks
- 23. monitor
- 24. send
- 25. tasks
- 26. monitor
- 27. Tajo Overall Architecture HDFS HBase HDFS HBase
- 28. Common Scenarios • Extraction, Transformation, Loading (ETL) • Interactive BI/analytics on web-‐scale big data • Data discovery/Exploratory analysis with R and existing SQL tools • Query federation (0.11 release)
- 29. Use Cases: Replacement of Commercial DW • Example: Telco Company (50 million users) • Goal: • Replacement of slow ETL workloads on several TB datasets • Lots daily reports generation about users’ behaviors • Ad-‐hoc analysis on Terabytes data sets • Key Benefits of Tajo: • Simplification of DW ETL, OLAP, and Hadoop ETL into an unified system • Saved license over commercial DW • Much less cost, more data analysis within the same SLA
- 30. Use Cases: Data Discovery • Example: Music streaming service (26 million users) • Goal: • Analysis on purchase history for target marketing • Benefits: • Query interactivity on large data sets • Ability to use existing BI visualization tools
- 31. When Tajo is right choice? • You want an unified system for batch and interactive queries on Hadoop, Amazon S3, or Hbase. • You want to use a mixed use of Hadoop-‐based DW and RDBMS-‐based DW or want to replace existing RDBMS DW. • You want to use existing SQL tools on Hadoop DW
- 32. Milestones • 0.9 – 2014.12 • 0.10 – 2015.03 • 0.11.0 – 2015.07 • Major release • Python UDF • Multi-‐tenancy scheduler Take 1 • Better storage support and Tablespace support • Query federation • In/Exsist Subquery • Nested scheme support • Better broadcast join and join optimization
- 33. Selected Features in 0.10
- 34. Hbase Storage Support • You can use SQL to access Hbase tables. • Tajo supports Hbase storage • CREATE (EXTERNAL)/DROP/INSERT (OVERWRITE) CREATE TABLE hbase_t1 (key TEXT, col1 TEXT, col2 INT) USING hbase WITH ( ‘table’ = ‘t1’, ‘columns’ = ‘:key,cf1:col1,cf2:col2`, ‘hbase.zookeeper.quorum’ = ‘host1:2181,host2:2181’ )
- 35. Better AWS support • Optimized for S3 and EMR environments • Fixed many bugs related to S3 • EMR bootstrap supported in AWS Labs Github repo • A quick guide for Tajo on EMR • http://www.gruter.com/blog/setting-‐up-‐a-‐tajo-‐cluster-‐on-‐amazon-‐emr/ • EMR bootstrap for Tajo on EMR • https://github.com/awslabs/emr-‐bootstrap-‐actions
- 36. Tajo JDBC Tajo Cluster ETL Tools BI Tools Reporting tools Better SQL tool support via thin JDBC HDFS HBase S3 Swift
- 38. Improved Performance and Stability • Offheap sort operator for ORDER BY (TAJO-‐907) • Hash shuffle IO improvement (TAJO-‐374, TAJO-‐987) • Skewness handling of hash shuffle • Automatic parallel degree choice during runtime • Lots of query optimizer improvements • Add Master HA (TAJO-‐704) • More stable and error-‐tolerant fetch (TAJO-‐789, TAJO-‐953)
- 39. What’s New in Tajo 0.11
- 40. Nested data • Nested data is becoming common • JSON, BSON, XML, Protocol Buffer, Avro, Parquet, … • Many web applications in common use JSON. • MongoDB by default uses JSON document • Many Hbase users also store JSON document in a cell. • Flattening causes lots of data/computation overhead. • Tajo 0.11 natively supports nested data format.
- 41. How to create a nested schema table Use ‘RECORD’ keyword to define nested schema. JSON must be line-‐delimited.
- 42. Loose schema in self-‐describing formats You can handle schema evolving with ALTER ADD COLUMN!
- 43. How to retrieve nested fields Input Data Table Definition SQL
- 44. Query federation and Tablespace support • Query support across multiple data sources • You can perform join or union among tables on different systems. • Benefits: • Data offload from RDBMS to Hadoop vice versa • A mixed use of existing RDBMS and Hadoop. • NoSQL and various storages allows • An unified interface for SQL tools HDFS NoSQL S3 Swift Apache Tajo
- 45. Sequence File RCFile Protocol Buffer Data Formats Storage Types Datasets stored in Various Formats/Storages
- 46. Tablespace Concept • Tablespace • Storage spaces identified URI • Configuration and Policy shared in all tables in the same tablespace • Multiple tablespaces are possible in single storage namespace. • HDFS-‐2832: Enable support for heterogeneous in HDFS. • e.g., • /warehouse/ (disk) • /today/ (ssd)
- 48. Create Table on a specified Tablespace CREATE TABLE uptodate (key TEXT, …) TABLESPACE hbase1; CREATE TABLE archive (l_orderkey bigint, …) TABLESPACE warehouse USING text WITH (‘text.delimiter’ = ‘|’); Tablespace Name Format name
- 49. Operation Push Down SELECT X, SUM(Y) FROM table1 WHERE x 100 GROUP BY x Underlying Storage Filter, Projection or Groupbycan be pushed down into Underlying storages (like RDBMS, Hbase, Elasticsearch, …)
- 50. Current Status of Storages • Storages: • HDFS support • Amazon S3 and Openstack Swift • Hbase Scanner and Writer -‐ HFile and Put Mode • JDBC-‐based Scanner and Writer (Working) • Kafka Scanner (Patch Available) • Elastic Search (Patch Available) • Data Formats • Text, JSON, RCFile, SequenceFile, Avro, Parquet, and ORC (Patch Available)
- 51. Python UDF • Python UDF and UDAF are supported in Tajo • http://tajo.apache.org/docs/devel/functions/python.html @output_type('int4') def return_one(): return 1 @output_type('text') def helloworld(): return 'Hello, World’ @output_type('int4') def sum_py(a,b): return a+b
- 52. Get Involved! • We are recruiting contributors! • General • http://tajo.apache.org • Getting Started • http://tajo.apache.org/docs/0.10.0/getting_started.html • Downloads • http://tajo.apache.org/downloads.html • Jira – Issue Tracker • https://issues.apache.org/jira/browse/TAJO • Join the mailing list • dev-‐subscribe@tajo.apache.org • issues-‐subscribe@tajo.apache.org
- 53. QA