Figures explain HDFS Erasure Coding

•

11 j'aime•4,774 vues

Erasure coding in HDFS provides redundancy for data blocks while using less storage space compared to simple replication. It works by splitting files into data blocks and parity blocks striped across multiple data nodes. When reading data, erasure coding reconstructs missing or corrupted blocks from the parity blocks to maintain data reliability even if some blocks are lost. Erasure coding reduces storage overhead compared to replication and improves read performance by allowing reconstruction of missing blocks in parallel from different nodes.

Données & analyses

図でわかる 
HDFS Erasure Coding
Kai Sasaki
Treasure Data Inc.

Who am I
佐々木海(Kai Sasaki)
Software Engineer at Treasure Data Inc. 
http://www.treasuredata.com
Hadoop, Spark, DL4J

Agenda
• Erasure Coding
• Under the Namespace
• Writing Side
• Reading Side

Replication
Block
Block
Block
Block
HDFS

Replication
Block
Block
Block
Block
HDFS
Capacity
Overhead  
x3

Replication
Block
Block
Block
Block
HDFS
Redundancy
2
Capacity
Overhead  
x3

Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block
RS-6-3

Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block6 out of 9

Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
BlockRedundancy
3

Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block
Capacity
Overhead  
x1.5
Redundancy
3

Erasure Coding
Block
Block
HDFS
Block
Block
Block
Block
Block
Block
Block
Block
BlockGroup

INode and BlockInfo
BlockInfo
INode
BlockInfo BlockInfo…

INode and BlockInfo
BlockInfo
INode
BlockInfo BlockInfo…
…
Block
Block
Block

INode and BlockInfo
BlockInfo
INode
BlockInfo BlockInfo…
…
Block
Block
Block
BlockGroup

BlockInfo
…
Block
Block
Block
BlockGroup

long BlockId
0 64
BlockInfo
…
Block
Block
Block
BlockGroup

long BlockId
0 64
BlockInfo
…
Block
Block
Block
BlockGroup
index GroupId
4bit 60bit

long BlockId
0 64
BlockInfo
…
Block
Block
Block
BlockGroup
index GroupId
4bit 60bit
index 0
index 2
index 1
Saving memory

Data
BlockGroup
0
Data Block Parity Block

Data
BlockGroup
0
Data Block Parity Block
Stripe

Data
BlockGroup
0
0
Saving diskspace usage

BlockGroup
0
0
200kb 500kb
Saving reading time

まとめ
• Namespace -> Saving memory 
BlockInfoStriped, BlockIdManager
• Writing Side -> Saving diskspace usage 
INodeFile
• Reading Side -> Saving reading time 
DFSStripedInputStream

Recommandé

Spark meetup london share and analyse genomic data at scale with spark, adam...Andy Petrella

20170927 pydata tokyo データサイエンスな皆様に送る分散処理の基礎の基礎、そしてPySparkの勘所Ryuji Tamagawa

PySparkの勘所（20170630 sapporo db analytics showcase） Ryuji Tamagawa

Retrieving big data for the non developerGustaf Cavanaugh

Fast Variant Calling with ADAM and avocadofnothaft

Hadoop training by keylabsSiva Sankar

20171012 found IT #9 PySparkの勘所Ryuji Tamagawa

ADAM—Spark Summit, 2014fnothaft

Recommandé

Spark meetup london share and analyse genomic data at scale with spark, adam...Andy Petrella

20170927 pydata tokyo データサイエンスな皆様に送る分散処理の基礎の基礎、そしてPySparkの勘所Ryuji Tamagawa

PySparkの勘所（20170630 sapporo db analytics showcase） Ryuji Tamagawa

Retrieving big data for the non developerGustaf Cavanaugh

Fast Variant Calling with ADAM and avocadofnothaft

Hadoop training by keylabsSiva Sankar

20171012 found IT #9 PySparkの勘所Ryuji Tamagawa

ADAM—Spark Summit, 2014fnothaft

Erasure codes and storage tiers on glusterRed_Hat_Storage

Less is More: 2X Storage Efficiency with HDFS Erasure CodingZhe Zhang

What's new in hadoop 3.0Heiko Loewe

HDFS Erasure Code Storage - Same Reliability at Better Storage EfficiencyDataWorks Summit

HDFS Erasure Coding in Action DataWorks Summit/Hadoop Summit

Erasure Coding Costs and BenefitsJohn Cook

Native erasure coding support inside hdfs presentationlin bao

Data Science Crash Course Hadoop Summit SJDaniel Madrigal

HDFS Deep DiveYifeng Jiang

Timeline Service v.2 (Hadoop Summit 2016)Sangjin Lee

Apache Hadoop Crash Course - HS16SJDataWorks Summit/Hadoop Summit

トランザクションの並行処理制御Takashi Hoshino

Apache Hadoop Crash CourseDataWorks Summit/Hadoop Summit

Hadoop crashcourse v3Hortonworks

Performance comparison of Distributed File Systems on 1Gbit networksMarian Marinov

Intro to Spark with Zeppelin Crash Course Hadoop Summit SJDaniel Madrigal

Hadoop Summit Tokyo Apache NiFi Crash CourseDataWorks Summit/Hadoop Summit

WebSocketのキホンYou_Kinjoh

#HSTokyo16 Apache Spark Crash Course DataWorks Summit/Hadoop Summit

Hadoop crash course workshop at Hadoop SummitDataWorks Summit

Design for a Distributed Name NodeAaron Cordova

Etu L2 Training - Hadoop 企業應用實作James Chen

Contenu connexe

En vedette

Erasure codes and storage tiers on glusterRed_Hat_Storage

Less is More: 2X Storage Efficiency with HDFS Erasure CodingZhe Zhang

What's new in hadoop 3.0Heiko Loewe

HDFS Erasure Code Storage - Same Reliability at Better Storage EfficiencyDataWorks Summit

HDFS Erasure Coding in Action DataWorks Summit/Hadoop Summit

Erasure Coding Costs and BenefitsJohn Cook

Native erasure coding support inside hdfs presentationlin bao

Data Science Crash Course Hadoop Summit SJDaniel Madrigal

HDFS Deep DiveYifeng Jiang

Timeline Service v.2 (Hadoop Summit 2016)Sangjin Lee

Apache Hadoop Crash Course - HS16SJDataWorks Summit/Hadoop Summit

トランザクションの並行処理制御Takashi Hoshino

Apache Hadoop Crash CourseDataWorks Summit/Hadoop Summit

Hadoop crashcourse v3Hortonworks

Performance comparison of Distributed File Systems on 1Gbit networksMarian Marinov

Intro to Spark with Zeppelin Crash Course Hadoop Summit SJDaniel Madrigal

Hadoop Summit Tokyo Apache NiFi Crash CourseDataWorks Summit/Hadoop Summit

WebSocketのキホンYou_Kinjoh

#HSTokyo16 Apache Spark Crash Course DataWorks Summit/Hadoop Summit

Hadoop crash course workshop at Hadoop SummitDataWorks Summit

En vedette (20)

Erasure codes and storage tiers on gluster

Less is More: 2X Storage Efficiency with HDFS Erasure Coding

What's new in hadoop 3.0

HDFS Erasure Code Storage - Same Reliability at Better Storage Efficiency

HDFS Erasure Coding in Action

Erasure Coding Costs and Benefits

Native erasure coding support inside hdfs presentation

Data Science Crash Course Hadoop Summit SJ

HDFS Deep Dive

Timeline Service v.2 (Hadoop Summit 2016)

Apache Hadoop Crash Course - HS16SJ

トランザクションの並行処理制御

Apache Hadoop Crash Course

Hadoop crashcourse v3

Performance comparison of Distributed File Systems on 1Gbit networks

Intro to Spark with Zeppelin Crash Course Hadoop Summit SJ

Hadoop Summit Tokyo Apache NiFi Crash Course

WebSocketのキホン

#HSTokyo16 Apache Spark Crash Course

Hadoop crash course workshop at Hadoop Summit

Similaire à Figures explain HDFS Erasure Coding

Design for a Distributed Name NodeAaron Cordova

Etu L2 Training - Hadoop 企業應用實作James Chen

Hadoop 1.x vs 2Rommel Garcia

HDFS introductioninjae yeo

Introduction to Machine Learning on Apache Spark MLlib by Juliet Hougland, Se...Cloudera, Inc.

Spark cassandra integration, theory and practiceDuyhai Doan

Breaking the 1 Million OPS/SEC Barrier in HOPS HadoopDataWorks Summit/Hadoop Summit

Tendências e Evoluções em Armazemamento de Dados Jefferson Alcantara

Ceph InternalsVictor Santos

Hadoop Interview Questions and AnswersMindsMapped Consulting

Hadoop Interview Questions and Answers | Big Data Interview Questions | Hadoo...Edureka!

Similaire à Figures explain HDFS Erasure Coding (11)

Design for a Distributed Name Node

Etu L2 Training - Hadoop 企業應用實作

Hadoop 1.x vs 2

HDFS introduction

Introduction to Machine Learning on Apache Spark MLlib by Juliet Hougland, Se...

Spark cassandra integration, theory and practice

Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop

Tendências e Evoluções em Armazemamento de Dados

Ceph Internals

Hadoop Interview Questions and Answers

Hadoop Interview Questions and Answers | Big Data Interview Questions | Hadoo...

Plus de Kai Sasaki

Graviton 2で実現する コスト効率のよいCDP基盤Kai Sasaki

Infrastructure for auto scaling distributed systemKai Sasaki

Continuous Optimization for Distributed BigData AnalysisKai Sasaki

Recent Changes and Challenges for Future PrestoKai Sasaki

Real World Storage in Treasure DataKai Sasaki

20180522 infra autoscaling_systemKai Sasaki

User Defined Partitioning on PlazmaDBKai Sasaki

Deep dive into deeplearn.jsKai Sasaki

Optimizing Presto Connector on Cloud StorageKai Sasaki

Presto updates to 0.178Kai Sasaki

How to ensure Presto scalability  in multi use case Kai Sasaki

Managing multi tenant resource toward Hive 2.0Kai Sasaki

Embulk makes Japan visibleKai Sasaki

Maintainable cloud architecture_of_hadoopKai Sasaki

Spark MLlib code reading ~optimization~Kai Sasaki

How I tried MADEKai Sasaki

Reading kernel orgKai Sasaki

Reading drillKai Sasaki

Kernel ext4Kai Sasaki

Kernel bootstrapKai Sasaki

Plus de Kai Sasaki (20)

Graviton 2で実現する コスト効率のよいCDP基盤

Infrastructure for auto scaling distributed system

Continuous Optimization for Distributed BigData Analysis

Recent Changes and Challenges for Future Presto

Real World Storage in Treasure Data

20180522 infra autoscaling_system

User Defined Partitioning on PlazmaDB

Deep dive into deeplearn.js

Optimizing Presto Connector on Cloud Storage

Presto updates to 0.178

How to ensure Presto scalability  in multi use case

Managing multi tenant resource toward Hive 2.0

Embulk makes Japan visible

Maintainable cloud architecture_of_hadoop

Spark MLlib code reading ~optimization~

How I tried MADE

Reading kernel org

Reading drill

Kernel ext4

Kernel bootstrap

Dernier

Decoding Patterns: Customer Churn Prediction Data Analysis ProjectBoston Institute of Analytics

INTRODUCTION TO Natural language processingsocarem879

Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...Boston Institute of Analytics

wepik-insightful-infographics-a-data-visualization-overview-20240401133220kwr...KarteekMane1

Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...Boston Institute of Analytics

Semantic Shed - Squashing and Squeezing.pptxMike Bennett

Advanced Machine Learning for Business ProfessionalsVICTOR MAESTRE RAMIREZ

6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...Dr Arash Najmaei ( Phd., MBA, BSc)

Networking Case Study prepared by teacher.pptxHimangsuNath

Data Analysis Project: Stroke PredictionBoston Institute of Analytics

Cyber awareness ppt on the recorded dataTecnoIncentive

SMOTE and K-Fold Cross Validation-Presentation.pptxHaritikaChhatwal1

Student profile product demonstration on grades, ability, well-being and mind...Seán Kennedy

modul pembelajaran robotic Workshop _ by Slidesgo.pptxaleedritatuxx

Student Profile Sample report on improving academic performance by uniting gr...Seán Kennedy

Bank Loan Approval Analysis: A Comprehensive Data Analysis ProjectBoston Institute of Analytics

Defining Constituents, Data Vizzes and Telling a Data StoryJeremy Anderson

Digital Marketing Plan, how digital marketing worksdeepakthakur548787

The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptxTasha Penwell

Learn How Data Science Changes Our WorldEduminds Learning

Dernier (20)

Decoding Patterns: Customer Churn Prediction Data Analysis Project

INTRODUCTION TO Natural language processing

Decoding the Heart: Student Presentation on Heart Attack Prediction with Data...

wepik-insightful-infographics-a-data-visualization-overview-20240401133220kwr...

Data Analysis Project Presentation: Unveiling Your Ideal Customer, Bank Custo...

Semantic Shed - Squashing and Squeezing.pptx

Advanced Machine Learning for Business Professionals

6 Tips for Interpretable Topic Models _ by Nicha Ruchirawat _ Towards Data Sc...

Networking Case Study prepared by teacher.pptx

Data Analysis Project: Stroke Prediction

Cyber awareness ppt on the recorded data

SMOTE and K-Fold Cross Validation-Presentation.pptx

Student profile product demonstration on grades, ability, well-being and mind...

modul pembelajaran robotic Workshop _ by Slidesgo.pptx

Student Profile Sample report on improving academic performance by uniting gr...

Bank Loan Approval Analysis: A Comprehensive Data Analysis Project

Defining Constituents, Data Vizzes and Telling a Data Story

Digital Marketing Plan, how digital marketing works

The Power of Data-Driven Storytelling_ Unveiling the Layers of Insight.pptx

Learn How Data Science Changes Our World

Figures explain HDFS Erasure Coding

1. 図でわかる  HDFS Erasure Coding Kai Sasaki Treasure Data Inc.

2. Who am I 佐々木海(Kai Sasaki) Software Engineer at Treasure Data Inc.  http://www.treasuredata.com Hadoop, Spark, DL4J

3. Agenda • Erasure Coding • Under the Namespace • Writing Side • Reading Side

4. Erasure Coding

5. Replication Block HDFS

6. Replication Block Block Block Block HDFS

7. Replication Block Block Block Block HDFS

8. Replication Block Block Block Block HDFS Capacity Overhead   x3

9. Replication Block Block Block Block HDFS Redundancy 2 Capacity Overhead   x3

10. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block RS-6-3

11. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9

12. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9

13. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9

14. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9

15. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block BlockRedundancy 3

16. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block Capacity Overhead   x1.5 Redundancy 3

17. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block BlockGroup

18. Under the Namespace

19. INode and BlockInfo BlockInfo INode

20. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo…

21. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block

22. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block BlockGroup

23. BlockInfo … Block Block Block BlockGroup

24. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup

25. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit

26. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit index 0 index 2 index 1 Saving memory

27. Writing Side

28. Data 0

29. Data64KB 0

30. Data BlockGroup 64KB 0

31. Data BlockGroup 0

32. Data BlockGroup 0

33. Data BlockGroup 0

34. Data BlockGroup 0

35. Data BlockGroup 0

36. Data BlockGroup 0

37. Data BlockGroup 0

38. Data BlockGroup 0 Data Block

39. Data BlockGroup 0 Data Block Parity Block

40. Data BlockGroup 0 Data Block Parity Block Stripe

41. Data BlockGroup 0

42. Data BlockGroup 0

43. Data BlockGroup 0

44. Data BlockGroup 0

45. Data BlockGroup 0

46. Data BlockGroup 0

47. Data BlockGroup 0 0

48. Data BlockGroup 0 0

49. Data BlockGroup 0 0 Saving diskspace usage

50. Reading Side

51. BlockGroup 0 0

52. BlockGroup 0 0 200kb 500kb

53. BlockGroup 0 0 200kb 500kb

54. BlockGroup 0 0 200kb 500kb

55. BlockGroup 0 0 200kb 500kb

56. BlockGroup 0 0 200kb 500kb

57. BlockGroup 0 0 200kb 500kb

58. BlockGroup 0 0 200kb 500kb

59. BlockGroup 0 0 200kb 500kb Saving reading time

60. まとめ • Namespace -> Saving memory  BlockInfoStriped, BlockIdManager • Writing Side -> Saving diskspace usage  INodeFile • Reading Side -> Saving reading time  DFSStripedInputStream

61. ありがとうございました