Sean Kandel - Data profiling: Assessing the overall content and quality of a data set
•
3 j'aime•1,233 vues
The task of “data profiling”—assessing the overall content and quality of a data set—is a core aspect of the analytic experience. Traditionally, profiling was a fairly cut-and-dried task: load the raw numbers into a stat package, run some basic descriptive statistics, and report the output in a summary file or perhaps a simple data visualization. However, data volumes can be so large today that traditional tools and methods for computing descriptive statistics become intractable; even with scalable infrastructure like Hadoop, aggressive optimization and statistical approximation techniques must be used. In this talk Sean will cover technical challenges in keeping data profiling agile in the Big Data era. He will discuss both research results and real-world best practices used by analysts in the field, including methods for sampling, summarizing and sketching data, and the pros and cons of using these various approaches. Sean is Trifacta’s Chief Technical Officer. He completed his Ph.D. at Stanford University, where his research focused on user interfaces for database systems. At Stanford, Sean led development of new tools for data transformation and discovery, such as Data Wrangler. He previously worked as a data analyst at Citadel Investment Group.
Recommandé
Contenu connexe
Similaire à Sean Kandel - Data profiling: Assessing the overall content and quality of a data set
Similaire à Sean Kandel - Data profiling: Assessing the overall content and quality of a data set (20)
Plus de huguk
Plus de huguk (20)
Dernier
Dernier (20)
Sean Kandel - Data profiling: Assessing the overall content and quality of a data set
- 1. Agile Data Profiling Sean Kandel
- 2. What’s in your data?
- 3. Opening Questions in the Data Lifecycle… Unboxing What’s in this data? Can I make use of it?
- 4. … Become Persistent Questions in the Data Lifecycle What’s in this data? Can I make use of it? Unboxing Transformation Analysis Visualization Productization
- 5. Unboxing Transformation Analysis Visualization Productization
- 6. Unboxing Transformation Analysis Visualization Productization STRUCTURING CLEANING ENRICHMENT DISTILLATION
- 7. “Its easy to just think you know what you are doing and not look at data at every intermediary step. An analysis has 30 different steps. Its tempting to just do this then that and then this. You have no idea in which ways you are wrong and what data is wrong.”
- 8. What’s in the data? • The Expected: Models, Densities, Constraints • The Unexpected: Residuals, Outlier, Anomalies
- 10. Expected
- 11. Unexpected
- 15. Overview of all variables
- 17. What to compute? • Densities and descriptive statistics • Identify anomalies and outliers
- 18. How often to compute it? Unboxing Transformation Analysis Visualization Productization
- 19. Challenge: Agility • Profiling throughout the lifecycle • Particularly important as you manipulate data
- 20. Design Space and Tradeoffs
- 21. Mapping out the Design Space How much data to examine? How accurate are the results? How fast can you get them?
- 22. Mapping out the Design Space Decide how your requirements fall on these axes Find a strategy (if one exists) that fits the requirements Accuracy Urgency Data Volume
- 23. Accuracy Urgency Data Volume Strategy vs Cost Head of file Good EnoughAnomaliesBig PictureUnbox
- 24. Strategy vs Cost Random Sample Accuracy Urgency Data Volume Good EnoughAnomaliesBig PictureUnbox
- 25. Strategy vs Cost Scan, summarize, collect samples Accuracy Urgency Data Volume Good EnoughAnomaliesBig PictureUnbox
- 26. Far better an approximate answer to the right question, which is often vague, than the exact answer to the wrong question, which can always be made precise. Data Analysis & Statistics, Tukey & Wilk 1966
- 28. Sanity Check: Is this really expensive? • Computers are fast • In-memory, column stores, OLAP, … • Still, “Big Data” can be hard • Big is sometimes really big • Big data can be raw: no indexes or precomputed summaries • Agility remains critical to harness the “informed human mind”
- 29. Two Useful Techniques Sampling • A variety of techniques available Sketches • One-pass memory-efficient structures for capturing distributions Accuracy Urgency Data Volume
- 31. Approaches to Sampling • Scan-based access • Head-of-file • Bernoulli • Reservoir • Random I/O Sampling • Block-level sampling
- 32. Head-of-File • Pros: • Very fast: small data, no disk seeks • Absolutely required when unboxing raw data • Nested data (JSON/XML), Text (logs, database dumps, etc.) • Cons: • Correlation of position and value
- 33. Bernoulli • Take a full pass, flip a (weighted) coin for each record • Pros: • trivial to implement • trivial to parallelize • almost no memory required • Cons: • requires a full scan of the data • output size proportional to input size, and random filter(lambda x : random() < 0.01, data)
- 34. Reservoir • Fix “reservoir”. For each item, with probability eject old for new • Pros: • trivial to implement • easy to parallelize • constant memory required • fixed-size output — need not know input size in advance • Cons: • Requires a full scan of the data … 61141217 139 res = data [0:k] //initialize: first k items counter = k for x in data [k:]: if random () < k/float(counter+1): res[randint(0,len(res)-1)] = x counter += 1
- 35. 1141217 Reservoir … 6 133 • Fix “reservoir”. For each item, with probability eject old for new • Pros: • trivial to implement • easy to parallelize • constant memory required • fixed-size output — need not know input size in advance • Cons: • Requires a full scan of the data res = data [0:k] //initialize: first k items counter = k for x in data [k:]: if random () < k/float(counter+1): res[randint(0,len(res)-1)] = x counter += 1
- 36. 41217 Reservoir … 6 137 3 • Fix “reservoir”. For each item, with probability eject old for new • Pros: • trivial to implement • easy to parallelize • constant memory required • fixed-size output — need not know input size in advance • Cons: • Requires a full scan of the data res = data [0:k] //initialize: first k items counter = k for x in data [k:]: if random () < k/float(counter+1): res[randint(0,len(res)-1)] = x counter += 1
- 37. Meta-Strategy: Stratified Sampling • Sometimes you need representative samples from each “group” • Coverage: e.g., displaying examples for every state in a map • Robustness: e.g., consider average income • if you miss the rare top tax bracket, estimate is way off
- 38. Stratification: the GroupBy / Agg pattern • Given: • A group-partitioning key for stratification • Sizes for each stratum • Easy to implement: partition, and construct sample per partition • your favorite sampling technique applies SELECT D.group_key, reservoir(D.value) FROM data D GROUP BY D.group_key;
- 39. Record Sampling • Randomly sample records? • r the % items sampled; p #rows/block • 20x random I/O penalty => read fewer than 5% of blocks!
- 40. Record Sampling • Randomly sample records? • r the % items sampled; p #rows/block • 20x random I/O penalty => read fewer than 5% of blocks! • Pretty inefficient: touches 1-(1-r)p blocks
- 41. Record Sampling % items sampled %blockstouched(expected) 1-(1-r)p with p = 100
- 42. Block Sampling • Randomly sample blocks of records from disk • Concern: clustering bias. • Techniques from database literature: assess bias and correct • Beware: even block sampling needs to be well below 5%.
- 43. Sampling in Hadoop • Larger unit of access: HDFS blocks (128MB vs. 64KB) • HDFS buffering makes forward seeking within block cheaper • But CPU costs may encourage sampling within the block. • …and Hadoop makes it easy to sample across nodes • Each worker only processes one block • Must find record boundaries • Tougher when dealing with quote escaping
- 45. Sketching • Family of algorithms for estimating contents of a data stream • Constant-sized memory footprint • Computed in 1 pass over the data • Classic Examples • Bloom filter: existence testing • HyperLogLog Sketches (FM): distinct values • CountMin (CM): a surprisingly versatile sketch for frequencies
- 46. CountMin Sketch: Initialization 0 dhashfunctions w hash buckets Count-Min Sketch 0 0 0 0 0 0 0 0 0 0 0 0 0 0
- 47. CountMin Sketch: Insertion dhashfunctions w hash buckets Count-Min Sketch Insert(7) h1 h2 hw
- 48. CountMin Sketch: Insertion dhashfunctions w hash buckets Count-Min Sketch 1 h1(7) h2(7) hw(7) 1 1
- 49. CountMin Sketch: Insertion dhashfunctions w hash buckets Count-Min Sketch Insert(4) h1 h2 hw
- 50. CountMin Sketch: Insertion dhashfunctions w hash buckets Count-Min Sketch 1 h1(4) h2(4) hw(4) 2 1
- 51. CountMin Sketch: Insertion dhashfunctions w hash buckets Count-Min Sketch 1 h1(4) h2(4) hw(4) 2 1
- 52. CountMin Sketch: Query dhashfunctions w hash buckets Count-Min Sketch Count(7)? h1 h2 hw
- 53. CountMin Sketch: Query dhashfunctions w hash buckets Count-Min Sketch 1 h1(7) h2(7) hw(7) 2 1 Count(7)?
- 54. CountMin Sketch: Query dhashfunctions w hash buckets Count-Min Sketch 1 h1(7) h2(7) hw(7) 2 1 min Count(7)
- 55. CountMin Sketch: Theorem & Tuning — Cormode/Muthukrishnan, J Algorithm 55(1) (2005). dhashfunctions w hash buckets Count-Min Sketch
- 56. CountMin Sketch: Theorem & Tuning — Cormode/Muthukrishnan, J Algorithm 55(1) (2005). dhashfunctions w hash buckets Count-Min Sketch an over-estimate
- 57. CountMin Sketch: Theorem & Tuning — Cormode/Muthukrishnan, J Algorithm 55(1) (2005). dhashfunctions w hash buckets Count-Min Sketch w controls expected error amount d controls probability of error Suppose we want: 0.1% error, 99.9% probability. w = 2000 d = 10
- 58. CountMeanMin Sketch dhashfunctions w hash buckets Count-Mean-Min Sketch Idea: subtract out expected overage. i.e. mean of other cells
- 59. CountMeanMin Sketch dhashfunctions w hash buckets Count-Mean-Min Sketch mean —
- 60. CountMeanMin Sketch dhashfunctions w hash buckets Count-Mean-Min Sketch mean — mean — median
- 61. CountMeanMin Sketch dhashfunctions w hash buckets Count-Mean-Min Sketch mean — mean — mean — median Count(7)
- 62. CountMin (and CountMeanMin) answer “point frequency queries”. Surprisingly, we can use them to answer many more questions • densities • even order statistics (median, quantiles, etc.) The Versatile CountMin Sketch
- 63. More Statistics • Count-Range Queries • Median • Quantiles • Histograms
- 64. 0001020304050607080910111213141516171819202122232425262728293031 Count(x=13) CountMin: Point Queries
- 65. 0001020304050607080910111213141516171819202122232425262728293031 Count(x ∊ [14-15]) CountMin(⌊x/2⌋): Pair Queries
- 66. 0001020304050607080910111213141516171819202122232425262728293031 Count(x ∊ [16-19]) CountMin(⌊x/4⌋): Quartet Queries
- 67. 0001020304050607080910111213141516171819202122232425262728293031 Maintain all of these, and answer arbitrary range queries. Count(x ∊ [13-24]) Dyadic CountMin: log2 CountMins x x/2 x/4 x/8 x/16
- 68. 0001020304050607080910111213141516171819202122232425262728293031 Maintain all of these, and answer arbitrary range queries. Count(x ∊ [13-24]) Dyadic CountMin: log2 CountMins x x/2 x/4 x/8 x/16
- 69. More Statistics • Count-Range Queries • Median • Quantiles • Histograms
- 70. 0001020304050607080910111213141516171819202122232425262728293031 Median Via binary search. (Suppose we have N elements, and the real median is 14)
- 71. 0001020304050607080910111213141516171819202122232425262728293031 Median Via binary search. (Suppose we have N elements, and the real median is 14)
- 72. 0001020304050607080910111213141516171819202122232425262728293031 Median Via binary search. (Suppose we have N elements, and the real median is 14)
- 73. 0001020304050607080910111213141516171819202122232425262728293031 Median Via binary search. (Suppose we have N elements, and the real median is 14)
- 74. 0001020304050607080910111213141516171819202122232425262728293031 Median Via binary search. (Suppose we have N elements, and the real median is 14)
- 75. More Statistics • Count-Range Queries • Median • Quantiles: generalization of Median • Histograms 0001020304050607080910111213141516171819202122232425262728293031
- 76. More Statistics • Count-Range Queries • Median • Quantiles • Histograms: • fixed-width bins: range queries • fixed-height bins: quantiles 1-10 11-20 21-30 31-40
- 78. Wrangling Revisited Good EnoughAnomaliesBig PictureUnbox
- 79. Wrangling Revisited Good EnoughAnomaliesBig PictureUnbox Head-of-file
- 80. Wrangling Revisited Good EnoughAnomaliesBig PictureUnbox Head-of-file Bernoulli Block Reservoir
- 81. Wrangling Revisited Good EnoughAnomaliesBig PictureUnbox Head-of-file Bernoulli Sketching Stratified Block Reservoir
- 82. Summary • ABP: Always Be Profiling • Tradeoff latency and accuracy • Approximation methods • Heuristics and reasonable assumptions
- 83. Acknowledgments Adam Silberstein, Joe Hellerstein