Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (19)
Similar to Promoting a Data Driven Culture in a Microservices Environment
Similar to Promoting a Data Driven Culture in a Microservices Environment (20)
More from PyData
More from PyData (20)
Recently uploaded
Recently uploaded (20)
Promoting a Data Driven Culture in a Microservices Environment
- 1. Democratizing Data Promoting a data driven culture in a world of microservices
- 2. Overview 1. Introduction to Hudl 2. Hudl Data Journey 3. #DataProblems 4. Data Engineering 5. Data Analytics 6. Key Takeaways 7. Summary
- 4. Basketball workflow animation or static images.
- 5. Basketball workflow animation or static images.
- 6. Basketball workflow animation or static images.
- 7. Capture and bring value to every moment in sports. 4.9 million users 150 thousand teams 4.5 billion video views last 12 months
- 9. 2006
- 10. 2010
- 11. 2014
- 12. 2014
- 13. 2015
- 14. 2015
- 15. #DataProblems
- 16. “Find all football teams that had 3 or more users watch video in 3 different months.”
- 20. SSH + SQL + Mongo + Excel/Python/etc.
- 22. Data Engineering
- 23. Data Engineering Just give me my damn data.
- 24. Three questions 1. Where do we put the data? 2. How does it get there? 3. How do people access it?
- 25. Three questions 1. Where do we put the data? 2. How does it get there? 3. How do people access it?
- 28. ● SQL ● Fully managed on AWS ● Reasonably priced Amazon Redshift
- 29. ● SQL ● Fully managed on AWS ● Reasonably priced Rob Story, Data Engineering Architecture at Simple, PyData Chicago Amazon Redshift
- 30. For the Google Cloud User: Google BigQuery For the Do-it-yourself-er: Hive / Impala / PrestoDB / Druid For the Enterprise User: Vertica / Teradata ? Alternatives
- 31. Three questions 1. Where do we put the data? 2. How does it get there? 3. How do people access it?
- 34. E T L
- 38. Extract Extract Extract Transform TransformTransform Load LoadLoad
- 39. Extract Extract Extract Transform TransformTransform Load LoadLoad
- 40. Use a workflow manager.
- 41. Luigi (Spotify) Airflow (Airbnb) Azkaban (LinkedIn)
- 42. Luigi (Spotify) Airflow (Airbnb) Azkaban (LinkedIn)
- 43. ● Dependency management ● Parallelism ● Idempotence
- 44. Think about your tooling.
- 48. ● UI ● Logging ● Triggers ○ Cron ○ Dependency ○ GitHub
- 51. Single machine jobs ● Zendesk ● Salesforce ● Google Sheets
- 52. Multi machine jobs ● Database exports ● Mongo processing
- 54. Three questions 1. Where do we put the data? 2. How does it get there? 3. How do people access it?
- 57. ● Everyone has access -- 430+ Hudlies ● Lots of data ○ 24+ TB ○ 100B+ rows Our needs
- 58. ● Looker ● Periscope ● Tableau Commercial options
- 59. ● Open source (Python!) ● Query editor + visualizations ● Hosted version or host your own re:dash
- 61. SSH + SQL + Mongo + Excel/Python/etc.
- 62. SSH + SQL + Mongo + Excel/Python/etc.
- 63. Data Analytics Helping employees use data to make better decisions.
- 64. Access for All
- 65. Finding Data isn’t Easy ● SQL ● So much data ● Only 3 data analysts
- 66. ● Education ● Derived Tables ● Report Automation Removing Roadblocks
- 67. ● Education ● Derived Tables ● Report Automation Removing Roadblocks
- 68. ● Relational Database Model ● Basic & intermediate SQL ● Table Familiarity ● Using re:dash ● Data Visualization Certification Topics
- 70. Data Dictionary screenshot here but delete this box after you place it
- 71. Understanding Relationships screenshot here but delete this box after you place it
- 72. ● Education ● Derived Tables ● Report Automation Removing Roadblocks
- 73. “Find how many football teams had 3 or more users watch video in 3 different months this year.”
- 75. hudl_daily_active_users Userid Teamid Date Has_Watched _Video Has_Tagged_ Video Has_uploaded_ video 1 1234 2016-08-15 True True False 2 2345 2016-08-15 False False True 3 5678 2016-08-15 True False False
- 76. ● Education ● Derived Tables ● Report Automation Removing Roadblocks
- 79. Slackalytics
- 81. September Stats ● 194 unique users executed a query ● 14,000 ad hoc queries executed ● 940 unique scheduled queries/week
- 82. ● Bad Data ● Slow Queries Cons
- 83. ● Being Data-driven is a team sport ● Get the data architecture in place ● Make data and metrics accessible ● Be Flexible Key Takeaways
- 84. Summary 1. Introduction to Hudl 2. Hudl Data Journey 3. #DataProblems 4. Data Engineering 5. Data Analytics 6. Key Takeaways 7. Summary
- 85. Tools we use Summary Jenkins Scheduling Luigi Workflow management Sqoop RDBMS Extraction Spark Data transformation AWS Lambda Event-driven processing Redshift Data warehouse re:dash Query interface + visualization