Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2018 MapR Technologies 1
DataOps: An Agile Method for
Data-driven Organizations
Ellen Friedman, PhD
Principal Technologi...
© 2018 MapR Technologies 2
Contact Information
Ellen Friedman, PhD
Principal Technologist, MapR Technologies
Committer Apa...
© 2018 MapR Technologies 3
Big Data Applications Used Widely in Production
Quoted from:
New Vantage Partner Big Data Execu...
© 2018 MapR Technologies 4
How do you measure Earth’s oceans?
By NASA Goddard Space Flight Center from Greenbelt, MD, USA ...
© 2018 MapR Technologies 5
© 2018 MapR Technologies 6
Changing How People Work with Data


A 19th century big data story
Matthew Fountain Maury
extra...
© 2018 MapR Technologies 7
Big data project: Maury’s Wind and Currents charts
© 2018 MapR Technologies 8
Big data project: Maury’s Wind and Currents charts
At first, nobody was
interested in them…
© 2018 MapR Technologies 9
Big data project: Maury’s Wind and Currents charts
At first, nobody was
interested in them…
…un...
© 2018 MapR Technologies 10
© 2014 Ellen Friedman
People with “vision” think with their eyes closed
© 2018 MapR Technologies 11
Aadhaar Project: Largest Biometric DB in the World
•  Unique 12 – digit number for each person...
© 2018 MapR Technologies 12
Changing Rhythm to How We Work with Data
Utility providers using
smart meters
Collect data eve...
© 2018 MapR Technologies 13
Image © E Friedman
Self-driving cars:
Huge volume of
sensor data
Time value of data
Analysis a...
© 2018 MapR Technologies 14
Changing Rhythm to How We Work with Data
Apache Drill SQL query engine with schema discovery f...
© 2018 MapR Technologies 15
We	
  need	
  a	
  better	
  fit	
  to	
  the	
  
way	
  business	
  happens	
  
© 2018 MapR Technologies 16
A Better Fit
•  The way business happens
•  A dataflow (datafabric) that matches the shape of ...
© 2018 MapR Technologies 17
Build a Global Data Fabric
Flexibility & agility to respond as life changes
© 2018 MapR Technologies 18
Global Data Fabric
•  Comprehensive view of data
•  Breaks silos
•  Works with multi-tenancy
•...
© 2018 MapR Technologies 19
A	
  DataOps	
  approach	
  improves	
  
a	
  project’s	
  ability	
  to	
  stay	
  
on	
  tar...
© 2018 MapR Technologies 20
DataOps: Brings Flexibility & Focus
Platform&network
Operations
Softwareengineering
Architectu...
© 2018 MapR Technologies 21
DataOps Principles
“DataOps teams seek to orchestrate data, tools, code and environments from
...
© 2018 MapR Technologies 22
Advantages of a DataOps Approach
•  Able to pivot & respond to real-world events as they happe...
© 2018 MapR Technologies 23
How	
  do	
  you	
  keep	
  people	
  from	
  
feeling	
  threatened	
  by	
  
change?	
  
© 2018 MapR Technologies 24
Don’t	
  be	
  threatening!	
  
© 2018 MapR Technologies 25
Why Stream?
Munich surfing wave Image © 2017 Ellen Friedman
© 2018 MapR Technologies 26
Stream	
  transport	
  supports	
  
microservices	
  	
  	
  
© 2018 MapR Technologies 27
Stream Transport that Decouples Producers & Consumers
P
P
P
C
C
C
Transport Processing
Kafka /...
© 2018 MapR Technologies 28
More on Streaming Microservices
•  Chapter 3 of Streaming Architecture by Ted Dunning & Ellen ...
© 2018 MapR Technologies 29
Legacy Applications
How Does MapR Solve This?
Big Data 1.0 Applications Next-Gen Applications
...
© 2018 MapR Technologies 30
With MapR, Geo-Distributed Data Appears Local
stream
Data
source
Consumer
© 2018 MapR Technologies 31
With MapR, Geo-Distributed Data Appears Local
stream
stream
Data
source
Consumer
© 2018 MapR Technologies 32
With MapR, Geo-distributed Data Appears Local
stream
stream
Data
source
ConsumerGlobal Data Ce...
© 2018 MapR Technologies 33
90% of the effort in successful
machine learning isn’t the
algorithm or the model…
It’s the lo...
© 2018 MapR Technologies 34
Why?
•  Just getting the training data is hard
•  ! The myth of the unitary model
•  Model-to-...
© 2018 MapR Technologies 35
Metrics
Metrics
ResultsRendezvous
Enter Rendezvous Architecture
Scores
ArchiveDecoy
m1
m2
m3
F...
© 2018 MapR Technologies 36
Best thing about Rendezvous: Agile deployment
•  Many “good” models ready and waiting
–  Alrea...
© 2018 MapR Technologies 37
Rendezvous to the Rescue: Better ML Logistics
•  Stream-1st architecture is a powerful approac...
© 2018 MapR Technologies 38
Preparation of Input (and Training) Data
Model 1
Model 2
Model 3
request
Raw
Add
external
data...
© 2018 MapR Technologies 39
Raw data is gold!
© 2018 MapR Technologies 40
Decoy Model in the Rendezvous Architecture
Input
Scores
Decoy
Model 2
Model 3
Archive
•  Looks...
© 2018 MapR Technologies 41
Why do you need new models?
Conditions may (will) change
© 2018 MapR Technologies 42
Advantages of Rendezvous Architecture
Real
model
Result
Canary
Decoy
Archive
Input
© 2018 MapR Technologies 43
Rendezvous: Mainly for Decisioning Type Systems
•  Decisioning style machine learning
–  Looki...
© 2018 MapR Technologies 44
Described in new book on ML management:
Download free pdf via @MapR:
https://mapr.com/ebook/ma...
© 2018 MapR Technologies 45
Example: Tensor Chicken
Label
training
data
Run the
model
Deploy
model
Gather
training
data
La...
© 2018 MapR Technologies 46
DataOps: A Good Way to Adapt to Emerging Data Practices
•  Faster time-to-value & better abili...
© 2018 MapR Technologies 47
Please support women in tech – help build
girls’ dreams of what they can accomplish
© Ellen Fr...
© 2018 MapR Technologies 48
Related events at Strata
•  “Better Machine Learning Logistics with Rendezvous
Architecture” t...
© 2018 MapR Technologies 49
Thank You !
© 2018 MapR Technologies 50
Additional Resources: Available Now
O’Reilly report by Ted Dunning & Ellen Friedman © March 20...
© 2018 MapR Technologies 51
Book signings at MapR booth
•  Wed afternoon break 3:35 pm – 4:15 pm
•  Thur morning break 10:...
© 2018 MapR Technologies 52
Please tell me how DataOps works out for you.
Ellen Friedman
Twitter @Ellen_Friedman
Email @ef...
Upcoming SlideShare
Loading in …5
×

6

Share

Download to read offline

DataOps: An Agile Method for Data-Driven Organizations

Download to read offline

DataOps expands DevOps philosophy to include data-heavy roles (data engineering & data science). DataOps uses better cross-functional collaboration for flexibility, fast time to value and an agile workflow for data-intensive applications including machine learning pipelines. (Strata Data San Jose March 2018)

DataOps: An Agile Method for Data-Driven Organizations

  1. 1. © 2018 MapR Technologies 1 DataOps: An Agile Method for Data-driven Organizations Ellen Friedman, PhD Principal Technologist 7 March 2018 #StrataData
  2. 2. © 2018 MapR Technologies 2 Contact Information Ellen Friedman, PhD Principal Technologist, MapR Technologies Committer Apache Drill & Apache Mahout projects O’Reilly author Email efriedman@mapr.com ellenf@apache.org Twitter @Ellen_Friedman #StrataData
  3. 3. © 2018 MapR Technologies 3 Big Data Applications Used Widely in Production Quoted from: New Vantage Partner Big Data Executive Surveys for 2016 & 2017 http://newvantage.com/wp-content/uploads/2016/01/Big-Data-Executive-Survey-2016-Findings-FINAL.pdf http://newvantage.com/wp-content/uploads/2017/01/Big-Data-Executive-Survey-2017-Executive-Summary.pdf
  4. 4. © 2018 MapR Technologies 4 How do you measure Earth’s oceans? By NASA Goddard Space Flight Center from Greenbelt, MD, USA (Full Disk Image of Earth Captured August 24, 2011) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
  5. 5. © 2018 MapR Technologies 5
  6. 6. © 2018 MapR Technologies 6 Changing How People Work with Data 
 A 19th century big data story Matthew Fountain Maury extracted data from ship’s logs to build amazing charts for navigation
  7. 7. © 2018 MapR Technologies 7 Big data project: Maury’s Wind and Currents charts
  8. 8. © 2018 MapR Technologies 8 Big data project: Maury’s Wind and Currents charts At first, nobody was interested in them…
  9. 9. © 2018 MapR Technologies 9 Big data project: Maury’s Wind and Currents charts At first, nobody was interested in them… …until Captain Jackson shaved a month off the run from Baltimore to Rio de Janeiro Then everybody wanted one!
  10. 10. © 2018 MapR Technologies 10 © 2014 Ellen Friedman People with “vision” think with their eyes closed
  11. 11. © 2018 MapR Technologies 11 Aadhaar Project: Largest Biometric DB in the World •  Unique 12 – digit number for each person in India •  Proof of identity, authenticated anytime, anywhere •  Runs on NoSQL database MapR-DB since 2014 Revolution: Changing a Society Photo credit PANOS, with permission 1.3 B people
  12. 12. © 2018 MapR Technologies 12 Changing Rhythm to How We Work with Data Utility providers using smart meters Collect data every 15 min instead of once a month
  13. 13. © 2018 MapR Technologies 13 Image © E Friedman Self-driving cars: Huge volume of sensor data Time value of data Analysis at the Speed of Life
  14. 14. © 2018 MapR Technologies 14 Changing Rhythm to How We Work with Data Apache Drill SQL query engine with schema discovery for data exploration May shorten prep time when running a new query from days/ weeks to hours Follow community on Twitter: @ApacheDrill
  15. 15. © 2018 MapR Technologies 15 We  need  a  better  fit  to  the   way  business  happens  
  16. 16. © 2018 MapR Technologies 16 A Better Fit •  The way business happens •  A dataflow (datafabric) that matches the shape of business •  Technologies with capabilities to support flexibility and timely response across data centers •  Organization at the human level matches as well
  17. 17. © 2018 MapR Technologies 17 Build a Global Data Fabric Flexibility & agility to respond as life changes
  18. 18. © 2018 MapR Technologies 18 Global Data Fabric •  Comprehensive view of data •  Breaks silos •  Works with multi-tenancy •  Computation (and data) where you want them •  Fine-grained control over who has (and does not have) access
  19. 19. © 2018 MapR Technologies 19 A  DataOps  approach  improves   a  project’s  ability  to  stay   on  target  &  on  time  
  20. 20. © 2018 MapR Technologies 20 DataOps: Brings Flexibility & Focus Platform&network Operations Softwareengineering Architecture&planning Dataengineering Datascience Productmanagement DataOps Team B DataOps Team A Cross functional DataOps teams •  Expands DevOps to include data-heavy roles •  Organized around data-related goals •  Better collaboration and communication between roles From Chap 2 of Machine Learning Logistics, by Ted Dunning & Ellen Friedman © 2018 (O’Reilly Media)
  21. 21. © 2018 MapR Technologies 21 DataOps Principles “DataOps teams seek to orchestrate data, tools, code and environments from beginning to end.” They “…measure performance of data analytics by the insights they deliver.” Thor Olavsrud interview with Ted Dunning & Ellen Friedman for CIO https://www.cio.com/article/3237694/analytics/what-is-dataops-data-operations-analytics.html
  22. 22. © 2018 MapR Technologies 22 Advantages of a DataOps Approach •  Able to pivot & respond to real-world events as they happen •  Improved efficiency and better use of people’s time •  Faster time-to-value •  A good fit to working with a global data fabric
  23. 23. © 2018 MapR Technologies 23 How  do  you  keep  people  from   feeling  threatened  by   change?  
  24. 24. © 2018 MapR Technologies 24 Don’t  be  threatening!  
  25. 25. © 2018 MapR Technologies 25 Why Stream? Munich surfing wave Image © 2017 Ellen Friedman
  26. 26. © 2018 MapR Technologies 26 Stream  transport  supports   microservices      
  27. 27. © 2018 MapR Technologies 27 Stream Transport that Decouples Producers & Consumers P P P C C C Transport Processing Kafka / MapR Streams Good stream transport is persistent, performant & pervasive!
  28. 28. © 2018 MapR Technologies 28 More on Streaming Microservices •  Chapter 3 of Streaming Architecture by Ted Dunning & Ellen Friedman © 2016 (O’Reilly Media) http://shop.oreilly.com/product/0636920049463.do •  “Streaming Microservices” chapter by Ted Dunning & Ellen Friedman in Encyclopedia of Big Data Technologies, Sherif Sakr and Albert Zomaya, editors. In press 2018 (Springer International Publishing) •  Chapter 4 in A Practical Guide to Microservices & Containers by James A. Scott © 2017 (MapR) https://mapr.com/ebooks/microservices-and-containers/title.html
  29. 29. © 2018 MapR Technologies 29 Legacy Applications How Does MapR Solve This? Big Data 1.0 Applications Next-Gen Applications MapR Converged Data Platform High Availability Real Time Unified Security Multi-tenancy Disaster Recovery Global Namespace Real-Time NoQL Database Stream TransportWeb-Scale Storage
  30. 30. © 2018 MapR Technologies 30 With MapR, Geo-Distributed Data Appears Local stream Data source Consumer
  31. 31. © 2018 MapR Technologies 31 With MapR, Geo-Distributed Data Appears Local stream stream Data source Consumer
  32. 32. © 2018 MapR Technologies 32 With MapR, Geo-distributed Data Appears Local stream stream Data source ConsumerGlobal Data Center Regional Data Center
  33. 33. © 2018 MapR Technologies 33 90% of the effort in successful machine learning isn’t the algorithm or the model… It’s the logistics
  34. 34. © 2018 MapR Technologies 34 Why? •  Just getting the training data is hard •  ! The myth of the unitary model •  Model-to-model evaluation •  Respond as the world changes: Agile roll out & roll back when deploy to production
  35. 35. © 2018 MapR Technologies 35 Metrics Metrics ResultsRendezvous Enter Rendezvous Architecture Scores ArchiveDecoy m1 m2 m3 Features / profiles InputRaw Rendezvous Architecture described in: -  Machine Learning Logistics book by Ted Dunning & Ellen Friedman © 2018 (O’Reilly) -  “Rendezvous Architecture” chapter in Encyclopedia of Big Data Technologies. Sherif Sakr and Albert Zomaya, editors. Springer International Publishing, in press 2018.
  36. 36. © 2018 MapR Technologies 36 Best thing about Rendezvous: Agile deployment •  Many “good” models ready and waiting –  Already running –  Ready to deploy into production •  To deploy a new model: just stop ignoring it
  37. 37. © 2018 MapR Technologies 37 Rendezvous to the Rescue: Better ML Logistics •  Stream-1st architecture is a powerful approach with surprisingly widespread advantages –  Innovative technologies emerging for streaming data •  Microservices approach provides flexibility –  Streaming supports microservices (if done right) •  Containers remove surprises –  Predictable environment for running models
  38. 38. © 2018 MapR Technologies 38 Preparation of Input (and Training) Data Model 1 Model 2 Model 3 request Raw Add external data Input Database The world Raw data may contain features you’ll want in future
  39. 39. © 2018 MapR Technologies 39 Raw data is gold!
  40. 40. © 2018 MapR Technologies 40 Decoy Model in the Rendezvous Architecture Input Scores Decoy Model 2 Model 3 Archive •  Looks like a model, but it just archives inputs •  Safe in a good streaming environment, less safe without good isolation
  41. 41. © 2018 MapR Technologies 41 Why do you need new models? Conditions may (will) change
  42. 42. © 2018 MapR Technologies 42 Advantages of Rendezvous Architecture Real model Result Canary Decoy Archive Input
  43. 43. © 2018 MapR Technologies 43 Rendezvous: Mainly for Decisioning Type Systems •  Decisioning style machine learning –  Looking for a “right answer” –  Simpler than interactive machine learning (such as in self-driving car) •  Examples include: –  Fraud detection –  Predictive analytics / market prediction –  Churn prediction (as in telecommunications) –  Yield optimization –  Deep learning in form of speech or image recognition, in some cases
  44. 44. © 2018 MapR Technologies 44 Described in new book on ML management: Download free pdf via @MapR: https://mapr.com/ebook/machine-learning-logistics/ Includes a discussion of DataOps
  45. 45. © 2018 MapR Technologies 45 Example: Tensor Chicken Label training data Run the model Deploy model Gather training data Labeled image files Train model Update model Deep learning project by software engineer Ian Downard (see blog + @tensorchicken)
  46. 46. © 2018 MapR Technologies 46 DataOps: A Good Way to Adapt to Emerging Data Practices •  Faster time-to-value & better ability to pivot –  Better collaboration/communication across skill groups –  Focused around data-related goals –  More efficient use of team members’ time •  A good fit to working with a data fabric •  A good fit for a streaming microservices style
  47. 47. © 2018 MapR Technologies 47 Please support women in tech – help build girls’ dreams of what they can accomplish © Ellen Friedman 2015#womenintech #datawomen
  48. 48. © 2018 MapR Technologies 48 Related events at Strata •  “Better Machine Learning Logistics with Rendezvous Architecture” talk by Ted Dunning Wed at 5:10pm •  “Rendezvous Architecture” booth talk at MapR booth Thur at 11:30 am •  Chat with us in the MapR booth
  49. 49. © 2018 MapR Technologies 49 Thank You !
  50. 50. © 2018 MapR Technologies 50 Additional Resources: Available Now O’Reilly report by Ted Dunning & Ellen Friedman © March 2017 Read free courtesy of MapR: https://mapr.com/geo-distribution-big-data-and-analytics/ O’Reilly book by Ted Dunning & Ellen Friedman © March 2016 Read free courtesy of MapR: https://mapr.com/streaming-architecture-using- apache-kafka-mapr-streams/
  51. 51. © 2018 MapR Technologies 51 Book signings at MapR booth •  Wed afternoon break 3:35 pm – 4:15 pm •  Thur morning break 10:30 am – 11:10 am Get a free copy of the book & meet the authors Ted Dunning & Ellen Friedman Or download free pdf via @MapR: https://mapr.com/ebook/machine-learning-logistics/
  52. 52. © 2018 MapR Technologies 52 Please tell me how DataOps works out for you. Ellen Friedman Twitter @Ellen_Friedman Email @efriedman@mapr.com ellenf@apache.org
  • DaphnDeTroch

    Jan. 26, 2019
  • mathieudumoulin2

    Oct. 31, 2018
  • 1brianjones

    Sep. 24, 2018
  • datakitchen99

    Jun. 14, 2018
  • tlockney

    Jun. 12, 2018
  • botswana99

    May. 18, 2018

DataOps expands DevOps philosophy to include data-heavy roles (data engineering & data science). DataOps uses better cross-functional collaboration for flexibility, fast time to value and an agile workflow for data-intensive applications including machine learning pipelines. (Strata Data San Jose March 2018)

Views

Total views

2,122

On Slideshare

0

From embeds

0

Number of embeds

26

Actions

Downloads

98

Shares

0

Comments

0

Likes

6

×