How to Survive as a Data Architect in a Polyglot Database World
•
17 likes•2,588 views
Karen Lopez talks to data architects and data moders how they can best deliver value on modern data drive projects beyond relational database technologies. She covers NoSQL Databases and Datastores, which data stories they best fit and which ones they don't. She ends with 10 tips for adding more value to ployschematic database solutions.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to How to Survive as a Data Architect in a Polyglot Database World
Similar to How to Survive as a Data Architect in a Polyglot Database World (20)
More from Karen Lopez
More from Karen Lopez (15)
Recently uploaded
Recently uploaded (20)
How to Survive as a Data Architect in a Polyglot Database World
- 1. Surviving as a Data Architect in a Polyglot Database World Karen Lopez @datachick
- 3. Karen López Karen has 20+ years of data and information architecture experience on large, multi-project programs. She is a frequent speaker on data modeling, data-driven methodologies and pattern data models. She wants you to love your data. www.datamodel.com
- 4. POLL: Audience Who are you?
- 5. Outcomes: Leave here understanding: Why multi- model designs are important Database / datastore types Multi-model thinking How to future-proof your data arch career How to learn more
- 6. A Good Data Architect Cost, Benefit, Risk Data Protection Business Requirements & models Design Models Fit to needs
- 9. What do you mean, SURVIVE? • Continue to be involved in data- related architectural decisions • Data models wanted and appreciated • Data model driven development still a thing • Valued for bringing business needs and database opportunities together • Team player • Knowledgeable of database features
- 11. Hybrid Purely relational (SQL) databases don’t exist any longer • Columnstore features • XML • JSON • Other NoSQL features 01 Applications make use of multiple database and datastore technologies 02 Schemas in a variety of places 03
- 12. Typical Characteristics of Traditional Apps Real-time request/response Synchronous Always connected Transactional Scale-Up Failover
- 13. Modern Cloud Apps Characteristics Asynchronous / Queued Eventual Consistency Microservices Command & Query Pattern Polyglot Persistence
- 14. Do you support/model/do stuff In relational technologies? In pre-relational tech? In post-relational (NoSQL) tech? How’s that going for ya?
- 15. A very fast review of NoSQL & SQL I have other recorded webinars at Dataversity that go into more detail
- 16. Basically Available Soft State Eventually Consistent BASE ACID Atomic Consistent Isolated Durable BASE - ACID
- 17. This used to be an Either/Or decision BASE ACID
- 18. Polyglot persistence •Optimized for data •Optimized for workload Not all new •EAV •XML •Architecture paradigm: OLAP/DW and OLTP But now….
- 19. NoSQL, Not Only SQL, SQL* Relational Key Value Columnar Column Family Document Hadoop Graph Plus all the hybrid versions of these…
- 20. Relational
- 21. Partition Key Row Key Value Marketing 00001 Name Age Bob 35 Marketing 00002 Name Age Karen 42 Marketing Department Name Count MKTG 104 Sales 00010 Name Age Sam 29 • Simple Map • Semi-Structured • De-normalized Key-Value Databases
- 23. • No schema, no relationships • Collections of documents • Documents are JSON objects Document databases { "title": "Polo Long Sleeve Shirt", "color": "red", "material": "cotton" } { "title": "NoSQL Distilled, "author": [ "Martin Fowler", “P. Sadalage" ], "isbn": "978-0321826626", "pages": 192 }
- 24. Document CosmosDB MongoDB BSON & JSON Databases, Documents, Collections
- 25. • Map-based • Relationships of Edges and Nodes Graph databases
- 27. Wait…What? Graph Database and Processing inside relational database engine
- 28. Column Family
- 29. Columnar HP Vertica SAP IQ
- 30. This means… • SQL vs. NoSQL isn’t a thing any longer • Other RDBMS vendors will be adding non- relational functions and structures • Data Architects who specialize in relational-only modeling and design will be overly specialized • We don’t have a choice to ignore these new database models
- 31. Most relational databases are adding NoSQL support Columnstore Graph XML JSON You’ve probably even developed your own as a table
- 32. Hybrid is the future, though • Column Family + Key Value • Relational + Graph + Columnstore • Relational + Columnstore • Graph + Column Family + Key Value + Document …all in the same engine. This is a major change in how things worked a decade ago.
- 33. Did you notice? • Database as a Service from Microsoft • Supports (for now) • Graph • Key Value • Column Family • Document • Globally distributed data • Scalable • Tunable Consistency CosmosDB
- 34. But what does this all mean? Data Modeling process impacts Data Modeler impacts
- 35. Quotes from team NoSQL • No need for a data model • Can’t be mapped to a data model • SQL is not flexible • SQL Databases don’t scale • SQL Databases die after about 1 GB size • SQL Databases use old technology
- 36. Quotes from team SQL (relational) • Eventual Consistency? WTH? • What do you mean data quality just doesn’t matter? • My tools won’t work with these databases • It’s all a fad; There will be a new fad tomorrow • Bbbbbut…but..but DATA QUALITY!!!!! INTEGRITY!!!
- 37. What is the role of existing Data Models?
- 38. Data Models – Traditional Process Conceptual (Data) Model Logical Data Model Physical Data Model(s) OLTP OLTP OLTP OLTP OLTP MARTMART OLTP OLTP OLTP
- 39. What about the Modeling Tasks & methods Schema Schemaless Schema on read Polyschematic
- 40. Schemas • In the database as physical structure • In a separate location such as • DSD • Schema layer on top of the physical structure • In the application code • In the data itself
- 41. New Process Apply Apply logical data model understandings to physical structure • Using data modeling and/or tools • Using a whiteboard • Using code Understand Understand data Understand Understand underlying architecture
- 42. Traditional Data Modeler Involvement Project Initiation Architecture and Infrastructure Design SW Requirements Development Deployment
- 43. Modern Data Modeler Involvement Project Initiation Architecture and Infrastructure Design SW Requirements Development Deployment
- 44. What do we really mean by scale? Rapidly add more compute & data power Massively parallel processing Cheap, commodity hardware, but lots of it Optimized for Query/Reads/Questions/Telling stories
- 45. Most NoSQL • Was developed with scale in mind • Has tunable consistency • Scale out, versus scale up • Uses distributed data (multiple copies), globally • Understanding workloads and workload trends is important This metadata isn’t often collected and modeled by data architect
- 46. Where Data Models Can Help • reverse engineer, where possible. • one entity per tab spreadsheet • some Normalized (master data + combinations) Create a Physical Data Model of Key Data Sources • definitions • gotchas • expected domains • metadata Model the metadata • Describe data consistently • Don’t prescribe structure • Naming standards • Datatype-ish Think Differently
- 47. Think Differently • Data Modeling tools have to catch up • New tools are being developed • Data Models can’t be prescriptions all the time • Naming standards over constraints • Data Types are general, higher level • Tunable data quality
- 48. How are you transitioning to “different”?
- 49. Entity-Relationship Modeling & IDEF1X Is it enough? Should we extend it? Should we create new notations for each type of Database? Scrap and start all of it again?
- 50. What about all those very sexy data modelers?
- 51. “Every design decision should include cost, benefit and risk” - Karen Lopez
- 52. Great Enterprise Data Modelers Characteristics Patient Zen balance / non-attachment Get-er done Collaboration not sentries/cops Broad Business Domain Broad, strategic view Deep, project empathy Respected Good project experience Good negotiators 52
- 53. So let’s summarize: 1. The more SQL-like features available for NoSQL databases, the more likely a data modeling tool is to support it. 2. Modeling tool vendors will support features that users ask for cause them to win deals. This is not a bad thing. 3. Serious NoSQL vendors* understand that hybrid is the enterprise data story. They want us to find a way. 4. Our data models have value, even if the NoSQL solution doesn’t require a lot of constraints.
- 54. 10 Tips for Data Modelers 1. Learn about these methods – don’t avoid them 2. Get hands-on training. Get certified even 3. Learn the lingo 4. Use the lingo 5. Be able to describe data modeling and data governance to the context of these database technologies and their use cases
- 55. 10 Tips for Data Modelers 6. Bring data models (and other models) to the team 7. Be ahead of the curve on new NoSQL and SQL features in your DBMSs 8. Understand the use cases for each type of database technology 9. Let go, a little, when it makes sense 10. Enjoy the new database smell!
- 56. What to do Learn Get Hands-on experience with these new data technologies Talk to your tool vendors Bring models you have
- 57. Resources
- 58. Making Sense of NoSQL clearly and concisely explains the concepts, features, benefits, potential, and limitations of NoSQL technologies. Using examples and use cases, illustrations, and plain, jargon-free writing, this guide shows how you can effectively assemble a NoSQL solution to replace or augment the traditional RDBMS you have now.
- 60. This book is written for anyone who is working with, or will be working with MongoDB, including business analysts, data modelers, database administrators, developers, project managers, and data scientists.
- 62. Questions?
- 63. • #TEAMDATA www.datamodel.com Thank you, you were great.