The Modern Data Platform - How to Conquer a New World with Old Problems
•Télécharger en tant que PPTX, PDF•
1 j'aime•1,316 vues
As Hadoop and Big Data technologies take root in the enterprise, they bring new challenges for data orchestration and interaction with traditional environments. The journey of transitioning to a modern data platform involves not only ingesting and offloading data from multiple sources, but also transforming the data, making it instantly available to the business, for on demand analytics. Manual efforts and scripting are no longer practical and projects seem to come to a grinding halt due to failures and delays. Join this session by Cloudwick and BMC and learn through real-world examples of a proven approach that meet the required standards of ease of use, scalability, security, data governance and service level management.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à The Modern Data Platform - How to Conquer a New World with Old Problems
Similaire à The Modern Data Platform - How to Conquer a New World with Old Problems (20)
Plus de DataWorks Summit/Hadoop Summit
Plus de DataWorks Summit/Hadoop Summit (20)
Dernier
Dernier (20)
The Modern Data Platform - How to Conquer a New World with Old Problems
- 1. © Copyright 2014 BMC Software, Inc. 1 — The Modern Data Platform How to Conquer a New World with Old Problems
- 2. © Copyright 2014 BMC Software, Inc. 2 BMC & Control-M Leader in Digital Enterprise Management 98 of Forbes Global 100 & 80%+ of Fortune 500 Workload Automation – #1 Market share – #1 Analyst Recognition – Big Data leadership – Customer satisfaction
- 3. © Copyright 2014 BMC Software, Inc. 3
- 4. © Copyright 2014 BMC Software, Inc. 4 DIGITAL WORLD WE LIVE IN A
- 5. © Copyright 2014 BMC Software, Inc. 5 of S&P 500 will be replaced by 2027 75%25%of businesses will lose competitive ranking due to digital incompetence by 2017 The future is here. Go digital or go extinct. © Copyright 2016 BMC Software, Inc.
- 6. © Copyright 2014 BMC Software, Inc. 6 Progress means building upon … not leaving behind
- 7. © Copyright 2014 BMC Software, Inc. 7 How will Enterprise IT manage Big Data?
- 8. © Copyright 2014 BMC Software, Inc. 8 © 2015 Cloudwick. All rights reserved. Confidential and Proprietary. Build, Operate & Manage Focused on Three Key Phases Assess & Plan Architect & Engineer Optimise & Scale Size, Environment & Use Cases Secure, Govern & Data Management Applications, Analytics & Visualisation
- 9. © Copyright 2014 BMC Software, Inc. 9 © 2015 Cloudwick. All rights reserved. Confidential and Proprietary. 7 Founded in 2010 200+ Data Engineers, Developers & Data Scientist 50k Hadoop Clusters Engineered & Operated On-Prem & Cloud 3 Global Regions Americas | EMEA | APAC 300+ Big Data & Security Enterprise Engagements 6 Strategic Partnerships Cloudwick Open Source, Big Data, Advanced Analytics & Cloud
- 10. © Copyright 2014 BMC Software, Inc. 10 Data Accessibility Typical Enterprise Challenges Unaffordable storage at Scale Data Ingestion / ETL Security Data Governance & Management Data Quality & Performance
- 11. © Copyright 2014 BMC Software, Inc. 11
- 12. © Copyright 2014 BMC Software, Inc. 12 Building on each stage of the Foundation Data Science ML, IoT, AI BI/Analytics Predictive & Prescriptive Reporting Transactional, Operational, Ad-hoc & Self Service Management Strategy, Govern, Quality, Platform, Multi-tenancy & Operations Consolidation Ingest, Cleanse/ETL, Store, Secure
- 13. © Copyright 2014 BMC Software, Inc. 13 Who holds the Key?
- 14. © Copyright 2014 BMC Software, Inc. 14 IT • Enabling the Answers • Reporting: Batch to Streaming • Operational foundation & management • People: Resourcing and developing • Optimise: Savings Focus • Design: Govern, Access & Secure Business • Driving the Questions • Demanding: Mining & Exploring • Immediate & Scalable • People: training & accessing • Generating: Revenue Focus • Enable: Govern, Access and secure Aligning Gaps
- 15. © Copyright 2014 BMC Software, Inc. 15 Prioritising Use Cases
- 16. © Copyright 2014 BMC Software, Inc. 16 Batch to Streaming & Beyond
- 17. © Copyright 2014 BMC Software, Inc. 17 STABILITY COMPLIANCE COST EFFICIENCY
- 18. © Copyright 2014 BMC Software, Inc. 18 Workflow Construction • Drag and Drop makes building workflows simple • Native out-of-the-box integration for traditional and big data applications and technologies • Same Look & Feel HDFS Java MapReduce Pig, Hive, Sqoop Spark Distcp Tajo, Streaming File Transfer Informatica DataStage Business Objects Cognos Oracle Sybase SQL Server SSIS PostgreSQL z/OS Linux/Unix/Windows Amazon EC2 / VMware NetBackup / TSM SAP / OEBS / Peoplesoft
- 19. © Copyright 2014 BMC Software, Inc. 19 AUTOMATION AS A COMPETITIVE WEAPON
- 20. © Copyright 2014 BMC Software, Inc. 20 Sky Q and Control-M © Copyright 2016 BMC Software, Inc.
- 21. © Copyright 2014 BMC Software, Inc. 21 21 Navistar’s OnCommand™ service reduces vehicle downtime by 30% Reduced unplanned repairs by up to 30% 20M+ data records/day collected and transformed from more than 160,000 trucks 20,000 Control-M jobs daily 20% improved time savings for Hadoop engineers Actionable data sets delivered 5x faster “We’ve just begun to tap the power of Control-M to help us use big data to enhance remote diagnostics, improve vehicle quality, and protect critical resources from unauthorized access, among other initiatives.”
- 22. © Copyright 2014 BMC Software, Inc. 22
Notes de l'éditeur
- Introduce myself and a few words about BMC Big Data project is a journey This is not a simple process - right tools, the right skills and even the right culture Enterprise-grade implementations = complex (Data ingestion, Processing, Analytics In this session : Cloudwick and BMC sharing experience
- WE have been in this business of Digital Enterprise Management for over 30 years now Workload Automation or Scheduling – providing end to end automation of business process across the enterprise MF, Unix, Linux, ERP, File Transfers, Java, Informatica, Cognos, Hadoop and so on and so on Analysts recognition
- A personal story – Vietnam Digital Transformation Technology adoption Customer expectations
- It may be cliche, we are truly living in a digital world Every speaker nowadays talk about the use of the mobile (flights, check-in, hotel, etc) Online doctor appointment via video on a mobile app Oh, and yes occasionally I will use it also for phone calls
- Going digital is not an option – it’s a must to maintain and grow the business You can see the numbers Davos World Economic Forum – The chairman of BBVA Talking about the Digital disruption … 20,000 banks in the world and many will disappear – those that will not adopt new technologies as he called it – the Uber moment
- So, let’s drop all the technologies we have been using for decades and go digital? Yeah? Really? Hmm… not quite Anyone recognizes the city in the photo? New York, 1904 , what happened in 1904? The first subway was operated .. But the original subway cars have been replaced, other things like plumbing, lighting, air conditioning, security have been improved as well Today, the subway is still the heartbeat of the city, moving over 4 million people every day, and tying buildings together New York City is a great example of building upon layers, rather than leaving them behind Modernizing the legacy subway infrastructure and at the same time adding the newest modern architectures - like an enterprise IT environment When going digital or specifically implementing Big Data and IOT initiatives, companies cannot drop existing technology and replace with new Like the subway in New York, the strong foundation, is what really allows the modernization to success and flourish
- So, when we talked to companies implementing Big Data we realized that the challenges they are facing … How do I schedule my Big Data workflow? how can I identify something went wrong and fix it on time? How do I assure my SLAs are met? ready for changes? Integrate? secure? scale? I will talk more about it and about our scheduling solution that helps you address all of these challenges, in a couple of minutes but first, let me turn over to our partner Tina Diamond from Cloudwick who will share from her experience and of how they have been helping customers in these areas
- How will Enterprises manage big data - It is about Building a Data Platform for efficiently, optimised for scale Enterprises Require: Security, Access and Control with optimum Performance Scalable architecture in the platform and software Built on a flexible infrastructure Advanced analytics and Apps to grow the business *I would like to introduce C
- I am the MD, I joined Cloudwick, a US based OS PS in Jan 2016 Focused on OS, Big Data& Advanced Analytics, delivering On prem, hybrid and cloud environments to Enterprises *Share old world challenges with new world solutions: What is our Differentiator: Cloudwick was founded on BD, it is our DNA Core/Foundation: Network & Consultants
- These are not new challenges New world challenges with Old World Problems still includes: Building a platform in order to access their data Security Governance: with access and control Predictive / Prescriptive Analytics Example of a UK customer that Transformed these challenges : UKDA
- UKDA – They used old existing way to base the new security… what happened, completely locked down, couldn’t get data in or out UKDA – Largest Government Research Agency for Social Sciences Security – 0 Data Loss – DR strategy took time Hybrid environment accessing data – security – Synergy between platform and analytics Strategy: share their architecture and data with other research compn Going a bit deeper, sharing where The most time and focus was spent understanding and building this solution
- Just like in Maslow’s model, each lower stage is a necessary foundation for the stages above it. If any foundation is weak, the higher stages may crumble and leak even if they initially work. Resources for building each stage involve a combination of human talent and the right set of tools for your needs. Collect: For a robust data pipeline, you want access to clean, reliable data then manage for - Govern, Quality, Platform, Multi-tenancy & Operations Manage: Once you have a consistent stream of clean data, you need to have a clear understanding of the data’s composition. Descriptive statistics is where you look into the past and ask questions that have concrete answers to them. TR: The other question I’m always interested in ,… who holds the key to building?
- Both IT & Business hold the key to unlock the value – the goal is the same, to secure, govern allow the individuals that have signing authority to access – once that is there, the individual can access as often as they like –rely on their valuable assets being in the safest and most secure location for them, with back up strategies and insurance in case of a robbery Business understands what data they want to get to IT unlocks and makes sure it’s secure and accessible to the business users Transition: Once the keys are established, how do you build alignment?
- IT builds from the bottom up to enable business to drive from the top down IT Builds for growth, Business demands growth Automated technology is allowing the Gap to close Though still about understand the old problems with new solutions Business is becoming more savvy with Tech IT are becoming aware of the business AZ…. Pharma that has both Once the synergy and conversations unfold, use cases start to emerge
- How to you…. Determine the Business value I see it by prioritsing Use Cases, and calculating 4 areas: Metrics to improve: application, activation, retention, revenue, referral, operation Expected improvement: 10x or more, 2x or more, 1-2x, 1x, <1x % of users affected: 100%, 80-99%, 50-79%, 20-49%, 10-19%, <10% Frequency of Use Case: Daily, few times/week, weekly, monthly, few times a year, once a year or less Calculate: As the previous slide showed, it’s a staged process, building upon each other, going from….
- Once you implement batch use cases, streaming will follow and accelerate the value for your compeletive edge Global Copyright infringment: Use Case / Batch – got them down from 4 hours / match to 1 second Next implementing Kafka use case Disrupting revenue streaming – Streaming is 50 days? TR: At this time, I’d like to hand over to Alon to share the BMC offering he helps his customers with
- We talked the challenges organizations implementing Big Data have and the many of these challenges have been around for a long time business keeps throwing things at us and it boils down to the needs of the business day With Control-M we are uniquely positioned to help our customers in reconciling these challenges Being consistent across all the technologies EDW offload job or Hadoop Pig or Hive or Scoop or Impala or any other tool to load and manipulate data Beyond Hadoop Self Service Cloud
- The process for building the workflows - Simple drag and drop For example - transfer data, running a Hadoop process like scoop or MapReduce, a few in parallel, creating data and then perhaps another file transfer + apply SLA That’s it - simple as that to put a workflow together No need of scripting or the headache to integrate all apps
- Automation is a must in order to support today’s modern application delivery Scripting is just not good enough The more I talk to orgs implementing Big Data, the more I hear about time wasted on scripting and failures - it doesn’t need 2 b this way What we offer - automate each in every step of your big data journey, from the data ingestion process, through data processing and analytics
- Sky, a British telecommunications – using Big Data to cost-effectively deliver their media services and content Converted from Jenkins to Control-M (management, speed, reliability) Uses Control-M for Hadoop to run customer analytics (make realistic estimations of what people are watching) Estimates a 40% reduction in time from Dev to Prod by replacing Jenkins with Control-M in Development phase
- Navistar, a $10B company who is on the latest frontier with big data and the Internet of Things (IoT) OnCommand Connection system is used to monitor the performance of more than 160,000 trucks It captures millions of data points from vehicles, including fuel economy, geo location, idle time, and potential failures They can diagnose issues as they develop and provide guidance to fleet owners—helping them route trucks to dealer service centers This means less time in the garage and more time on the road, and that translates into a higher return on the investment in the vehicle