Let's build a simple ingest to cloud datawarehouse with low code From DevOps Stage 2020 Using NiFi to load data warehouse, cloud data, iot, sensors, rest, csv, network, events, logs and all types of data into various sql, nosql, cloud and on premise data stores. principal dataflow field engineer tim spann

1. Let's Build a Simple Ingest to Cloud Data
Warehouse With Low Code
Timothy Spann
Principal DataFlow Field Engineer
Cloudera
@PaasDev

3. 3
Future of Data - Princeton (Global Youtube)
@PaasDev
https://www.meetup.com/futureofdata-princeton/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to Microservices to ...

4. 4
Speakers
Tim Spann
Principal DataFlow Field Engineer
@PaasDev
DZone Zone Leader and Big Data MVB
Princeton NJ Future of Data Meetup
https://github.com/tspannhw
https://www.datainmotion.dev/

5. © 2020 Cloudera, Inc. All rights reserved. 5
STORAGE LAYER
prices
CFM REFERENCE ARCHITECTURE
Apache NiFi
Apache Kafka
DATA SYNDICATION
SERVICE BY KAFKA
Kafka Topic
iot
DATA FLOW APPS
POWERED BY NIFI
Apache Impala
Deep Learning & Machine
Learning
MODEL EXECUTION
REST

6. © 2019 Cloudera, Inc. All rights reserved. 6
Trillions of Messages to SQL Databases and Data Warehouses
PROCESS DELIVER
HASH
MERGE
EXTRACT
DUPLICATE
SPLIT
ROUTE TEXT
ROUTE CONTENT
ROUTE CONTEXT
CONTROL RATE
DISTRIBUTE LOAD
GEOENRICH
SCAN
REPLACE
TRANSLATE
CONVERT
ENCRYPT
TALL
EVALUATE
EXECUTE

7. 7
Reference Architecture
Files to RDBMS
Source Files Read Files Cloudera DataFlow
Pipeline
PutDatabaseRecord or
PutKudu or
PutHbaseRecord or
PutHiveStreaming

8. © 2019 Cloudera, Inc. All rights reserved. 8
Messages to Databases

9. ‘QueryRecord’ Processor
https://medium.com/@abdelkrim.hadjidj/democratizing-nifi-record-processors-with-automatic-schemas-inferenc
e-4f2b2794c427

10. ADVANCED XML PROCESSING
https://www.datainmotion.dev/2019/03/advanced-xml-processing-with-apache.html
https://pierrevillard.com/2018/06/28/nifi-1-7-xml-reader-writer-and-forkrecord-processor/