As a Data Engineer I am often tasked with taking Machine Learning and Deep Learning models into production, sometimes in the cloud and sometimes at the edge. I have developed Java code that allows us to run these models at the edge and as part of a sensor/webcam/images/data stream. I have developed custom interfaces in Apache NiFi to enable real-time classification against MXNet models directly through the Java API or through DJL.AI's Java interface. I will demo running models on NVIDIA Jetson Nanos and NVIDIA Xavier NX devices as well as in the cloud. # Technologies Utilized: # Apache MXNet, DJL.AI, NVIDIA Jetson Nano, NVIDIA Jetson XAVIER, Apache NiFi, MiNIFi, Java, Python.

1. Using Apache MXNet in Production Deep
Learning Streaming Pipelines
Timothy Spann
Principal DataFlow Field Engineer
Cloudera
@PaasDev

2. 2
Tim Spann
Who am I?
Cloudera Principal DataFlow Field Engineer
@PaasDev
DZone Zone Leader and Big Data MVB
Future of Data Meetup Leader
ex-Pivotal Field Engineer
https://github.com/tspannhw https://www.datainmotion.dev/

3. 3
Welcome to Future of Data - Princeton - Virtual
@PaasDev
https://www.meetup.com/futureofdata-princeton/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to Microservices to ...

4. Apache MXNet Native Processor for Apache NiFi
This is a beta, community release by me using the new beta Java API for Apache MXNet.
https://github.com/tspannhw/nifi-mxnetinference-processor
https://community.hortonworks.com/articles/229215/apache-nifi-processor-for-apache-mxnet-ssd-single.html
https://www.youtube.com/watch?v=Q4dSGPvqXSA

5. • Great documentation
• MXNet Models and Model Zoo
• Fast Java
• Action Recognition
• Image Classification
• Instance Segmentation
• Object Detection
• Pose Estimation
• Q&A
• Support also for PyTorch, Tensorflow and ONNX
• Training
https://djl.ai/ https://github.com/awslabs/djl/blob/master/mxnet/mxnet-model-zoo/README.md
https://github.com/awslabs/djl/blob/master/mxnet/README.md https://d2l.djl.ai/ https://github.com/awslabs/djl/blob/master/docs/faq.md
https://github.com/aws-samples/djl-demo https://github.com/awslabs/djl/releases/tag/v0.8.0 https://djl.ai/pytorch/pytorch-engine/

6. Apache MXNet Native Processor through DJL.AI for Apache
NiFi
This processor uses the DJL.AI Java Interface
https://github.com/tspannhw/nifi-djl-processor
https://dev.to/tspannhw/easy-deep-learning-in-apache-nifi-with-djl-2d79

7. DJL NiFi Processors - Sentiment Analysis
https://www.datainmotion.dev/2020/09/using-djlai-for-deep-learning-based.html
https://github.com/tspannhw/nifi-djlsentimentanalysis-processor
probnegative
0.99
No value set
probnegativeperc
99.44
No value set
probpositive
0.01
No value set
probpositiveperc
0.56
No value set
rawclassification
[class: "Negative", probability: 0.99440, class: "Positive",
probability: 0.00559]

8. BERT QA through DJL.AI for Apache NiFi
This processor uses the DJL.AI Java Interface
https://github.com/tspannhw/nifi-djlqa-processor
https://www.datainmotion.dev/2020/09/using-djlai-for-deep-learning-bert-q-in.html
https://dev.to/tspannhw/easy-deep-learning-in-apache-nifi-with-djl-2d79
The pretrained model is DistilBERT model
trained by HuggingFace using PyTorch.
BERT QA

9. 9
https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/
NiFi Processing Billions of Events

10. Demo

11. 11
Edge AI to Cloud Streaming AI Pipeline
Device Data
SensorsEnergy Logs
Weather
Sensors
Aggregates
Energy
SQL
Analytics
MiNiFi
Agent
Deep Learning
Classification

12. Learn More

13. 13© 2020 Cloudera, Inc. All rights reserved.
REFERENCES
● https://github.com/tspannhw/ApacheConAtHome2020
● https://www.datainmotion.dev/2020/10/flank-streaming-edgeai-on-new-nvidia.html
● https://www.youtube.com/watch?v=h6mS08WDRHY&t=12s
● https://dev.to/tspannhw/using-djl-ai-for-deep-learning-based-sentiment-analysis-in-nifi-dataflow-
3c3a
● https://community.cloudera.com/t5/Community-Articles/Apache-NiFi-Processor-for-Apache-MX
Net-SSD-Single-Shot/ta-p/249240
● https://github.com/tspannhw/nifi-mxnetinference-processor
● https://github.com/tspannhw/nifi-djl-processor
● https://github.com/tspannhw/nifi-djlsentimentanalysis-processor

14. ● https://www.datainmotion.dev/2019/12/easy-deep-learning-in-apache-nifi-with.html
● https://github.com/tspannhw/ApacheConAtHome2020
● https://docs.djl.ai/mxnet/mxnet-engine/index.html
● https://github.com/tspannhw/ApacheDeeplearning202
● https://github.com/tspannhw/ApacheDeepLearning201
● https://github.com/tspannhw/nifi-gluoncv-yolo3
● https://github.com/tspannhw/ApacheDeepLearning101
REFERENCES

15. 15© 2020 Cloudera, Inc. All rights reserved.
DEEPER CONTENT
Title: "Using Apache MXNet in Production Deep Learning Streaming Pipelines"
Abstract:
As a Data Engineer I am often tasked with taking Machine Learning and Deep Learning models into production, sometimes in the cloud
and sometimes at the edge. I have developed Java code that allows us to run these models at the edge and as part of a
sensor/webcam/images/data stream. I have developed custom interfaces in Apache NiFi to enable real-time classification against MXNet
models directly through the Java API or through DJL.AI's Java interface. I will demo running models on NVIDIA Jetson Nanos and
NVIDIA Xavier NX devices as well as in the cloud.
Technologies Utilized:
Apache MXNet, DJL.AI, NVIDIA Jetson Nano, NVIDIA Jetson XAVIER, Apache NiFi, MiNIFi, Java, Python.

16. © 2020 Cloudera, Inc. All rights reserved. 16
TH N Y U