This document provides an overview of deep learning on mobile devices. It discusses why deep learning is important for mobile, including issues like privacy, reliability and latency. It then covers topics like how to train models for mobile using techniques like transfer learning and fine-tuning. The document also discusses frameworks for running models efficiently on mobile like Core ML, TensorFlow Lite and Google's ML Kit. It explores how hardware impacts performance and how to optimize models. Finally, it touches on applications of deep learning on mobile and techniques like federated learning.
27. @PRACTICALDLBOOK@PRACTICALDLBOOK
TensorFlow Lite is Fast
27
Takes advantage of on-device
hardware acceleration
FlatBuffers
•Reduces code footprint,
memory usage
•Reduces CPU cycles on
serialization and
deserialization
•Improves startup time
Pre-fused activations
Combining batch
normalization layer with
previous Convolution
Static memory and static
execution plan
Decreases load time
31. @PRACTICALDLBOOK@PRACTICALDLBOOK
ML Kit
31
Simple Abstraction over
TensorFlow Lite
Built in APIs for
Image Labeling, OCR, Face
Detection, Barcode
scanning, Landmark
detection, Smart reply
Model management
with Firebase
Upload model on web
interface to distribute
A/B Testing
41. @PRACTICALDLBOOK 41
Alchemy by Fritz
https://alchemy.fritz.ai/
Python library to analyze and
estimate mobile performance
No need to deploy on mobile
43. @PRACTICALDLBOOK 43
Battery
Standpoint
Won’t AI inference kill the battery quickly?
Answers: You don’t usually run AI models constantly, you run
it for a few seconds.
With a modern flagship phone, running Mobilenet
at 30 fps should burn battery in 2-3 hours.
Bigger question - Do you really need to run it at 30
FPS? Or could it be run 1 FPS?
46. @PRACTICALDLBOOK
Seeing AI
Audible Barcode recognition
Aim: Help blind users identify products using barcode
Issue: Blind users don’t know where the barcode is
Solution: Guide user in finding a barcode with audio cues
46
47. @PRACTICALDLBOOK
AR Hand Puppets
Hart Woolery from 2020CV
Object Detection (Hand) + Key Point Estimation
47
[https://twitter.com/2020cv_inc/status/1093219359676280832]
AR Hand Puppets, Hart Woolery from 2020CV, Object Detection (Hand) + Key Point Estimation
58. @PRACTICALDLBOOK
Model Pruning
Aim: Remove all connections with absolute weights below a threshold
58
Song Han, Jeff Pool, John Tran, William J. Dally, "Learning both Weights and Connections for Efficient Neural Networks", 2015
59. @PRACTICALDLBOOK
Pruning in Keras
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation=tf.nn.relu),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
59
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(),
prune.Prune(tf.keras.layers.Dense(512, activation=tf.nn.relu)),
tf.keras.layers.Dropout(0.2),
prune.Prune(tf.keras.layers.Dense(10, activation=tf.nn.softmax))
])
60. @PRACTICALDLBOOK
So many techniques - so little time!
Quantization
Weight sharing
Channel pruning
Filter pruning (ThiNet)
Better Layers (Dilated Conv, HetConv, OctConv)
Knowledge Distillation
Binary networks (BNN, XNOR-Net)
Lottery Ticket Hypothesis
and many more ...
60
64. @PRACTICALDLBOOK@PRACTICALDLBOOK
AutoML – Let AI Design an Efficient Arch
64
Neural Architecture Search (NAS) - An
automated approach for designing models using
reinforcement learning while maximizing
accuracy.
Hardware Aware NAS = Maximizes accuracy
while minimizing run-time on device
Incorporates latency information into the reward
objective function
Measure real-world inference latency by
executing on a particular platform
1.5x faster than MobileNetV2 (MnasNet)
ResNet-50 accuracy with 19x less parameters
SSD300 mAP with 35x less FLOPs
66. @PRACTICALDLBOOK
ProxylessNAS – Per Hardware Tuned CNNs
66
Han Cai and Ligeng Zhu and Song Han, "ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware", ICLR 2019
69. @PRACTICALDLBOOK
On-Device Training in Core ML
69
let updateTask = try MLUpdateTask(
forModelAt: modelUrl,
trainingData: trainingData,
configuration: configuration,
completionHandler: { [weak self]
self.model = context.model
context.model.write(to: newModelUrl)
})
⁻ Core ML 3 introduced on device learning
⁻ Never have to send training data to the server with the
help of MLUpdateTask.
⁻ Schedule training when device is charging to save power
72. @PRACTICALDLBOOK 72
TensorFlow
Federated
Train a global model using 1000s of devices
without access to data
Encryption + Secure Aggregation Protocol
Can take a few days to wait for aggregations to
build up
https://github.com/tensorflow/federated
73. @PRACTICALDLBOOK
What We Learnt Today
73
⁻ Why deep learning on mobile?
⁻ Building a model
⁻ Running a model
⁻ Hardware factors
⁻ Benchmarking
⁻ State-of-the-art applications
⁻ Making a model more efficient
⁻ Federated Learning
[M] Let’s just say that in the 21st century, the human attention span is a bit low.
Amazon published the results of an A/B test they did way back in 2008! They found that every 100ms increase in latency, correlated with a 1% decrease in profits. Imagine what a couple of seconds can do? Actually we don’t need to imagine.
We have the results from a Google study about a decade later where they found that a loading time of 3 seconds or more on a mobile website resulted in 53% probability of a user leaving the webpage.
If you ever had to tell someone the Moore’s law equivalent for human attention span, you can refer them to these numbers that have been to be true as early as 1968 and as late as 1993. It’s very possible it’s gotten worse since push notifications became a thing.
The study found that:
0.1 second is about the limit where the user feels that the system is reacting instantly, like typing characters on the keyboard and having them appear on the screen. no special feedback is necessary except to display the result.
1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data. Or about the amount of time I spend thinking before buying the next Apple product.
10 seconds is about the limit for keeping the user's attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.
At a high level, the recipe for a great deep learning app consists of two things- an efficient inference engine and an efficient model. So how do we train a model?
You don't need Microsoft s or Google’s ocean boiling GPU cluster
We’ve looked at how to do fine-tuning, but we figured it’d cool to demo it live. They say doing a live demo is a bad idea, but what are we if not full of bad ideas. In the spirit of adventure, let’s do it.
[S] We’ve discussing fine-tuning in the context of images, but let’s try it out on a harder input – audio. In the interest of time, we won’t show the process of actually collecting the data, but all we did was to collect or record some audio files for each class. We collected applause because we know there’s going to be a lot of it in this session.
It’s the same revolution that desktop gpus brought about. Core ML models run 38% faster on iOS 12 compared to iOS 11. We’re just at the beginning of an incredible wave of mobile experiences powered by on-device machine learning. Processors like the A12 are going to make it happen.
Minimal dependencies -> Easier to package and deploy
Just one line of code!
On 2 billion devices. Google assistant is on 1 billion devices. Photos, Gboard, Gmail, Nest!
Develop one tflite model for both ios and android apps
[s] Even though an iPhone 10s looks smaller than a macbook air, guess what, its stronger.
I could give you the numbers, but they say showing is better than telling. what's the result of all that powerful hardware
Additionally also gives a layer by layer breakdown of how much processing power it needs
2015 was the year of gpu - Aim for 10 FPS on oldest phones for real-time UX
[s]
As we all have painfully experienced, in real life, what you really want, is not what you can always afford.
And that's the same in machine learning,
We all know deep learning works if you have large GPU servers, what about when you want to run it on a 3 year old device.
What's the number 1 limitation, it turns out to be memory.
If you look at image net in the last couple of years, it started with 240 megabytes. VGG was over half a gig.
So the question we will solve now is how to get these neural networks do these amazing things yet have a very small memory footprint
Pruning redundant, non-informative weights in a previously trained network reduces the size of the network at inference time.
Take a network, prune, and then retrain the remaining connections
Train, prune, retrain all in a loop