SlideShare une entreprise Scribd logo
1  sur  86
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
Deep Learning
Convolutional Neural Networks
Christian S. Perone
christian.perone@gmail.com
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHO AM I
Christian S. Perone
Software Designer
Blog
http://blog.christianperone.com
Open-source projects
https://github.com/perone
Twitter @tarantulae
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
AGENDA
DEEP LEARNING
Introduction
Traditional vs Deep learning
ImageNet Challenge
Deep learning in art
NEURAL NETWORKS
Neural network basics
Making it possible
CONVOLUTIONAL NEURAL NETWORKS
Architecture overview
Convolutional layer
Pooling layer
Dense layers and classification
Deep CNNs
Important ideas
Transfer learning
INTERESTING CASES
Recommendation
Natural language processing
Image/video processing
Q&A
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
Section I
DEEP LEARNING
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT IS DEEP LEARNING ?
Multiple definitions, however, these definitions have in common:
Multiple layers of processing units;
Supervised or unsupervised learning of feature representations
in each layer, with the layers forming a hierarchy from
low-level to high-level features.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
COMPOSITIONAL DATA
NATURAL DATA
IS COMPOSITIONAL.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
COMPOSITIONAL DATA
Image
Source: Convolutional Deep Belief Networks. Honglak Lee, et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
COMPOSITIONAL DATA
Sound
Source: Large Scale Deep Learning. Jeff Dean, joint work with Google.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
ONE SLIDE INTRO TO MACHINE LEARNING
Source: Scikit-Learn (scikit-learn.org)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
TRADITIONAL VS DEEP LEARNING
For many years, we developed feature extractors.
Source: Deep Learning Methods for Vision (Honglak Lee)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
TRADITIONAL VS DEEP LEARNING
Feature extractors, required:
Expert knowledge
Time-consuming hand-tuning
In industrial applications, this is 90% of the time
Sometimes are problem specific
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
TRADITIONAL VS DEEP LEARNING
Feature extractors, required:
Expert knowledge
Time-consuming hand-tuning
In industrial applications, this is 90% of the time
Sometimes are problem specific
But, what if we could learn feature extractors ?
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
TRADITIONAL VS DEEP LEARNING
TRADITIONAL APPROACH
The traditional approach uses fixed feature extractors.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
TRADITIONAL VS DEEP LEARNING
TRADITIONAL APPROACH
The traditional approach uses fixed feature extractors.
DEEP LEARNING APPROACH
Deep Learning approach uses trainable feature extractors.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
TRADITIONAL VS DEEP LEARNING
Source: Lee et.al., ICML2009
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
IMAGENET
Source: t-SNE visualization of CNN codes. Andrej Karpathy
≈ 20.000 object classes
≈ 14 million images
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
IMAGENET
Source: ImageNet
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
IMAGE CLASSIFICATION
Source: We’ve Been Dressing Animals Up as People Way Before the Internet. Jes Greene.
Image classification, can get really hard.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
IMAGENET CHALLENGE
Source: Musings on Deep Learning. Li Jiang.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
DEEP DREAMS
Source: Google Inceptionism
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
ART STYLE
Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
ART STYLE
Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
ART STYLE
Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
ART STYLE
Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
Section II
NEURAL NETWORKS
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
NEURAL NETWORK ARCHITECTURE
Source: Neural Networks and Deep Learning. Michael Nielsen.
Source: Practical Deep N. Networks. Yuhuang Hu et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
MNIST DIGITS CLASSIFICATION
Segmented digits
MNIST digit format (28 x 28 = 784 pixels)
Source: Neural Networks and Deep Learning. Michael Nielsen.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
NEURAL NETWORK ARCHITECTURE
Source: Neural Networks and Deep Learning. Michael Nielsen.
2.225 of 10.000 test images (22.25 % accuracy)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
NEURAL NETWORK ARCHITECTURE
Source: Neural Networks and Deep Learning. Michael Nielsen.
2.225 of 10.000 test images (22.25 % accuracy)
An SVM classifier can get 9.435 of 10.000 ( % 94.35)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
NEURAL NETWORK ARCHITECTURE
Source: Neural Networks and Deep Learning. Michael Nielsen.
2.225 of 10.000 test images (22.25 % accuracy)
An SVM classifier can get 9.435 of 10.000 ( % 94.35)
SVM with hyperparameter optimization can get 98.5%
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
NEURAL NETWORK ARCHITECTURE
Can we do better ?
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
NEURAL NETWORK ARCHITECTURE
Can we do better ?
In fact, yes. The current record is from 2013 and it classifies 9.979
of 10.000 images correctly. The performance is human-equivalent
(or better).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
NEURAL NETWORK ARCHITECTURE
Can we do better ?
In fact, yes. The current record is from 2013 and it classifies 9.979
of 10.000 images correctly. The performance is human-equivalent
(or better).
Source: Neural Networks and Deep Learning. Michael Nielsen.
Neural networks can accurately classify all but 21 of the 10,000 test
images.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
For approximately 20 years, attempts were made to train deeper
neural networks (with more than one hidden layer), however rarely
with benefits (vanishing gradient).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
In 2006, a major breakthrough was made in deep architectures,
following three key principles:
Unsupervised learning of representations is used to pre-train
each layer
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
In 2006, a major breakthrough was made in deep architectures,
following three key principles:
Unsupervised learning of representations is used to pre-train
each layer
Unsupervised training of one layer at a time, on top of the
previously trained ones. The representation learned at each
level is the input for the next layer.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
In 2006, a major breakthrough was made in deep architectures,
following three key principles:
Unsupervised learning of representations is used to pre-train
each layer
Unsupervised training of one layer at a time, on top of the
previously trained ones. The representation learned at each
level is the input for the next layer.
Use supervised training to fine-tune all the layers (in addition
to one or more additional layers that are dedicated to
producing predictions).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
After the 2006 breakthrough, a lot of ideas were also developed.
Nowadays, pre-training is almost obsolete.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
After the 2006 breakthrough, a lot of ideas were also developed.
Nowadays, pre-training is almost obsolete.
New activation functions
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
After the 2006 breakthrough, a lot of ideas were also developed.
Nowadays, pre-training is almost obsolete.
New activation functions
Regularization methods
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
After the 2006 breakthrough, a lot of ideas were also developed.
Nowadays, pre-training is almost obsolete.
New activation functions
Regularization methods
Initialization methods
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
After the 2006 breakthrough, a lot of ideas were also developed.
Nowadays, pre-training is almost obsolete.
New activation functions
Regularization methods
Initialization methods
Data augmentation
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
After the 2006 breakthrough, a lot of ideas were also developed.
Nowadays, pre-training is almost obsolete.
New activation functions
Regularization methods
Initialization methods
Data augmentation
Optimization techniques
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
Another reason on why Deep Learning is possible, is the availability
of lots of data (i.e. ImageNet).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
Another reason on why Deep Learning is possible, is the availability
of lots of data (i.e. ImageNet).
GPGPU also plays an important role on this. For instance, an
NVIDIA GPU (1 Tesla K40 GPU) training a 7 layer Convolutional
Neural Network is nearly 9x faster than CPU.
Convolutions — 80-90% of execution time
Pooling
Activations
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
Companies are working on solutions for Deep Learning
acceleration:
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
Companies are working on solutions for Deep Learning
acceleration:
NVIDIA
NVIDIA created a entire plaftorm stack dedicated to work with Deep
Learning, called DIGITS. Their GPUs are widely used in Deep Learning.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
Companies are working on solutions for Deep Learning
acceleration:
NVIDIA
NVIDIA created a entire plaftorm stack dedicated to work with Deep
Learning, called DIGITS. Their GPUs are widely used in Deep Learning.
AMAZON
Amazon AWS also create EC2 instances with NVIDIA GPUs (with 4GB
of memory and 1536 CUDA cores). Lots of AMIs with Deep Learning
software ecosystem already installed.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WHAT CHANGED ?
Companies are working on solutions for Deep Learning
acceleration:
NVIDIA
NVIDIA created a entire plaftorm stack dedicated to work with Deep
Learning, called DIGITS. Their GPUs are widely used in Deep Learning.
AMAZON
Amazon AWS also create EC2 instances with NVIDIA GPUs (with 4GB
of memory and 1536 CUDA cores). Lots of AMIs with Deep Learning
software ecosystem already installed.
MICROSOFT
Microsoft announced that it will offer NVIDIA GPUs on its Azure cloud
platform.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
Section III
CONVOLUTIONAL NEURAL NETWORKS
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
CONVOLUTIONAL NEURAL NETWORKS
Convolutional Neural Networks (or convnets) are based on the
following principles:
Local receptive fields
Shared weights
Pooling (or down-sampling)
This special neural network architecture takes advantage of the
spatial structure of data.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
CONVOLUTIONAL NEURAL NETWORKS
Convolutional Neural Networks (or convnets) are based on the
following principles:
Local receptive fields
Shared weights
Pooling (or down-sampling)
This special neural network architecture takes advantage of the
spatial structure of data.
Source: Deeply-Supervised Nets. Zhuowen Tu.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
LOCAL CONNECTIVITY
Let’s take the MNIST digits images as input of our convnet. These
images are 28x28 pixels:
28x28 image Local connectivity (5x5)
Source: Neural Networks and Deep Learning. Michael Nielsen.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
LOCAL CONNECTIVITY
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
LOCAL CONNECTIVITY
By “sliding” it, we create a feature map of or 24x24 neurons in the
hidden layer. We can also have a different stride and padding.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
SHARED WEIGHTS
In this local receptive field, Convolutional Neural Networks use the
same shared weights for each of the 24x24 hidden neurons. This
means that we have a great advantage of parameter reduction, for
instance, for a 5x5 receptive field, we’ll need only 25 shared
weights1.
1
Excluding the bias
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
SHARED WEIGHTS
In this local receptive field, Convolutional Neural Networks use the
same shared weights for each of the 24x24 hidden neurons. This
means that we have a great advantage of parameter reduction, for
instance, for a 5x5 receptive field, we’ll need only 25 shared
weights1.
20 feature maps using 5x5 — 20*26 = 520 weights
A fully connected first layer, with 784=28*28 input neurons,
and a relatively modest 30 hidden neurons, would produce
784*30 = 23.520 weights, more than 40 times as many
parameters as the convolutional layer.
1
Excluding the bias
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
CONVOLUTIONAL LAYER
The shared weights and bias are called kernel or filter.
Convolutional layers provides
translation invariance. Since these
filters works on every part of the
image, they are “searching” for
the same feature everywhere in
the image.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
ARCHITECTURE OVERVIEW
Input layer – 28x28 pixels
Convolutional layer — 3 feature maps (5x5 kernel)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
POOLING LAYER
Pooling layers are usually present after a convolutional layer. They
provide a down-sampling of the convolution output.
In the example above, a 2x2 region is being used as input of the
pooling.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
POOLING LAYER
There are different types of pooling, the most used is the
max-pooling and average pooling:
Pooling layers downsamples the volume spatially, reducing small
translations of the features. They also provide a parameter reduction.
Source: CS231n Convolutional Neural Networks for Visual Recognition.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
POOLING LAYER
Max-pooling is how the network asks whether a feature is found
anywhere in some region of the image. After that, it will lose the
exact position.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
ARCHITECTURE OVERVIEW
Input layer – 28x28 pixels
Convolutional layer — 3 feature maps (5x5 kernel)
Pooling Layer — 2x2, resulting in 3x12x12
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
CLASSIFICATION
As you can see, we then add a dense fully-connected layer (usually
using softmax) at the end of the neural network in order to get
predictions for the problem we’re working on (10 classes, 10 digits).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
GOING DEEPER
We have defined all the components required to create a
Convolutional Neural Network, but you’ll rarely see a shallow
convnet like that.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
GOING DEEPER
Actually, experiments demonstrated that the replication of
convolutional + pooling layers produces better results the deeper
you go. Winners of ImageNet challenge, have more than 15 layers
(VGGNet has 19 layers).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
GOING DEEPER
Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al.
Galaxy Zoo best performing network (winner of the challenge).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
DROPOUT TECHNIQUE
Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”. Nitish Srivastava et. al.
The dropout technique helps with the overfitting, specially on dense
layers. Drop occur only at training time, not on test time.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
ACTIVATION FUNCTIONS
Source: Big Data Analytics. Fei Wang.
ReLu helps with the vanishing gradient problem
ReLu generates sparsity
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
DATA AUGMENTATION
Data augmentation can help with overfitting and will certainly
improve improve results.
Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
DATA AUGMENTATION
Data augmentation can help with overfitting and will certainly
improve improve results.
Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al.
Small rotations
Small translation
Scaling
Flipping
Brightness
Noise
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
TRANSFER LEARNING
Features learned by Convolutional Neural Networks on large dataset
problem (i.e. ImageNet), can be helpful on different problems. It’s very
common to pre-train a convnet on ImageNet and then use it as a fixed
feature extractor or as initialization.
CONVNETS AS FEATURE EXTRACTORS
We can remove the last layer and then use these features to extract features,
these features are very useful features for classification. Some people use
these features with LSH (locality-sensitive hashing) to scale large databases
for image search. You can also use these features as input for a SVM
classifier for instance.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
TRANSFER LEARNING
Features learned by Convolutional Neural Networks on large dataset
problem (i.e. ImageNet), can be helpful on different problems. It’s very
common to pre-train a convnet on ImageNet and then use it as a fixed
feature extractor or as initialization.
FINE-TUNING THE CONVNETS
You can use a pre-trained convnet to continue its training on your data
and thus fine-tune the weights for your problem. First layers of a convnet
contains generic features (i.e. edge detectors, etc.) that should be helpful in
many tasks. Deeper layers becomes progressively specific to the details of
the classes of the original problem.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
Section IV
INTERESTING CASES
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
MUSIC RECOMMENDATION
Source: Recommending music on Spotify with deep learning. Sander Dieleman.
This is an example architecture from Spotify, using Convolutional
Neural Network for music recommendation.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
MUSIC RECOMMENDATION
Learned filters at first convolutional layer.
The time axis is horizontal, the frequency axis is vertical (frequency
increases from top to bottom)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
NATURAL LANGUAGE PROCESSING
Source: Text understanding from scratch. Xiang Zhang, Yann LeCun.
Deciding if a review posted on Amazon is positive or negative with
96% accuracy, and predict the actual number of stars with 73%
accuracy.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WORD2VEC
Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov.
Word vectors (trained with up to hundreds of billions of words).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WORD2VEC
Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov.
Word vectors (trained with up to hundreds of billions of words).
With nice properties:
v(’Paris’) - v(’France’) + v(’Italy’ ) ≈ v(’Rome’)
v(’king’) - v(’man’) + v(’woman’) ≈ v(’queen’)
No deep learning and no convnet, but a great distributed representation
example.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
WORD2VEC
Machine Translation
Source: DL4J.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
DOC2VEC
Sentiment analysis
Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
INTERESTING FRAMES
Google recently put in production a Deep Neural Network to
improve YouTube video thumbnails.
Source: Google Research.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
SPATIAL TRANSFORMER NETWOKRS
Spatial Transformer Networks can learn transformations.
Source: Spatial Transformer Networks. Max Jaderberg, et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
DEEPFACE BY FACEBOOK
Source: DeepFace: Closing the Gap to Human-Level Performance in Face Verification. Yaniv Taigman, et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
Section V
Q&A
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A
Q&A

Contenu connexe

Tendances

Convolutional Neural Network (CNN) - image recognition
Convolutional Neural Network (CNN)  - image recognitionConvolutional Neural Network (CNN)  - image recognition
Convolutional Neural Network (CNN) - image recognitionYUNG-KUEI CHEN
 
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...Simplilearn
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Simplilearn
 
Deep Learning: Application & Opportunity
Deep Learning: Application & OpportunityDeep Learning: Application & Opportunity
Deep Learning: Application & OpportunityiTrain
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Gaurav Mittal
 
Convolutional Neural Network and Its Applications
Convolutional Neural Network and Its ApplicationsConvolutional Neural Network and Its Applications
Convolutional Neural Network and Its ApplicationsKasun Chinthaka Piyarathna
 
Convolutional Neural Networks
Convolutional Neural NetworksConvolutional Neural Networks
Convolutional Neural NetworksAshray Bhandare
 
Convolutional neural network
Convolutional neural networkConvolutional neural network
Convolutional neural networkFerdous ahmed
 
Deep learning - A Visual Introduction
Deep learning - A Visual IntroductionDeep learning - A Visual Introduction
Deep learning - A Visual IntroductionLukas Masuch
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkKnoldus Inc.
 
Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Suraj Aavula
 
Deep Learning in Computer Vision
Deep Learning in Computer VisionDeep Learning in Computer Vision
Deep Learning in Computer VisionSungjoon Choi
 
CNN and its applications by ketaki
CNN and its applications by ketakiCNN and its applications by ketaki
CNN and its applications by ketakiKetaki Patwari
 
Introduction to Deep learning
Introduction to Deep learningIntroduction to Deep learning
Introduction to Deep learningleopauly
 
Neural networks and deep learning
Neural networks and deep learningNeural networks and deep learning
Neural networks and deep learningJörgen Sandig
 
Deep Learning - Overview of my work II
Deep Learning - Overview of my work IIDeep Learning - Overview of my work II
Deep Learning - Overview of my work IIMohamed Loey
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkYan Xu
 
Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Basit Rafiq
 

Tendances (20)

Convolutional Neural Network (CNN) - image recognition
Convolutional Neural Network (CNN)  - image recognitionConvolutional Neural Network (CNN)  - image recognition
Convolutional Neural Network (CNN) - image recognition
 
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
 
Deep Learning: Application & Opportunity
Deep Learning: Application & OpportunityDeep Learning: Application & Opportunity
Deep Learning: Application & Opportunity
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)
 
Convolutional Neural Network and Its Applications
Convolutional Neural Network and Its ApplicationsConvolutional Neural Network and Its Applications
Convolutional Neural Network and Its Applications
 
Convolutional Neural Networks
Convolutional Neural NetworksConvolutional Neural Networks
Convolutional Neural Networks
 
Convolutional neural network
Convolutional neural networkConvolutional neural network
Convolutional neural network
 
Deep learning - A Visual Introduction
Deep learning - A Visual IntroductionDeep learning - A Visual Introduction
Deep learning - A Visual Introduction
 
Introduction to Deep learning
Introduction to Deep learningIntroduction to Deep learning
Introduction to Deep learning
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural Network
 
Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Convolution Neural Network (CNN)
Convolution Neural Network (CNN)
 
Deep Learning in Computer Vision
Deep Learning in Computer VisionDeep Learning in Computer Vision
Deep Learning in Computer Vision
 
CNN and its applications by ketaki
CNN and its applications by ketakiCNN and its applications by ketaki
CNN and its applications by ketaki
 
AlexNet
AlexNetAlexNet
AlexNet
 
Introduction to Deep learning
Introduction to Deep learningIntroduction to Deep learning
Introduction to Deep learning
 
Neural networks and deep learning
Neural networks and deep learningNeural networks and deep learning
Neural networks and deep learning
 
Deep Learning - Overview of my work II
Deep Learning - Overview of my work IIDeep Learning - Overview of my work II
Deep Learning - Overview of my work II
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural Network
 
Convolution Neural Network (CNN)
Convolution Neural Network (CNN)Convolution Neural Network (CNN)
Convolution Neural Network (CNN)
 

En vedette

Deep Learning - Convolutional Neural Networks - Architectural Zoo
Deep Learning - Convolutional Neural Networks - Architectural ZooDeep Learning - Convolutional Neural Networks - Architectural Zoo
Deep Learning - Convolutional Neural Networks - Architectural ZooChristian Perone
 
Deep Learning And Business Models (VNITC 2015-09-13)
Deep Learning And Business Models (VNITC 2015-09-13)Deep Learning And Business Models (VNITC 2015-09-13)
Deep Learning And Business Models (VNITC 2015-09-13)Ha Phuong
 
Intro To Convolutional Neural Networks
Intro To Convolutional Neural NetworksIntro To Convolutional Neural Networks
Intro To Convolutional Neural NetworksMark Scully
 
Word Embeddings - Introduction
Word Embeddings - IntroductionWord Embeddings - Introduction
Word Embeddings - IntroductionChristian Perone
 
An Introduction to Deep Learning
An Introduction to Deep LearningAn Introduction to Deep Learning
An Introduction to Deep LearningPoo Kuan Hoong
 
Deep Learning Models for Question Answering
Deep Learning Models for Question AnsweringDeep Learning Models for Question Answering
Deep Learning Models for Question AnsweringSujit Pal
 
Intro to Deep Learning for Question Answering
Intro to Deep Learning for Question AnsweringIntro to Deep Learning for Question Answering
Intro to Deep Learning for Question AnsweringTraian Rebedea
 
Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...
Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...
Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...Andrew Gardner
 
Deep learning - Conceptual understanding and applications
Deep learning - Conceptual understanding and applicationsDeep learning - Conceptual understanding and applications
Deep learning - Conceptual understanding and applicationsBuhwan Jeong
 
Artificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningArtificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningSujit Pal
 
Deep Learning through Examples
Deep Learning through ExamplesDeep Learning through Examples
Deep Learning through ExamplesSri Ambati
 
Transform your Business with AI, Deep Learning and Machine Learning
Transform your Business with AI, Deep Learning and Machine LearningTransform your Business with AI, Deep Learning and Machine Learning
Transform your Business with AI, Deep Learning and Machine LearningSri Ambati
 

En vedette (14)

Deep Learning - Convolutional Neural Networks - Architectural Zoo
Deep Learning - Convolutional Neural Networks - Architectural ZooDeep Learning - Convolutional Neural Networks - Architectural Zoo
Deep Learning - Convolutional Neural Networks - Architectural Zoo
 
Deep Learning And Business Models (VNITC 2015-09-13)
Deep Learning And Business Models (VNITC 2015-09-13)Deep Learning And Business Models (VNITC 2015-09-13)
Deep Learning And Business Models (VNITC 2015-09-13)
 
Intro To Convolutional Neural Networks
Intro To Convolutional Neural NetworksIntro To Convolutional Neural Networks
Intro To Convolutional Neural Networks
 
Word Embeddings - Introduction
Word Embeddings - IntroductionWord Embeddings - Introduction
Word Embeddings - Introduction
 
Moving object detection
Moving object detectionMoving object detection
Moving object detection
 
An Introduction to Deep Learning
An Introduction to Deep LearningAn Introduction to Deep Learning
An Introduction to Deep Learning
 
Deep Learning Models for Question Answering
Deep Learning Models for Question AnsweringDeep Learning Models for Question Answering
Deep Learning Models for Question Answering
 
Intro to Deep Learning for Question Answering
Intro to Deep Learning for Question AnsweringIntro to Deep Learning for Question Answering
Intro to Deep Learning for Question Answering
 
Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...
Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...
Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...
 
Deep learning - Conceptual understanding and applications
Deep learning - Conceptual understanding and applicationsDeep learning - Conceptual understanding and applications
Deep learning - Conceptual understanding and applications
 
Artificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningArtificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep Learning
 
Deep Learning through Examples
Deep Learning through ExamplesDeep Learning through Examples
Deep Learning through Examples
 
Transform your Business with AI, Deep Learning and Machine Learning
Transform your Business with AI, Deep Learning and Machine LearningTransform your Business with AI, Deep Learning and Machine Learning
Transform your Business with AI, Deep Learning and Machine Learning
 
AI and the Future of Growth
AI and the Future of GrowthAI and the Future of Growth
AI and the Future of Growth
 

Similaire à Deep Learning - Convolutional Neural Networks

An Introduction to Deep Learning (May 2018)
An Introduction to Deep Learning (May 2018)An Introduction to Deep Learning (May 2018)
An Introduction to Deep Learning (May 2018)Julien SIMON
 
Problems with CNNs and Introduction to capsule neural networks
Problems with CNNs and Introduction to capsule neural networksProblems with CNNs and Introduction to capsule neural networks
Problems with CNNs and Introduction to capsule neural networksVipul Vaibhaw
 
What's Next: Cloudy with a chance of AI – Deep Learning
What's Next: Cloudy with a chance of AI – Deep LearningWhat's Next: Cloudy with a chance of AI – Deep Learning
What's Next: Cloudy with a chance of AI – Deep LearningOgilvy Consulting
 
Deep learning for real life applications
Deep learning for real life applicationsDeep learning for real life applications
Deep learning for real life applicationsAnas Arram, Ph.D
 
An introduction to deep learning concepts
An introduction to deep learning conceptsAn introduction to deep learning concepts
An introduction to deep learning conceptsAmazon Web Services
 
Big Data LDN 2017: Deep Learning Demystified
Big Data LDN 2017: Deep Learning DemystifiedBig Data LDN 2017: Deep Learning Demystified
Big Data LDN 2017: Deep Learning DemystifiedMatt Stubbs
 
Introduction to deep learning
Introduction to deep learningIntroduction to deep learning
Introduction to deep learningdoppenhe
 
Intro deep learning
Intro deep learningIntro deep learning
Intro deep learningmravendi
 
Neural Networks, Spark MLlib, Deep Learning
Neural Networks, Spark MLlib, Deep LearningNeural Networks, Spark MLlib, Deep Learning
Neural Networks, Spark MLlib, Deep LearningAsim Jalis
 
Deep Learning: concepts and use cases (October 2018)
Deep Learning: concepts and use cases (October 2018)Deep Learning: concepts and use cases (October 2018)
Deep Learning: concepts and use cases (October 2018)Julien SIMON
 
Introduction to deep learning
Introduction to deep learningIntroduction to deep learning
Introduction to deep learningAmr Rashed
 
An Introduction to Deep Learning I AWS Dev Day 2018
An Introduction to Deep Learning I AWS Dev Day 2018An Introduction to Deep Learning I AWS Dev Day 2018
An Introduction to Deep Learning I AWS Dev Day 2018AWS Germany
 
An Introduction to Deep Learning (April 2018)
An Introduction to Deep Learning (April 2018)An Introduction to Deep Learning (April 2018)
An Introduction to Deep Learning (April 2018)Julien SIMON
 
[PR12] understanding deep learning requires rethinking generalization
[PR12] understanding deep learning requires rethinking generalization[PR12] understanding deep learning requires rethinking generalization
[PR12] understanding deep learning requires rethinking generalizationJaeJun Yoo
 
[PR12] Inception and Xception - Jaejun Yoo
[PR12] Inception and Xception - Jaejun Yoo[PR12] Inception and Xception - Jaejun Yoo
[PR12] Inception and Xception - Jaejun YooJaeJun Yoo
 
Biological Foundations for Deep Learning: Towards Decision Networks
 Biological Foundations for Deep Learning: Towards Decision Networks Biological Foundations for Deep Learning: Towards Decision Networks
Biological Foundations for Deep Learning: Towards Decision Networksdiannepatricia
 
Case study on deep learning
Case study on deep learningCase study on deep learning
Case study on deep learningHarshitBarde
 
Promises of Deep Learning
Promises of Deep LearningPromises of Deep Learning
Promises of Deep LearningDavid Khosid
 

Similaire à Deep Learning - Convolutional Neural Networks (20)

An Introduction to Deep Learning (May 2018)
An Introduction to Deep Learning (May 2018)An Introduction to Deep Learning (May 2018)
An Introduction to Deep Learning (May 2018)
 
Problems with CNNs and Introduction to capsule neural networks
Problems with CNNs and Introduction to capsule neural networksProblems with CNNs and Introduction to capsule neural networks
Problems with CNNs and Introduction to capsule neural networks
 
What's Next: Cloudy with a chance of AI – Deep Learning
What's Next: Cloudy with a chance of AI – Deep LearningWhat's Next: Cloudy with a chance of AI – Deep Learning
What's Next: Cloudy with a chance of AI – Deep Learning
 
Deep learning for real life applications
Deep learning for real life applicationsDeep learning for real life applications
Deep learning for real life applications
 
An introduction to deep learning concepts
An introduction to deep learning conceptsAn introduction to deep learning concepts
An introduction to deep learning concepts
 
Big Data LDN 2017: Deep Learning Demystified
Big Data LDN 2017: Deep Learning DemystifiedBig Data LDN 2017: Deep Learning Demystified
Big Data LDN 2017: Deep Learning Demystified
 
Introduction to deep learning
Introduction to deep learningIntroduction to deep learning
Introduction to deep learning
 
Intro deep learning
Intro deep learningIntro deep learning
Intro deep learning
 
C1_W1.pdf
C1_W1.pdfC1_W1.pdf
C1_W1.pdf
 
Neural Networks, Spark MLlib, Deep Learning
Neural Networks, Spark MLlib, Deep LearningNeural Networks, Spark MLlib, Deep Learning
Neural Networks, Spark MLlib, Deep Learning
 
Deep Learning: concepts and use cases (October 2018)
Deep Learning: concepts and use cases (October 2018)Deep Learning: concepts and use cases (October 2018)
Deep Learning: concepts and use cases (October 2018)
 
Introduction to deep learning
Introduction to deep learningIntroduction to deep learning
Introduction to deep learning
 
An Introduction to Deep Learning I AWS Dev Day 2018
An Introduction to Deep Learning I AWS Dev Day 2018An Introduction to Deep Learning I AWS Dev Day 2018
An Introduction to Deep Learning I AWS Dev Day 2018
 
An Introduction to Deep Learning (April 2018)
An Introduction to Deep Learning (April 2018)An Introduction to Deep Learning (April 2018)
An Introduction to Deep Learning (April 2018)
 
[PR12] understanding deep learning requires rethinking generalization
[PR12] understanding deep learning requires rethinking generalization[PR12] understanding deep learning requires rethinking generalization
[PR12] understanding deep learning requires rethinking generalization
 
[PR12] Inception and Xception - Jaejun Yoo
[PR12] Inception and Xception - Jaejun Yoo[PR12] Inception and Xception - Jaejun Yoo
[PR12] Inception and Xception - Jaejun Yoo
 
Biological Foundations for Deep Learning: Towards Decision Networks
 Biological Foundations for Deep Learning: Towards Decision Networks Biological Foundations for Deep Learning: Towards Decision Networks
Biological Foundations for Deep Learning: Towards Decision Networks
 
Case study on deep learning
Case study on deep learningCase study on deep learning
Case study on deep learning
 
Promises of Deep Learning
Promises of Deep LearningPromises of Deep Learning
Promises of Deep Learning
 
Introduction to deep learning
Introduction to deep learningIntroduction to deep learning
Introduction to deep learning
 

Plus de Christian Perone

Gradient-based optimization for Deep Learning: a short introduction
Gradient-based optimization for Deep Learning: a short introductionGradient-based optimization for Deep Learning: a short introduction
Gradient-based optimization for Deep Learning: a short introductionChristian Perone
 
Bayesian modelling for COVID-19 seroprevalence studies
Bayesian modelling for COVID-19 seroprevalence studiesBayesian modelling for COVID-19 seroprevalence studies
Bayesian modelling for COVID-19 seroprevalence studiesChristian Perone
 
Uncertainty Estimation in Deep Learning
Uncertainty Estimation in Deep LearningUncertainty Estimation in Deep Learning
Uncertainty Estimation in Deep LearningChristian Perone
 
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Apache Spark - Intro to Large-scale recommendations with Apache Spark and PythonApache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Apache Spark - Intro to Large-scale recommendations with Apache Spark and PythonChristian Perone
 
Machine Learning com Python e Scikit-learn
Machine Learning com Python e Scikit-learnMachine Learning com Python e Scikit-learn
Machine Learning com Python e Scikit-learnChristian Perone
 
Python - Introdução Básica
Python - Introdução BásicaPython - Introdução Básica
Python - Introdução BásicaChristian Perone
 
C++0x :: Introduction to some amazing features
C++0x :: Introduction to some amazing featuresC++0x :: Introduction to some amazing features
C++0x :: Introduction to some amazing featuresChristian Perone
 

Plus de Christian Perone (9)

PyTorch 2 Internals
PyTorch 2 InternalsPyTorch 2 Internals
PyTorch 2 Internals
 
Gradient-based optimization for Deep Learning: a short introduction
Gradient-based optimization for Deep Learning: a short introductionGradient-based optimization for Deep Learning: a short introduction
Gradient-based optimization for Deep Learning: a short introduction
 
Bayesian modelling for COVID-19 seroprevalence studies
Bayesian modelling for COVID-19 seroprevalence studiesBayesian modelling for COVID-19 seroprevalence studies
Bayesian modelling for COVID-19 seroprevalence studies
 
Uncertainty Estimation in Deep Learning
Uncertainty Estimation in Deep LearningUncertainty Estimation in Deep Learning
Uncertainty Estimation in Deep Learning
 
PyTorch under the hood
PyTorch under the hoodPyTorch under the hood
PyTorch under the hood
 
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Apache Spark - Intro to Large-scale recommendations with Apache Spark and PythonApache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
 
Machine Learning com Python e Scikit-learn
Machine Learning com Python e Scikit-learnMachine Learning com Python e Scikit-learn
Machine Learning com Python e Scikit-learn
 
Python - Introdução Básica
Python - Introdução BásicaPython - Introdução Básica
Python - Introdução Básica
 
C++0x :: Introduction to some amazing features
C++0x :: Introduction to some amazing featuresC++0x :: Introduction to some amazing features
C++0x :: Introduction to some amazing features
 

Dernier

AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 

Dernier (20)

AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 

Deep Learning - Convolutional Neural Networks

  • 1. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Deep Learning Convolutional Neural Networks Christian S. Perone christian.perone@gmail.com
  • 2. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHO AM I Christian S. Perone Software Designer Blog http://blog.christianperone.com Open-source projects https://github.com/perone Twitter @tarantulae
  • 3. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A AGENDA DEEP LEARNING Introduction Traditional vs Deep learning ImageNet Challenge Deep learning in art NEURAL NETWORKS Neural network basics Making it possible CONVOLUTIONAL NEURAL NETWORKS Architecture overview Convolutional layer Pooling layer Dense layers and classification Deep CNNs Important ideas Transfer learning INTERESTING CASES Recommendation Natural language processing Image/video processing Q&A
  • 4. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section I DEEP LEARNING
  • 5. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT IS DEEP LEARNING ? Multiple definitions, however, these definitions have in common: Multiple layers of processing units; Supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features.
  • 6. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA NATURAL DATA IS COMPOSITIONAL.
  • 7. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA Image Source: Convolutional Deep Belief Networks. Honglak Lee, et. al.
  • 8. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA Sound Source: Large Scale Deep Learning. Jeff Dean, joint work with Google.
  • 9. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ONE SLIDE INTRO TO MACHINE LEARNING Source: Scikit-Learn (scikit-learn.org)
  • 10. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING For many years, we developed feature extractors. Source: Deep Learning Methods for Vision (Honglak Lee)
  • 11. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Feature extractors, required: Expert knowledge Time-consuming hand-tuning In industrial applications, this is 90% of the time Sometimes are problem specific
  • 12. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Feature extractors, required: Expert knowledge Time-consuming hand-tuning In industrial applications, this is 90% of the time Sometimes are problem specific But, what if we could learn feature extractors ?
  • 13. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING TRADITIONAL APPROACH The traditional approach uses fixed feature extractors.
  • 14. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING TRADITIONAL APPROACH The traditional approach uses fixed feature extractors. DEEP LEARNING APPROACH Deep Learning approach uses trainable feature extractors.
  • 15. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Source: Lee et.al., ICML2009
  • 16. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET Source: t-SNE visualization of CNN codes. Andrej Karpathy ≈ 20.000 object classes ≈ 14 million images
  • 17. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET Source: ImageNet
  • 18. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGE CLASSIFICATION Source: We’ve Been Dressing Animals Up as People Way Before the Internet. Jes Greene. Image classification, can get really hard.
  • 19. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET CHALLENGE Source: Musings on Deep Learning. Li Jiang.
  • 20. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DEEP DREAMS Source: Google Inceptionism
  • 21. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
  • 22. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
  • 23. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
  • 24. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
  • 25. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section II NEURAL NETWORKS
  • 26. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. Source: Practical Deep N. Networks. Yuhuang Hu et. al.
  • 27. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MNIST DIGITS CLASSIFICATION Segmented digits MNIST digit format (28 x 28 = 784 pixels) Source: Neural Networks and Deep Learning. Michael Nielsen.
  • 28. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy)
  • 29. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy) An SVM classifier can get 9.435 of 10.000 ( % 94.35)
  • 30. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy) An SVM classifier can get 9.435 of 10.000 ( % 94.35) SVM with hyperparameter optimization can get 98.5%
  • 31. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ?
  • 32. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ? In fact, yes. The current record is from 2013 and it classifies 9.979 of 10.000 images correctly. The performance is human-equivalent (or better).
  • 33. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ? In fact, yes. The current record is from 2013 and it classifies 9.979 of 10.000 images correctly. The performance is human-equivalent (or better). Source: Neural Networks and Deep Learning. Michael Nielsen. Neural networks can accurately classify all but 21 of the 10,000 test images.
  • 34. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? For approximately 20 years, attempts were made to train deeper neural networks (with more than one hidden layer), however rarely with benefits (vanishing gradient).
  • 35. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer
  • 36. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer Unsupervised training of one layer at a time, on top of the previously trained ones. The representation learned at each level is the input for the next layer.
  • 37. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer Unsupervised training of one layer at a time, on top of the previously trained ones. The representation learned at each level is the input for the next layer. Use supervised training to fine-tune all the layers (in addition to one or more additional layers that are dedicated to producing predictions).
  • 38. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete.
  • 39. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions
  • 40. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods
  • 41. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods
  • 42. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods Data augmentation
  • 43. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods Data augmentation Optimization techniques
  • 44. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Another reason on why Deep Learning is possible, is the availability of lots of data (i.e. ImageNet).
  • 45. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Another reason on why Deep Learning is possible, is the availability of lots of data (i.e. ImageNet). GPGPU also plays an important role on this. For instance, an NVIDIA GPU (1 Tesla K40 GPU) training a 7 layer Convolutional Neural Network is nearly 9x faster than CPU. Convolutions — 80-90% of execution time Pooling Activations
  • 46. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration:
  • 47. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning.
  • 48. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning. AMAZON Amazon AWS also create EC2 instances with NVIDIA GPUs (with 4GB of memory and 1536 CUDA cores). Lots of AMIs with Deep Learning software ecosystem already installed.
  • 49. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning. AMAZON Amazon AWS also create EC2 instances with NVIDIA GPUs (with 4GB of memory and 1536 CUDA cores). Lots of AMIs with Deep Learning software ecosystem already installed. MICROSOFT Microsoft announced that it will offer NVIDIA GPUs on its Azure cloud platform.
  • 50. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section III CONVOLUTIONAL NEURAL NETWORKS
  • 51. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL NEURAL NETWORKS Convolutional Neural Networks (or convnets) are based on the following principles: Local receptive fields Shared weights Pooling (or down-sampling) This special neural network architecture takes advantage of the spatial structure of data.
  • 52. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL NEURAL NETWORKS Convolutional Neural Networks (or convnets) are based on the following principles: Local receptive fields Shared weights Pooling (or down-sampling) This special neural network architecture takes advantage of the spatial structure of data. Source: Deeply-Supervised Nets. Zhuowen Tu.
  • 53. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY Let’s take the MNIST digits images as input of our convnet. These images are 28x28 pixels: 28x28 image Local connectivity (5x5) Source: Neural Networks and Deep Learning. Michael Nielsen.
  • 54. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY
  • 55. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY By “sliding” it, we create a feature map of or 24x24 neurons in the hidden layer. We can also have a different stride and padding.
  • 56. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SHARED WEIGHTS In this local receptive field, Convolutional Neural Networks use the same shared weights for each of the 24x24 hidden neurons. This means that we have a great advantage of parameter reduction, for instance, for a 5x5 receptive field, we’ll need only 25 shared weights1. 1 Excluding the bias
  • 57. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SHARED WEIGHTS In this local receptive field, Convolutional Neural Networks use the same shared weights for each of the 24x24 hidden neurons. This means that we have a great advantage of parameter reduction, for instance, for a 5x5 receptive field, we’ll need only 25 shared weights1. 20 feature maps using 5x5 — 20*26 = 520 weights A fully connected first layer, with 784=28*28 input neurons, and a relatively modest 30 hidden neurons, would produce 784*30 = 23.520 weights, more than 40 times as many parameters as the convolutional layer. 1 Excluding the bias
  • 58. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL LAYER The shared weights and bias are called kernel or filter. Convolutional layers provides translation invariance. Since these filters works on every part of the image, they are “searching” for the same feature everywhere in the image.
  • 59. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ARCHITECTURE OVERVIEW Input layer – 28x28 pixels Convolutional layer — 3 feature maps (5x5 kernel)
  • 60. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER Pooling layers are usually present after a convolutional layer. They provide a down-sampling of the convolution output. In the example above, a 2x2 region is being used as input of the pooling.
  • 61. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER There are different types of pooling, the most used is the max-pooling and average pooling: Pooling layers downsamples the volume spatially, reducing small translations of the features. They also provide a parameter reduction. Source: CS231n Convolutional Neural Networks for Visual Recognition.
  • 62. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER Max-pooling is how the network asks whether a feature is found anywhere in some region of the image. After that, it will lose the exact position.
  • 63. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ARCHITECTURE OVERVIEW Input layer – 28x28 pixels Convolutional layer — 3 feature maps (5x5 kernel) Pooling Layer — 2x2, resulting in 3x12x12
  • 64. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CLASSIFICATION As you can see, we then add a dense fully-connected layer (usually using softmax) at the end of the neural network in order to get predictions for the problem we’re working on (10 classes, 10 digits).
  • 65. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER We have defined all the components required to create a Convolutional Neural Network, but you’ll rarely see a shallow convnet like that.
  • 66. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER Actually, experiments demonstrated that the replication of convolutional + pooling layers produces better results the deeper you go. Winners of ImageNet challenge, have more than 15 layers (VGGNet has 19 layers).
  • 67. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al. Galaxy Zoo best performing network (winner of the challenge).
  • 68. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DROPOUT TECHNIQUE Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”. Nitish Srivastava et. al. The dropout technique helps with the overfitting, specially on dense layers. Drop occur only at training time, not on test time.
  • 69. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ACTIVATION FUNCTIONS Source: Big Data Analytics. Fei Wang. ReLu helps with the vanishing gradient problem ReLu generates sparsity
  • 70. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DATA AUGMENTATION Data augmentation can help with overfitting and will certainly improve improve results. Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al.
  • 71. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DATA AUGMENTATION Data augmentation can help with overfitting and will certainly improve improve results. Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al. Small rotations Small translation Scaling Flipping Brightness Noise
  • 72. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRANSFER LEARNING Features learned by Convolutional Neural Networks on large dataset problem (i.e. ImageNet), can be helpful on different problems. It’s very common to pre-train a convnet on ImageNet and then use it as a fixed feature extractor or as initialization. CONVNETS AS FEATURE EXTRACTORS We can remove the last layer and then use these features to extract features, these features are very useful features for classification. Some people use these features with LSH (locality-sensitive hashing) to scale large databases for image search. You can also use these features as input for a SVM classifier for instance.
  • 73. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRANSFER LEARNING Features learned by Convolutional Neural Networks on large dataset problem (i.e. ImageNet), can be helpful on different problems. It’s very common to pre-train a convnet on ImageNet and then use it as a fixed feature extractor or as initialization. FINE-TUNING THE CONVNETS You can use a pre-trained convnet to continue its training on your data and thus fine-tune the weights for your problem. First layers of a convnet contains generic features (i.e. edge detectors, etc.) that should be helpful in many tasks. Deeper layers becomes progressively specific to the details of the classes of the original problem.
  • 74. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section IV INTERESTING CASES
  • 75. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MUSIC RECOMMENDATION Source: Recommending music on Spotify with deep learning. Sander Dieleman. This is an example architecture from Spotify, using Convolutional Neural Network for music recommendation.
  • 76. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MUSIC RECOMMENDATION Learned filters at first convolutional layer. The time axis is horizontal, the frequency axis is vertical (frequency increases from top to bottom)
  • 77. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NATURAL LANGUAGE PROCESSING Source: Text understanding from scratch. Xiang Zhang, Yann LeCun. Deciding if a review posted on Amazon is positive or negative with 96% accuracy, and predict the actual number of stars with 73% accuracy.
  • 78. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov. Word vectors (trained with up to hundreds of billions of words).
  • 79. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov. Word vectors (trained with up to hundreds of billions of words). With nice properties: v(’Paris’) - v(’France’) + v(’Italy’ ) ≈ v(’Rome’) v(’king’) - v(’man’) + v(’woman’) ≈ v(’queen’) No deep learning and no convnet, but a great distributed representation example.
  • 80. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Machine Translation Source: DL4J.
  • 81. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DOC2VEC Sentiment analysis Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov.
  • 82. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A INTERESTING FRAMES Google recently put in production a Deep Neural Network to improve YouTube video thumbnails. Source: Google Research.
  • 83. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SPATIAL TRANSFORMER NETWOKRS Spatial Transformer Networks can learn transformations. Source: Spatial Transformer Networks. Max Jaderberg, et. al.
  • 84. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DEEPFACE BY FACEBOOK Source: DeepFace: Closing the Gap to Human-Level Performance in Face Verification. Yaniv Taigman, et. al.
  • 85. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section V Q&A
  • 86. DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Q&A