SlideShare une entreprise Scribd logo
1  sur  55
Télécharger pour lire hors ligne
Practical Data Science
An Introduction to Supervised Machine Learning
and Pattern Classification: The Big Picture
Michigan State University
NextGen Bioinformatics Seminars - 2015
Sebastian Raschka
Feb. 11, 2015
A Little Bit About Myself ...
Developing software & methods for
- Protein ligand docking
- Large scale drug/inhibitor discovery
PhD candidate in Dr. L. Kuhn’s Lab:
and some other machine learning side-projects …
What is Machine Learning?
http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram
"Field of study that gives computers the
ability to learn without being explicitly
programmed.”
(Arthur Samuel, 1959)
By Phillip Taylor [CC BY 2.0]
http://commons.wikimedia.org/wiki/
File:American_book_company_1916._letter_envelope-2.JPG#filelinks
[public domain]
https://flic.kr/p/5BLW6G [CC BY 2.0]
Text Recognition
Spam Filtering
Biology
Examples of Machine Learning
Examples of Machine Learning
http://googleresearch.blogspot.com/2014/11/a-picture-is-worth-thousand-coherent.html
By Steve Jurvetson [CC BY 2.0]
Self-driving cars
Photo search
and many, many
more ...
Recommendation systems
http://commons.wikimedia.org/wiki/File:Netflix_logo.svg [public domain]
How many of you have used
machine learning before?
Our Agenda
• Concepts and the big picture
• Workflow
• Practical tips & good habits
Learning
• Labeled data
• Direct feedback
• Predict outcome/future
• Decision process
• Reward system
• Learn series of actions
• No labels
• No feedback
• “Find hidden structure”
Unsupervised
Supervised
Reinforcement
Unsupervised
learning
Supervised
learning
Clustering:
[DBSCAN on a toy dataset]
Classification:
[SVM on 2 classes of the Wine dataset]
Regression:
[Soccer Fantasy Score prediction]
Today’s topic
Supervised LearningUnsupervised Learning
Instances (samples, observations)
Features (attributes, dimensions) Classes (targets)
Nomenclature
sepal_length sepal_width petal_length petal_width class
1 5.1 3.5 1.4 0.2 setosa
2 4.9 3.0 1.4 0.2 setosa
… … … … … …
50 6.4 3.2 4.5 1.5 veriscolor
… … … … … …
150 5.9 3.0 5.1 1.8 virginica
https://archive.ics.uci.edu/ml/datasets/Iris
IRIS
Classification
x1
x2
class1
class2
1) Learn from training data
2) Map unseen (new) data
?
Feature Extraction
Feature Selection
Dimensionality Reduction
Feature Scaling
Raw Data Collection
Pre-Processing
Sampling
Test Dataset
Training Dataset
Learning Algorithm
Training
Post-Processing
Cross Validation
Final Classification/
Regression Model
New DataPre-Processing
Refinement
Prediction
Split
Supervised
Learning
Sebastian Raschka 2014
Missing Data
Performance Metrics
Model Selection
Hyperparameter
Optimization
This work is licensed under a Creative Commons Attribution 4.0 International License.
Final Model
Evaluation
Feature Extraction
Feature Selection
Dimensionality Reduction
Feature Scaling
Raw Data Collection
Pre-Processing
Sampling
Test Dataset
Training Dataset
Learning Algorithm
Training
Post-Processing
Cross Validation
Final Classification/
Regression Model
New DataPre-Processing
Refinement
Prediction
Split
Supervised
Learning
Sebastian Raschka 2014
Missing Data
Performance Metrics
Model Selection
Hyperparameter
Optimization
This work is licensed under a Creative Commons Attribution 4.0 International License.
Final Model
Evaluation
A Few Common Classifiers
Decision Tree
Perceptron Naive Bayes
Ensemble Methods: Random Forest, Bagging, AdaBoost
Support Vector Machine
K-Nearest Neighbor
Logistic Regression
Artificial Neural Network / Deep Learning
Discriminative Algorithms
Generative Algorithms
• Models a more general problem: how the data was generated.
• I.e., the distribution of the class; joint probability distribution p(x,y).
• Naive Bayes, Bayesian Belief Network classifier, Restricted
Boltzmann Machine …
• Map x → y directly.
• E.g., distinguish between people speaking different languages
without learning the languages.
• Logistic Regression, SVM, Neural Networks …
Examples of Discriminative Classifiers:
Perceptron
xi1
xi2
w1
w2
Σ yi
y = wTx = w0 + w1x1 + w2x2
1
F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.
x1
x2
y ∈ {-1,1}
w0
wj = weight
xi = training sample
yi = desired output
yi = actual output
t = iteration step
η = learning rate
θ = threshold (here 0)
update rule:
wj(t+1) = wj(t) + η(yi - yi)xi
1 if wTxi ≥ θ
-1 otherwise
^
^
^
^
yi
^
until
t+1 = max iter
or error = 0
Discriminative Classifiers:
Perceptron
F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.
- Binary classifier (one vs all, OVA)
- Convergence problems (set n iterations)
- Modification: stochastic gradient descent
- “Modern” perceptron: Support Vector Machine (maximize margin)
- Multilayer perceptron (MLP)
xi1
xi2
w1
w2
Σ yi
1
y ∈ {-1,1}
w0
^
x1
x2
Generative Classifiers:
Naive Bayes
Bayes Theorem: P(ωj | xi) =
P(xi | ωj) P(ωj)
P(xi)
Posterior probability =
Likelihood x Prior probability
Evidence
Iris example: P(“Setosa"| xi), xi = [4.5 cm, 7.4 cm]
Generative Classifiers:
Naive Bayes
Decision Rule:
Bayes Theorem: P(ωj | xi) =
P(xi | ωj) P(ωj)
P(xi)
pred. class label ωj argmax P(ωj | xi)
i = 1, …, m
e.g., j ∈ {Setosa, Versicolor, Virginica}
Class-conditional
probability
(here Gaussian kernel):
Generative Classifiers:
Naive Bayes
Prior probability:
Evidence:
(cancels out)
(class frequency)
P(ωj | xi) =
P(xi | ωj) P(ωj)
P(xi)
P(ωj) =
Nωj
Nc
P(xik |ωj) =
1
√ (2 π σωj
2)
exp
( )-
(xik - μωj)2
2σωj
2
P(xi |ωj) P(xik |ωj)
Generative Classifiers:
Naive Bayes
- Naive conditional independence assumption typically
violated
- Works well for small datasets
- Multinomial model still quite popular for text classification
(e.g., spam filter)
Non-Parametric Classifiers:
K-Nearest Neighbor
- Simple!
- Lazy learner
- Very susceptible to curse of dimensionality
k=3
e.g., k=1
Iris Example
Setosa Virginica Versicolor
k = 3
mahalanobis dist.
uniform weights
C = 3
depth = 2
Decision Tree
Entropy =
depth = 4
petal length <= 2.45?
petal length <= 4.75?
Setosa
Virginica Versicolor
Yes No
e.g., 2 (- 0.5 log2(0.5)) = 1
∑−pi logk pi
i
Information Gain =
entropy(parent) – [avg entropy(children)]
NoYes
depth = 2
"No Free Lunch" :(
Roughly speaking:
“No one model works best for all possible situations.”
Our model is a simplification of reality
Simplification is based on assumptions (model bias)
Assumptions fail in certain situations
D. H. Wolpert. The supervised learning no-free-lunch theorems. In Soft Computing and Industry, pages 25–42. Springer, 2002.
Which Algorithm?
• What is the size and dimensionality of my training set?
• Is the data linearly separable?
• How much do I care about computational efficiency?
- Model building vs. real-time prediction time
- Eager vs. lazy learning / on-line vs. batch learning
- prediction performance vs. speed
• Do I care about interpretability or should it "just work well?"
• ...
Feature Extraction
Feature Selection
Dimensionality Reduction
Feature Scaling
Raw Data Collection
Pre-Processing
Sampling
Test Dataset
Training Dataset
Learning Algorithm
Training
Post-Processing
Cross Validation
Final Classification/
Regression Model
New DataPre-Processing
Refinement
Prediction
Split
Supervised
Learning
Sebastian Raschka 2014
Missing Data
Performance Metrics
Model Selection
Hyperparameter
Optimization
This work is licensed under a Creative Commons Attribution 4.0 International License.
Final Model
Evaluation
Missing Values:
- Remove features (columns)
- Remove samples (rows)
- Imputation (mean, nearest neighbor, …)
Sampling:
- Random split into training and validation sets
- Typically 60/40, 70/30, 80/20
- Don’t use validation set until the very end! (overfitting)
Feature Scaling:
e.g., standardization:
- Faster convergence (gradient descent)
- Distances on same scale (k-NN with Euclidean distance)
- Mean centering for free
- Normal distributed data
- Numerical stability by avoiding small weights
z =
xik - μk
σk
(use same parameters for the test/new data!)
Categorical Variables
color size prize class
label
0 green M 10.1 class1
1 red L 13.5 class2
2 blue XL 15.3 class1
ordinalnominal
green → (1,0,0)
red → (0,1,0)
blue → (0,0,1)
class
label
color=blue color=green color=red prize size
0 0 0 1 0 10.1 1
1 1 0 0 1 13.5 2
2 0 1 0 0 15.3 3
M → 1
L → 2
XL → 3
Feature Extraction
Feature Selection
Dimensionality Reduction
Feature Scaling
Raw Data Collection
Pre-Processing
Sampling
Test Dataset
Training Dataset
Learning Algorithm
Training
Post-Processing
Cross Validation
Final Classification/
Regression Model
New DataPre-Processing
Refinement
Prediction
Split
Supervised
Learning
Sebastian Raschka 2014
Missing Data
Performance Metrics
Model Selection
Hyperparameter
Optimization
This work is licensed under a Creative Commons Attribution 4.0 International License.
Final Model
Evaluation
Generalization Error and Overfitting
How well does the model perform on unseen data?
Generalization Error and Overfitting
Error Metrics: Confusion Matrix
TP
[Linear SVM on sepal/petal lengths]
TN
FN
FP
here: “setosa” = “positive”
Error Metrics
TP
[Linear SVM on sepal/petal lengths]
TN
FN
FP
here: “setosa” = “positive”
TP + TN
FP +FN +TP +TN
Accuracy =
= 1 - Error
FP
N
TP
P
False Positive Rate =
TP
TP + FP
Precision =
True Positive Rate =
(Recall)
“micro” and “macro”
averaging for multi-class
Receiver Operating Characteristic
(ROC) Curves
Test set
Training dataset Test dataset
Complete dataset
Test set
Test set
Test set
1st iteration calc. error
calc. error
calc. error
calc. error
calculate
avg. error
k-fold cross-validation (k=4):
2nd iteration
3rd iteration
4th iteration
fold 1 fold 2 fold 3 fold 4
Model Selection
k-fold CV and ROC
Feature Selection
- Domain knowledge
- Variance threshold
- Exhaustive search
- Decision trees
- …
IMPORTANT!
(Noise, overfitting, curse of dimensionality, efficiency)
X = [x1, x2, x3, x4]start:
stop:
(if d = k)
X = [x1, x3, x4]
X = [x1, x3]
Simplest example:
Greedy Backward Selection
Dimensionality Reduction
• Transformation onto a new feature subspace
• e.g., Principal Component Analysis (PCA)
• Find directions of maximum variance
• Retain most of the information
0. Standardize data
1. Compute covariance matrix
z =
xik - μk
σik = ∑ (xij - µj) (xik - µk)
σk
1
in -1
σ2
1 σ12 σ13 σ14
σ21 σ2
2 σ23 σ24
σ31 σ32 σ2
3 σ34
σ41 σ42 σ43 σ2
4
∑ =
PCA in 3 Steps
2. Eigendecomposition and sorting eigenvalues
PCA in 3 Steps
X v = λ v
Eigenvectors
[[ 0.52237162 -0.37231836 -0.72101681 0.26199559]
[-0.26335492 -0.92555649 0.24203288 -0.12413481]
[ 0.58125401 -0.02109478 0.14089226 -0.80115427]
[ 0.56561105 -0.06541577 0.6338014 0.52354627]]
Eigenvalues
[ 2.93035378 0.92740362 0.14834223 0.02074601]
(from high to low)
3. Select top k eigenvectors and transform data
PCA in 3 Steps
Eigenvectors
[[ 0.52237162 -0.37231836 -0.72101681 0.26199559]
[-0.26335492 -0.92555649 0.24203288 -0.12413481]
[ 0.58125401 -0.02109478 0.14089226 -0.80115427]
[ 0.56561105 -0.06541577 0.6338014 0.52354627]]
Eigenvalues
[ 2.93035378 0.92740362 0.14834223 0.02074601]
[First 2 PCs of Iris]
Hyperparameter Optimization:
GridSearch in scikit-learn
Non-Linear Problems
- XOR gate
k=11
uniform weights
C=1
C=1000,
gamma=0.1
depth=4
Kernel Trick
Kernel function
Kernel
Map onto high-dimensional space (non-linear combinations)
Kernel Trick
Trick: No explicit dot product!
Radius Basis Function (RBF) Kernel:
Kernel PCA
PC1, linear PCA PC1, kernel PCA
Feature Extraction
Feature Selection
Dimensionality Reduction
Feature Scaling
Raw Data Collection
Pre-Processing
Sampling
Test Dataset
Training Dataset
Learning Algorithm
Training
Post-Processing
Cross Validation
Final Classification/
Regression Model
New DataPre-Processing
Refinement
Prediction
Split
Supervised
Learning
Sebastian Raschka 2014
Missing Data
Performance Metrics
Model Selection
Hyperparameter
Optimization
This work is licensed under a Creative Commons Attribution 4.0 International License.
Final Model
Evaluation
Questions?
mail@sebastianraschka.com
https://github.com/rasbt
@rasbt
Thanks!
Additional Slides
Inspiring Literature
P. N. Klein. Coding the Matrix: Linear
Algebra Through Computer Science
Applications. Newtonian Press, 2013.
R. Schutt and C. O’Neil. Doing Data
Science: Straight Talk from the Frontline.
O’Reilly Media, Inc., 2013.
S. Gutierrez. Data Scientists at Work.
Apress, 2014.
R. O. Duda, P. E. Hart, and D. G. Stork.
Pattern classification. 2nd. Edition. New
York, 2001.
Useful Online Resources
https://www.coursera.org/course/ml
http://stats.stackexchange.com
http://www.kaggle.com
My Favorite Tools
http://stanford.edu/~mwaskom/software/seaborn/
Seaborn
http://www.numpy.org
http://pandas.pydata.org
http://scikit-learn.org/stable/
http://ipython.org/notebook.html
class1
class2
Which one to pick?
class1
class2
The problem of overfitting
Generalization error!

Contenu connexe

Tendances

Support Vector Machine - How Support Vector Machine works | SVM in Machine Le...
Support Vector Machine - How Support Vector Machine works | SVM in Machine Le...Support Vector Machine - How Support Vector Machine works | SVM in Machine Le...
Support Vector Machine - How Support Vector Machine works | SVM in Machine Le...Simplilearn
 
Understanding Bagging and Boosting
Understanding Bagging and BoostingUnderstanding Bagging and Boosting
Understanding Bagging and BoostingMohit Rajput
 
Methods of Optimization in Machine Learning
Methods of Optimization in Machine LearningMethods of Optimization in Machine Learning
Methods of Optimization in Machine LearningKnoldus Inc.
 
Introduction to Machine Learning Classifiers
Introduction to Machine Learning ClassifiersIntroduction to Machine Learning Classifiers
Introduction to Machine Learning ClassifiersFunctional Imperative
 
Back propagation
Back propagationBack propagation
Back propagationNagarajan
 
Deep neural networks
Deep neural networksDeep neural networks
Deep neural networksSi Haem
 
Binary Class and Multi Class Strategies for Machine Learning
Binary Class and Multi Class Strategies for Machine LearningBinary Class and Multi Class Strategies for Machine Learning
Binary Class and Multi Class Strategies for Machine LearningPaxcel Technologies
 
Decision trees in Machine Learning
Decision trees in Machine Learning Decision trees in Machine Learning
Decision trees in Machine Learning Mohammad Junaid Khan
 
Feature Engineering in Machine Learning
Feature Engineering in Machine LearningFeature Engineering in Machine Learning
Feature Engineering in Machine LearningKnoldus Inc.
 
Semi-Supervised Learning
Semi-Supervised LearningSemi-Supervised Learning
Semi-Supervised LearningLukas Tencer
 
Regularization in deep learning
Regularization in deep learningRegularization in deep learning
Regularization in deep learningKien Le
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronMostafa G. M. Mostafa
 
Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep LearningOswald Campesato
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine LearningRahul Jain
 
Machine Learning with Decision trees
Machine Learning with Decision treesMachine Learning with Decision trees
Machine Learning with Decision treesKnoldus Inc.
 

Tendances (20)

Support Vector Machine - How Support Vector Machine works | SVM in Machine Le...
Support Vector Machine - How Support Vector Machine works | SVM in Machine Le...Support Vector Machine - How Support Vector Machine works | SVM in Machine Le...
Support Vector Machine - How Support Vector Machine works | SVM in Machine Le...
 
Understanding Bagging and Boosting
Understanding Bagging and BoostingUnderstanding Bagging and Boosting
Understanding Bagging and Boosting
 
Methods of Optimization in Machine Learning
Methods of Optimization in Machine LearningMethods of Optimization in Machine Learning
Methods of Optimization in Machine Learning
 
Naive bayes
Naive bayesNaive bayes
Naive bayes
 
Support vector machine
Support vector machineSupport vector machine
Support vector machine
 
Introduction to Machine Learning Classifiers
Introduction to Machine Learning ClassifiersIntroduction to Machine Learning Classifiers
Introduction to Machine Learning Classifiers
 
Back propagation
Back propagationBack propagation
Back propagation
 
Deep neural networks
Deep neural networksDeep neural networks
Deep neural networks
 
Binary Class and Multi Class Strategies for Machine Learning
Binary Class and Multi Class Strategies for Machine LearningBinary Class and Multi Class Strategies for Machine Learning
Binary Class and Multi Class Strategies for Machine Learning
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent method
 
Decision trees in Machine Learning
Decision trees in Machine Learning Decision trees in Machine Learning
Decision trees in Machine Learning
 
Feature Engineering in Machine Learning
Feature Engineering in Machine LearningFeature Engineering in Machine Learning
Feature Engineering in Machine Learning
 
Semi-Supervised Learning
Semi-Supervised LearningSemi-Supervised Learning
Semi-Supervised Learning
 
Regularization in deep learning
Regularization in deep learningRegularization in deep learning
Regularization in deep learning
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer Perceptron
 
Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep Learning
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
 
Machine Learning with Decision trees
Machine Learning with Decision treesMachine Learning with Decision trees
Machine Learning with Decision trees
 
K - Nearest neighbor ( KNN )
K - Nearest neighbor  ( KNN )K - Nearest neighbor  ( KNN )
K - Nearest neighbor ( KNN )
 
Perceptron & Neural Networks
Perceptron & Neural NetworksPerceptron & Neural Networks
Perceptron & Neural Networks
 

En vedette

Tutorial on Deep learning and Applications
Tutorial on Deep learning and ApplicationsTutorial on Deep learning and Applications
Tutorial on Deep learning and ApplicationsNhatHai Phan
 
Introduction to Big Data/Machine Learning
Introduction to Big Data/Machine LearningIntroduction to Big Data/Machine Learning
Introduction to Big Data/Machine LearningLars Marius Garshol
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural networkDEEPASHRI HK
 
Hadoop and Machine Learning
Hadoop and Machine LearningHadoop and Machine Learning
Hadoop and Machine Learningjoshwills
 
How to Interview a Data Scientist
How to Interview a Data ScientistHow to Interview a Data Scientist
How to Interview a Data ScientistDaniel Tunkelang
 
Data By The People, For The People
Data By The People, For The PeopleData By The People, For The People
Data By The People, For The PeopleDaniel Tunkelang
 
A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)Prof. Dr. Diego Kuonen
 
Hands-on Deep Learning in Python
Hands-on Deep Learning in PythonHands-on Deep Learning in Python
Hands-on Deep Learning in PythonImry Kissos
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning SystemsXavier Amatriain
 
Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Big Data [sorry] & Data Science: What Does a Data Scientist Do?Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Big Data [sorry] & Data Science: What Does a Data Scientist Do?Data Science London
 
How to Become a Data Scientist
How to Become a Data ScientistHow to Become a Data Scientist
How to Become a Data Scientistryanorban
 
A tutorial on deep learning at icml 2013
A tutorial on deep learning at icml 2013A tutorial on deep learning at icml 2013
A tutorial on deep learning at icml 2013Philip Zheng
 
Introduction to Mahout and Machine Learning
Introduction to Mahout and Machine LearningIntroduction to Mahout and Machine Learning
Introduction to Mahout and Machine LearningVarad Meru
 
Deep Learning for Natural Language Processing
Deep Learning for Natural Language ProcessingDeep Learning for Natural Language Processing
Deep Learning for Natural Language ProcessingDevashish Shanker
 
Machine Learning and Data Mining: 12 Classification Rules
Machine Learning and Data Mining: 12 Classification RulesMachine Learning and Data Mining: 12 Classification Rules
Machine Learning and Data Mining: 12 Classification RulesPier Luca Lanzi
 
Myths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data ScientistsMyths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data ScientistsDavid Pittman
 
Tips for data science competitions
Tips for data science competitionsTips for data science competitions
Tips for data science competitionsOwen Zhang
 
10 R Packages to Win Kaggle Competitions
10 R Packages to Win Kaggle Competitions10 R Packages to Win Kaggle Competitions
10 R Packages to Win Kaggle CompetitionsDataRobot
 
Artificial Intelligence Presentation
Artificial Intelligence PresentationArtificial Intelligence Presentation
Artificial Intelligence Presentationlpaviglianiti
 
Agile Software Development Scrum Vs Lean
Agile Software Development Scrum Vs LeanAgile Software Development Scrum Vs Lean
Agile Software Development Scrum Vs LeanAbdul Wahid
 

En vedette (20)

Tutorial on Deep learning and Applications
Tutorial on Deep learning and ApplicationsTutorial on Deep learning and Applications
Tutorial on Deep learning and Applications
 
Introduction to Big Data/Machine Learning
Introduction to Big Data/Machine LearningIntroduction to Big Data/Machine Learning
Introduction to Big Data/Machine Learning
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural network
 
Hadoop and Machine Learning
Hadoop and Machine LearningHadoop and Machine Learning
Hadoop and Machine Learning
 
How to Interview a Data Scientist
How to Interview a Data ScientistHow to Interview a Data Scientist
How to Interview a Data Scientist
 
Data By The People, For The People
Data By The People, For The PeopleData By The People, For The People
Data By The People, For The People
 
A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)A Statistician's View on Big Data and Data Science (Version 1)
A Statistician's View on Big Data and Data Science (Version 1)
 
Hands-on Deep Learning in Python
Hands-on Deep Learning in PythonHands-on Deep Learning in Python
Hands-on Deep Learning in Python
 
10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems
 
Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Big Data [sorry] & Data Science: What Does a Data Scientist Do?Big Data [sorry] & Data Science: What Does a Data Scientist Do?
Big Data [sorry] & Data Science: What Does a Data Scientist Do?
 
How to Become a Data Scientist
How to Become a Data ScientistHow to Become a Data Scientist
How to Become a Data Scientist
 
A tutorial on deep learning at icml 2013
A tutorial on deep learning at icml 2013A tutorial on deep learning at icml 2013
A tutorial on deep learning at icml 2013
 
Introduction to Mahout and Machine Learning
Introduction to Mahout and Machine LearningIntroduction to Mahout and Machine Learning
Introduction to Mahout and Machine Learning
 
Deep Learning for Natural Language Processing
Deep Learning for Natural Language ProcessingDeep Learning for Natural Language Processing
Deep Learning for Natural Language Processing
 
Machine Learning and Data Mining: 12 Classification Rules
Machine Learning and Data Mining: 12 Classification RulesMachine Learning and Data Mining: 12 Classification Rules
Machine Learning and Data Mining: 12 Classification Rules
 
Myths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data ScientistsMyths and Mathemagical Superpowers of Data Scientists
Myths and Mathemagical Superpowers of Data Scientists
 
Tips for data science competitions
Tips for data science competitionsTips for data science competitions
Tips for data science competitions
 
10 R Packages to Win Kaggle Competitions
10 R Packages to Win Kaggle Competitions10 R Packages to Win Kaggle Competitions
10 R Packages to Win Kaggle Competitions
 
Artificial Intelligence Presentation
Artificial Intelligence PresentationArtificial Intelligence Presentation
Artificial Intelligence Presentation
 
Agile Software Development Scrum Vs Lean
Agile Software Development Scrum Vs LeanAgile Software Development Scrum Vs Lean
Agile Software Development Scrum Vs Lean
 

Similaire à An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture

Introduction to conventional machine learning techniques
Introduction to conventional machine learning techniquesIntroduction to conventional machine learning techniques
Introduction to conventional machine learning techniquesXavier Rafael Palou
 
Machine Learning ebook.pdf
Machine Learning ebook.pdfMachine Learning ebook.pdf
Machine Learning ebook.pdfHODIT12
 
Lecture 02: Machine Learning for Language Technology - Decision Trees and Nea...
Lecture 02: Machine Learning for Language Technology - Decision Trees and Nea...Lecture 02: Machine Learning for Language Technology - Decision Trees and Nea...
Lecture 02: Machine Learning for Language Technology - Decision Trees and Nea...Marina Santini
 
Data mining classifiers.
Data mining classifiers.Data mining classifiers.
Data mining classifiers.ShwetaPatil174
 
Lecture 8: Decision Trees & k-Nearest Neighbors
Lecture 8: Decision Trees & k-Nearest NeighborsLecture 8: Decision Trees & k-Nearest Neighbors
Lecture 8: Decision Trees & k-Nearest NeighborsMarina Santini
 
. An introduction to machine learning and probabilistic ...
. An introduction to machine learning and probabilistic .... An introduction to machine learning and probabilistic ...
. An introduction to machine learning and probabilistic ...butest
 
Presentation on Text Classification
Presentation on Text ClassificationPresentation on Text Classification
Presentation on Text ClassificationSai Srinivas Kotni
 
Machine Learning and Real-World Applications
Machine Learning and Real-World ApplicationsMachine Learning and Real-World Applications
Machine Learning and Real-World ApplicationsMachinePulse
 
Large Scale Data Clustering: an overview
Large Scale Data Clustering: an overviewLarge Scale Data Clustering: an overview
Large Scale Data Clustering: an overviewVahid Mirjalili
 
Machine learning for_finance
Machine learning for_financeMachine learning for_finance
Machine learning for_financeStefan Duprey
 
Chris Dyer - 2017 - Neural MT Workshop Invited Talk: The Neural Noisy Channel...
Chris Dyer - 2017 - Neural MT Workshop Invited Talk: The Neural Noisy Channel...Chris Dyer - 2017 - Neural MT Workshop Invited Talk: The Neural Noisy Channel...
Chris Dyer - 2017 - Neural MT Workshop Invited Talk: The Neural Noisy Channel...Association for Computational Linguistics
 
Machine Learning: Foundations Course Number 0368403401
Machine Learning: Foundations Course Number 0368403401Machine Learning: Foundations Course Number 0368403401
Machine Learning: Foundations Course Number 0368403401butest
 
Machine Learning ICS 273A
Machine Learning ICS 273AMachine Learning ICS 273A
Machine Learning ICS 273Abutest
 
MS CS - Selecting Machine Learning Algorithm
MS CS - Selecting Machine Learning AlgorithmMS CS - Selecting Machine Learning Algorithm
MS CS - Selecting Machine Learning AlgorithmKaniska Mandal
 
Free Ebooks Download ! Edhole.com
Free Ebooks Download ! Edhole.comFree Ebooks Download ! Edhole.com
Free Ebooks Download ! Edhole.comEdhole.com
 
Machine Learning and Statistical Analysis
Machine Learning and Statistical AnalysisMachine Learning and Statistical Analysis
Machine Learning and Statistical Analysisbutest
 
Machine Learning and Statistical Analysis
Machine Learning and Statistical AnalysisMachine Learning and Statistical Analysis
Machine Learning and Statistical Analysisbutest
 
Machine Learning and Statistical Analysis
Machine Learning and Statistical AnalysisMachine Learning and Statistical Analysis
Machine Learning and Statistical Analysisbutest
 

Similaire à An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture (20)

Introduction to conventional machine learning techniques
Introduction to conventional machine learning techniquesIntroduction to conventional machine learning techniques
Introduction to conventional machine learning techniques
 
Machine Learning ebook.pdf
Machine Learning ebook.pdfMachine Learning ebook.pdf
Machine Learning ebook.pdf
 
Lecture 02: Machine Learning for Language Technology - Decision Trees and Nea...
Lecture 02: Machine Learning for Language Technology - Decision Trees and Nea...Lecture 02: Machine Learning for Language Technology - Decision Trees and Nea...
Lecture 02: Machine Learning for Language Technology - Decision Trees and Nea...
 
Data mining classifiers.
Data mining classifiers.Data mining classifiers.
Data mining classifiers.
 
Clustering
ClusteringClustering
Clustering
 
Lecture 8: Decision Trees & k-Nearest Neighbors
Lecture 8: Decision Trees & k-Nearest NeighborsLecture 8: Decision Trees & k-Nearest Neighbors
Lecture 8: Decision Trees & k-Nearest Neighbors
 
. An introduction to machine learning and probabilistic ...
. An introduction to machine learning and probabilistic .... An introduction to machine learning and probabilistic ...
. An introduction to machine learning and probabilistic ...
 
Presentation on Text Classification
Presentation on Text ClassificationPresentation on Text Classification
Presentation on Text Classification
 
Machine Learning and Real-World Applications
Machine Learning and Real-World ApplicationsMachine Learning and Real-World Applications
Machine Learning and Real-World Applications
 
Large Scale Data Clustering: an overview
Large Scale Data Clustering: an overviewLarge Scale Data Clustering: an overview
Large Scale Data Clustering: an overview
 
Machine learning for_finance
Machine learning for_financeMachine learning for_finance
Machine learning for_finance
 
Chris Dyer - 2017 - Neural MT Workshop Invited Talk: The Neural Noisy Channel...
Chris Dyer - 2017 - Neural MT Workshop Invited Talk: The Neural Noisy Channel...Chris Dyer - 2017 - Neural MT Workshop Invited Talk: The Neural Noisy Channel...
Chris Dyer - 2017 - Neural MT Workshop Invited Talk: The Neural Noisy Channel...
 
Machine Learning: Foundations Course Number 0368403401
Machine Learning: Foundations Course Number 0368403401Machine Learning: Foundations Course Number 0368403401
Machine Learning: Foundations Course Number 0368403401
 
Machine Learning ICS 273A
Machine Learning ICS 273AMachine Learning ICS 273A
Machine Learning ICS 273A
 
MS CS - Selecting Machine Learning Algorithm
MS CS - Selecting Machine Learning AlgorithmMS CS - Selecting Machine Learning Algorithm
MS CS - Selecting Machine Learning Algorithm
 
My7class
My7classMy7class
My7class
 
Free Ebooks Download ! Edhole.com
Free Ebooks Download ! Edhole.comFree Ebooks Download ! Edhole.com
Free Ebooks Download ! Edhole.com
 
Machine Learning and Statistical Analysis
Machine Learning and Statistical AnalysisMachine Learning and Statistical Analysis
Machine Learning and Statistical Analysis
 
Machine Learning and Statistical Analysis
Machine Learning and Statistical AnalysisMachine Learning and Statistical Analysis
Machine Learning and Statistical Analysis
 
Machine Learning and Statistical Analysis
Machine Learning and Statistical AnalysisMachine Learning and Statistical Analysis
Machine Learning and Statistical Analysis
 

Dernier

The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 

Dernier (17)

The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 

An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture

  • 1. Practical Data Science An Introduction to Supervised Machine Learning and Pattern Classification: The Big Picture Michigan State University NextGen Bioinformatics Seminars - 2015 Sebastian Raschka Feb. 11, 2015
  • 2. A Little Bit About Myself ... Developing software & methods for - Protein ligand docking - Large scale drug/inhibitor discovery PhD candidate in Dr. L. Kuhn’s Lab: and some other machine learning side-projects …
  • 3. What is Machine Learning? http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram "Field of study that gives computers the ability to learn without being explicitly programmed.” (Arthur Samuel, 1959) By Phillip Taylor [CC BY 2.0]
  • 5. Examples of Machine Learning http://googleresearch.blogspot.com/2014/11/a-picture-is-worth-thousand-coherent.html By Steve Jurvetson [CC BY 2.0] Self-driving cars Photo search and many, many more ... Recommendation systems http://commons.wikimedia.org/wiki/File:Netflix_logo.svg [public domain]
  • 6. How many of you have used machine learning before?
  • 7. Our Agenda • Concepts and the big picture • Workflow • Practical tips & good habits
  • 8. Learning • Labeled data • Direct feedback • Predict outcome/future • Decision process • Reward system • Learn series of actions • No labels • No feedback • “Find hidden structure” Unsupervised Supervised Reinforcement
  • 9. Unsupervised learning Supervised learning Clustering: [DBSCAN on a toy dataset] Classification: [SVM on 2 classes of the Wine dataset] Regression: [Soccer Fantasy Score prediction] Today’s topic Supervised LearningUnsupervised Learning
  • 10. Instances (samples, observations) Features (attributes, dimensions) Classes (targets) Nomenclature sepal_length sepal_width petal_length petal_width class 1 5.1 3.5 1.4 0.2 setosa 2 4.9 3.0 1.4 0.2 setosa … … … … … … 50 6.4 3.2 4.5 1.5 veriscolor … … … … … … 150 5.9 3.0 5.1 1.8 virginica https://archive.ics.uci.edu/ml/datasets/Iris IRIS
  • 11. Classification x1 x2 class1 class2 1) Learn from training data 2) Map unseen (new) data ?
  • 12. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 13. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 14. A Few Common Classifiers Decision Tree Perceptron Naive Bayes Ensemble Methods: Random Forest, Bagging, AdaBoost Support Vector Machine K-Nearest Neighbor Logistic Regression Artificial Neural Network / Deep Learning
  • 15. Discriminative Algorithms Generative Algorithms • Models a more general problem: how the data was generated. • I.e., the distribution of the class; joint probability distribution p(x,y). • Naive Bayes, Bayesian Belief Network classifier, Restricted Boltzmann Machine … • Map x → y directly. • E.g., distinguish between people speaking different languages without learning the languages. • Logistic Regression, SVM, Neural Networks …
  • 16. Examples of Discriminative Classifiers: Perceptron xi1 xi2 w1 w2 Σ yi y = wTx = w0 + w1x1 + w2x2 1 F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. x1 x2 y ∈ {-1,1} w0 wj = weight xi = training sample yi = desired output yi = actual output t = iteration step η = learning rate θ = threshold (here 0) update rule: wj(t+1) = wj(t) + η(yi - yi)xi 1 if wTxi ≥ θ -1 otherwise ^ ^ ^ ^ yi ^ until t+1 = max iter or error = 0
  • 17. Discriminative Classifiers: Perceptron F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. - Binary classifier (one vs all, OVA) - Convergence problems (set n iterations) - Modification: stochastic gradient descent - “Modern” perceptron: Support Vector Machine (maximize margin) - Multilayer perceptron (MLP) xi1 xi2 w1 w2 Σ yi 1 y ∈ {-1,1} w0 ^ x1 x2
  • 18. Generative Classifiers: Naive Bayes Bayes Theorem: P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) Posterior probability = Likelihood x Prior probability Evidence Iris example: P(“Setosa"| xi), xi = [4.5 cm, 7.4 cm]
  • 19. Generative Classifiers: Naive Bayes Decision Rule: Bayes Theorem: P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) pred. class label ωj argmax P(ωj | xi) i = 1, …, m e.g., j ∈ {Setosa, Versicolor, Virginica}
  • 20. Class-conditional probability (here Gaussian kernel): Generative Classifiers: Naive Bayes Prior probability: Evidence: (cancels out) (class frequency) P(ωj | xi) = P(xi | ωj) P(ωj) P(xi) P(ωj) = Nωj Nc P(xik |ωj) = 1 √ (2 π σωj 2) exp ( )- (xik - μωj)2 2σωj 2 P(xi |ωj) P(xik |ωj)
  • 21. Generative Classifiers: Naive Bayes - Naive conditional independence assumption typically violated - Works well for small datasets - Multinomial model still quite popular for text classification (e.g., spam filter)
  • 22. Non-Parametric Classifiers: K-Nearest Neighbor - Simple! - Lazy learner - Very susceptible to curse of dimensionality k=3 e.g., k=1
  • 23. Iris Example Setosa Virginica Versicolor k = 3 mahalanobis dist. uniform weights C = 3 depth = 2
  • 24. Decision Tree Entropy = depth = 4 petal length <= 2.45? petal length <= 4.75? Setosa Virginica Versicolor Yes No e.g., 2 (- 0.5 log2(0.5)) = 1 ∑−pi logk pi i Information Gain = entropy(parent) – [avg entropy(children)] NoYes depth = 2
  • 25. "No Free Lunch" :( Roughly speaking: “No one model works best for all possible situations.” Our model is a simplification of reality Simplification is based on assumptions (model bias) Assumptions fail in certain situations D. H. Wolpert. The supervised learning no-free-lunch theorems. In Soft Computing and Industry, pages 25–42. Springer, 2002.
  • 26. Which Algorithm? • What is the size and dimensionality of my training set? • Is the data linearly separable? • How much do I care about computational efficiency? - Model building vs. real-time prediction time - Eager vs. lazy learning / on-line vs. batch learning - prediction performance vs. speed • Do I care about interpretability or should it "just work well?" • ...
  • 27. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 28. Missing Values: - Remove features (columns) - Remove samples (rows) - Imputation (mean, nearest neighbor, …) Sampling: - Random split into training and validation sets - Typically 60/40, 70/30, 80/20 - Don’t use validation set until the very end! (overfitting) Feature Scaling: e.g., standardization: - Faster convergence (gradient descent) - Distances on same scale (k-NN with Euclidean distance) - Mean centering for free - Normal distributed data - Numerical stability by avoiding small weights z = xik - μk σk (use same parameters for the test/new data!)
  • 29. Categorical Variables color size prize class label 0 green M 10.1 class1 1 red L 13.5 class2 2 blue XL 15.3 class1 ordinalnominal green → (1,0,0) red → (0,1,0) blue → (0,0,1) class label color=blue color=green color=red prize size 0 0 0 1 0 10.1 1 1 1 0 0 1 13.5 2 2 0 1 0 0 15.3 3 M → 1 L → 2 XL → 3
  • 30. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 31. Generalization Error and Overfitting How well does the model perform on unseen data?
  • 32. Generalization Error and Overfitting
  • 33. Error Metrics: Confusion Matrix TP [Linear SVM on sepal/petal lengths] TN FN FP here: “setosa” = “positive”
  • 34. Error Metrics TP [Linear SVM on sepal/petal lengths] TN FN FP here: “setosa” = “positive” TP + TN FP +FN +TP +TN Accuracy = = 1 - Error FP N TP P False Positive Rate = TP TP + FP Precision = True Positive Rate = (Recall) “micro” and “macro” averaging for multi-class
  • 36. Test set Training dataset Test dataset Complete dataset Test set Test set Test set 1st iteration calc. error calc. error calc. error calc. error calculate avg. error k-fold cross-validation (k=4): 2nd iteration 3rd iteration 4th iteration fold 1 fold 2 fold 3 fold 4 Model Selection
  • 38. Feature Selection - Domain knowledge - Variance threshold - Exhaustive search - Decision trees - … IMPORTANT! (Noise, overfitting, curse of dimensionality, efficiency) X = [x1, x2, x3, x4]start: stop: (if d = k) X = [x1, x3, x4] X = [x1, x3] Simplest example: Greedy Backward Selection
  • 39. Dimensionality Reduction • Transformation onto a new feature subspace • e.g., Principal Component Analysis (PCA) • Find directions of maximum variance • Retain most of the information
  • 40. 0. Standardize data 1. Compute covariance matrix z = xik - μk σik = ∑ (xij - µj) (xik - µk) σk 1 in -1 σ2 1 σ12 σ13 σ14 σ21 σ2 2 σ23 σ24 σ31 σ32 σ2 3 σ34 σ41 σ42 σ43 σ2 4 ∑ = PCA in 3 Steps
  • 41. 2. Eigendecomposition and sorting eigenvalues PCA in 3 Steps X v = λ v Eigenvectors [[ 0.52237162 -0.37231836 -0.72101681 0.26199559] [-0.26335492 -0.92555649 0.24203288 -0.12413481] [ 0.58125401 -0.02109478 0.14089226 -0.80115427] [ 0.56561105 -0.06541577 0.6338014 0.52354627]] Eigenvalues [ 2.93035378 0.92740362 0.14834223 0.02074601] (from high to low)
  • 42. 3. Select top k eigenvectors and transform data PCA in 3 Steps Eigenvectors [[ 0.52237162 -0.37231836 -0.72101681 0.26199559] [-0.26335492 -0.92555649 0.24203288 -0.12413481] [ 0.58125401 -0.02109478 0.14089226 -0.80115427] [ 0.56561105 -0.06541577 0.6338014 0.52354627]] Eigenvalues [ 2.93035378 0.92740362 0.14834223 0.02074601] [First 2 PCs of Iris]
  • 44. Non-Linear Problems - XOR gate k=11 uniform weights C=1 C=1000, gamma=0.1 depth=4
  • 45. Kernel Trick Kernel function Kernel Map onto high-dimensional space (non-linear combinations)
  • 46. Kernel Trick Trick: No explicit dot product! Radius Basis Function (RBF) Kernel:
  • 47. Kernel PCA PC1, linear PCA PC1, kernel PCA
  • 48. Feature Extraction Feature Selection Dimensionality Reduction Feature Scaling Raw Data Collection Pre-Processing Sampling Test Dataset Training Dataset Learning Algorithm Training Post-Processing Cross Validation Final Classification/ Regression Model New DataPre-Processing Refinement Prediction Split Supervised Learning Sebastian Raschka 2014 Missing Data Performance Metrics Model Selection Hyperparameter Optimization This work is licensed under a Creative Commons Attribution 4.0 International License. Final Model Evaluation
  • 51. Inspiring Literature P. N. Klein. Coding the Matrix: Linear Algebra Through Computer Science Applications. Newtonian Press, 2013. R. Schutt and C. O’Neil. Doing Data Science: Straight Talk from the Frontline. O’Reilly Media, Inc., 2013. S. Gutierrez. Data Scientists at Work. Apress, 2014. R. O. Duda, P. E. Hart, and D. G. Stork. Pattern classification. 2nd. Edition. New York, 2001.
  • 55. class1 class2 The problem of overfitting Generalization error!