Machine Learning

1. Machine Learning
Algorithms
Girish Khanzode

2. Contents
• Supervised Learning Model
• Linear Regression
• KNN
• DecisionTree Learning
• OptimizedTree Induction
• Random Forest
• Logistic Regression
• SVM
• Naive Bayes Classifier
• Clustering
• K-means Clustering
• Cluster Classification
• AlgorithmicComplexity
• Dimensionality Reduction
• PCA
• Fisher Linear Discriminant
• LDA
• Kernel PCA
• SVD
• HMM
• Model Evaluation
• Confusion Matrix
• K-Fold CrossValidation
• References

3. Machine Learning and Pattern Classification
• Predictive modelling is building a model capable of making predictions
• Such a model includes a machine learning algorithm that learns certain properties from a
training dataset in order to make those predictions
• Predictive modelling types - Regression and pattern classification
• Regression models analyze relationships between variables and trends in order to make
predictions about continuous variables
– Prediction of the maximum temperature for the upcoming days in weather forecasting
• Pattern classification assigns discrete class labels to particular observations as outcomes of
a prediction
– Prediction of a sunny, rainy or snowy day

4. Machine Learning Methodologies
• Supervised learning
– Learning from labelled data
– Classification, Regression, Prediction, Function Approximation
• Unsupervised learning
– Learning from unlabelled data
– Clustering,Visualization, Dimensionality Reduction

5. Machine Learning Methodologies
• Semi-supervised learning
– mix of Supervised and Unsupervised learning
– usually small part of data is labelled
• Reinforcement learning
– Model learns from a series of actions by maximizing a reward function
– The reward function can either be maximized by penalizing bad actions
and/or rewarding good actions
– Example - training of self-driving car using feedback from the environment

6. Applications
• Speech recognition
• Effective web search
• Recommendation systems
• Computer vision
• Information retrieval
• Spam filtering
• Computational finance
• Fraud detection
• Medical diagnosis
• Stock market analysis
• Structural health monitoring

7. LearningTypes

8. Machine Learning Algorithms

9. Learning Process
• Supervised LearningAlgorithms are used in classification and prediction
• Training set - each record contains a set of attributes, one of the
attributes is the class
• Classification or prediction algorithm learns from training data about
relationship between predictor variables and outcome variable
• This process results in
– Classification model
– Predictive model

10. Learning Process

11. Typical Steps in ML

12. Supervised Learning Model
• The class labels in the dataset used to build the classification model are
known
• Example - a dataset for spam filtering would contain spam messages as
well as "ham" (= not-spam) messages
• In a supervised learning problem, it is known which message in the
training set is spam or ham and this information is used to train our model
in order to classify new unseen messages

13. Supervised Learning Model

14. Classification and Regression

15. Linear Regression
• A standard and simple mathematical technique for predicting numeric outcome
• Oldest and most widely used predictive model
• Goal - minimize the sum of the squared errors to fit a straight line to a set of data points
• Fits a linear function to a set of data points
• Form of the function
– Y = β0 + β1*X1 + β2*X2 + … + βn*Xn
– Y is the target variable and X1, X2, ... Xn are the predictor variables
– β1, β2, … βn are the coefficients that multiply the predictor variables
– β0 is constant
• Linear regression with multiple variables
– Scale the data, and implement the gradient descent and the cost function

16. Linear Regression

17. K Nearest Neighbors - KNN
• A simple algorithm that stores all available cases and classifies new cases based on a
similarity measure
• Extremely simple to implement
• Lazy Learning - function is only approximated locally and all computation is deferred until
classification
• Has a weighted version and can also be used for regression
• Usually works very well when there is a distance between examples (Euclidean, Manhattan)
• Slow speed when training set is large (say 10^6 examples) and distance calculation is non-
trivial
• Only a single hyper-parameter – K (usually optimized using cross-validation)

18. KNN

19. KNN Classification
Non-Default
Default
Age
Loan$

20. DecisionTree Learning
• Decision trees classify instances or examples by starting at the root of the
tree and moving through it until a leaf node
• A method for approximating discrete-valued functions
• Decision tree is a classifier in the form of a tree structure
– Decision node - specifies a test on a single attribute
– Leaf node - indicates the value of the target attribute
– Branch - split of one attribute
– Path - a disjunction of test to make the final decision

21. When to Consider DecisionTrees
• Attribute-value description- object or case must be expressible in terms
of a fixed collection of properties or attributes
– hot, mild, cold
• Predefined classes (target values) - the target function has discrete
output values
– Boolean or multiclass
– Sufficient data - enough training cases should be provided to learn the model
• Possibly noisy training data
• Missing attribute values

22. DecisionTree Applications
• Credit risk analysis
• Manufacturing – chemical material evaluation
• Production – Process optimization
• Biomedical Engineering – identify features to use in implantable devices
• Astronomy – filter noise from Hubble telescope images
• Molecular biology – analyze amino acid sequences in Human Genome project
• Pharmacology – drug efficacy analysis
• Planning – scheduling of PCB assembly lines
• Medicine – analysis of syndromes

23. Strengths
• Trees are inexpensive to construct
• Extremely fast at classifying unknown records
• Easy to interpret for small-sized trees
• Accuracy is comparable to other classification techniques for many simple
data sets
• Generates understandable rules
• Handles continuous and categorical variables
• Provides a clear indication of which fields are most important for
prediction or classification

24. Weaknesses
• Not suitable for prediction of continuous attribute
• Perform poorly with many classes and small data
• Computationally expensive to train
– At each node each candidate splitting field must be sorted before its best split can
be found
– In some algorithms combinations of fields are used and a search must be made for
optimal combining weights
– Pruning algorithms can also be expensive since many candidate sub-trees must be
formed and compared
• Not suitable for non-rectangular regions

25. Tree Representation
• Each node in the tree specifies a test for some
attribute of the instance
• Each branch corresponds to an attribute value
• Each leaf node assigns a classification

26. Tree Representation

27. Tree Induction

28. Problems of Random split
• The tree can grow huge
• These trees are hard to understand
• Larger trees are typically less accurate than smaller trees
• So most tree construction methods use some greedy manner
– find the feature that best divides positive examples from negative
examples for Information gain

29. OptimizedTree Induction
• Greedy strategy - Split the records based on an attribute test
that optimizes certain criterion
• Issues
– Determine root node
– Determine how to split the records
• How to specify the attribute test condition?
• How to determine the best split?
– Determine when to stop splitting

30. OptimizedTree Induction
• Selection of an attribute at each node
– Choose the most useful attribute for classifying training examples
• Information gain
– Measures how well a given attribute separates the training examples
according to their target classification
– This measure is used to select among the candidate attributes at each
step while growing the tree

31. Entropy
• A measure of homogeneity of the set of examples
• Given a set S of positive and negative examples of some target
concept (a 2-class problem), the entropy of set S relative to this
binary classification
– E(S) = - p(P)log2 p(P) – p(N)log2 p(N)
• Example
– Suppose S has 25 examples, 15 positive and 10 negatives [15+, 10-]
– Then entropy of S relative to this classification
• E(S)=-(15/25) log2(15/25) - (10/25) log2 (10/25)

32. Entropy

33. Information Gain
• Information gain measures the expected reduction in entropy or uncertainty
• Values(A) is the set of all possible values for attributeA andSv is subset of S for which attributeA has
value v, Sv = {s in S |A(s) = v}
• First term in the equation is the entropy of the original collection S
• Second term is the expected value of the entropy after S is partitioned using attributeA
• It is the expected reduction in entropy caused by partitioning the examples according to this attribute
• It is the number of bits saved when encoding the target value of an arbitrary member of S by knowing the
value of attributeA
( )
( , ) ( ) ( )v
v
v Values A
S
Gain S A Entropy S Entropy S
S
  

34. A simple example
• Guess the outcome of next week's game between the MallRats and the
Chinooks
• Available knowledge / Attribute
– was the game at Home or Away
– was the starting time 5pm, 7pm or 9pm
– Did Joe play center, or forward
– whether that opponent's center was tall or not
– …..

35. Basket ball data

36. Problem Data
• The game will be away at 9pm and that Joe will play center on offense…
• A classification problem
• Generalizing the learned rule to new examples

37. Examples
• Before partitioning, the entropy is
– H(10/20, 10/20) = - 10/20 log(10/20) - 10/20 log(10/20) = 1
• Using the where attribute, divide into 2 subsets
– Entropy of the first set H(home) = - 6/12 log(6/12) - 6/12 log(6/12) = 1
– Entropy of the second set H(away) = - 4/8 log(6/8) - 4/8 log(4/8) = 1
• Expected entropy after partitioning
– 12/20 * H(home) + 8/20 * H(away) = 1

38. Examples
• Using the when attribute, divide into 3 subsets
– Entropy of the first set H(5pm) = - 1/4 log(1/4) - 3/4 log(3/4);
– Entropy of the second set H(7pm) = - 9/12 log(9/12) - 3/12 log(3/12);
– Entropy of the second set H(9pm) = - 0/4 log(0/4) - 4/4 log(4/4) = 0
• Expected entropy after partitioning
– 4/20 * H(1/4, 3/4) + 12/20 * H(9/12, 3/12) + 4/20 * H(0/4, 4/4) = 0.65
• Information gain 1-0.65 = 0.35

39. Decision
• Knowing the when attribute values provides larger information gain than where
• Therefore the when attribute should be chosen for testing prior to the where
attribute
• Similarly we can compute the information gain for other attributes
• At each node choose the attribute with the largest information gain
• Stopping rule
– Every attribute has already been included along this path through the tree or
– The training examples associated with this leaf node all have the same target attribute
value - entropy is zero

40. Continuous Attribute
• Each non-leaf node is a test
• Its edge partitions the attribute into subsets (easy for discrete attribute)
• For continuous attribute
– Partition the continuous value of attribute A into a discrete set of intervals
– Create a new Boolean attribute Ac, looking for a threshold c,
How to choose c ?
if
otherwise
c
c
true A c
A
false

 


41. Evaluation
• Training accuracy
– How many training instances can be correctly classify based on the available data?
– Is it high when tree is deep/large or when there is less confliction in the training
instances
– Higher value does not mean good generalization
• Testing accuracy
– Given a number of new instances how many of them can be correctly classified?
– Cross validation

42. DecisionTree Creation Algorithms
• ID3
• C4.5
• Hunt’s Algorithm
• CART
• SLIQ,SPRINT

43. Random Forest
• An ensemble classifier that consists of many decision trees
• Outputs the class that is the mode of the class's output by
individual trees
• The method combines Breiman's bagging idea and the
random selection of features
• Used for classification and regression

44. Random Forest

45. Algorithm
• Let the number of training cases be N and number of variables in the classifier M
• The number m of input variables to be used to determine the decision at a node of the tree -
m should be much less than M
• Choose a training set for this tree by choosing n times with replacement from all N available
training cases
• Use the rest of cases to estimate the error of the tree by predicting their classes
• For each node of the tree, randomly choose m variables on which to base the decision at
that node
• Calculate the best split based on these m variables in the training set
• Each tree is fully grown and not pruned

46. Gini Index
• Random forest uses Gini index taken from CART learning system to
construct decision trees
• The Gini Index of node impurity is the measure most commonly chosen
for classification type problems
• How to select N? - Build trees until the error no longer decreases
• How to select M? -Try to recommend defaults, half of them and twice of
them and pick the best

47. Random Forest - Flow Chart

48. Working of Random Forest
• For prediction, a new sample is pushed down the tree
• It is assigned the label of the training sample in the terminal node it ends
up in
• This procedure is iterated over all trees in the ensemble
• Average vote of all trees is reported as random forest prediction

49. Random Forest - Advantages
• One of the most accurate learning algorithms
• Produces a highly accurate classifier
• Runs efficiently on large databases
• Handles thousands of input variables without variable deletion
• Gives estimates of what variables are important in classification
• Generates an internal unbiased estimate of the generalization error as
the forest building progresses
• Effective method for estimating missing data and maintains accuracy
when a large proportion of the data are missing

50. Random Forest - Advantages
• Methods for balancing error in class population unbalanced data sets
• Prototypes are computed that give information about the relation
between the variables and the classification
• Computes proximities between pairs of cases that can be used in
clustering, locating outliers or by scaling gives interesting views of data
• Above capabilities can be extended to unlabeled data, leading to
unsupervised clustering, data views and outlier detection
• Offers an experimental method for detecting variable interactions

51. Random Forest - Disadvantages
• Random forests have been observed to overfit for some datasets with
noisy classification/regression tasks
• For data including categorical variables with different number of levels,
random forests are biased in favor of those attributes with more levels
• Therefore the variable importance scores from random forest are not
reliable for this type of data

52. Logistic Regression
• Models the relationship between a dependent and one or more independent variables
• Allows to look at the fit of the model as well as at the significance of the relationships
(between dependent and independent variables) being modelled
• Estimates the probability of an event occurring - the probability of a pupil continuing in
education post 16
• Predict from a knowledge of relevant independent variables the probability (p) that it is 1
(event occurring) rather than 0
• While in linear regression the relationship between the dependent and the independent
variables is linear, this assumption is not made in logistic regression

53. Logistic Regression
• Logistic regression function $$ P =
frac{e^{alpha+{beta}x}}{1+e^{alpha+{beta}x}} $$
• P is the probability of a 1 and e is base of natural logarithm (about 2.718)
• $$alpha$$ and $$beta$$ are the parameters of the model
• The value of $$alpha$$ yields P when x is zero and $$beta$$ indicates how the
probability of a 1 changes when x changes by a single unit
• Because the relation between x and P is nonlinear, $$beta$$ does not have as
straightforward an interpretation in this model as it does in ordinary linear
regression

54. Logistic Regression

55. SupportVector Machine - SVM
• A supervised learning model with associated learning algorithms that analyze
data and recognize patterns
• Given a set of training examples, each marked for belonging to one of two
categories, SVM training algorithm builds a model that assigns new examples into
one category or the other, making it a non-probabilistic binary linear classifier
• An SVM model is a representation of the examples as points in space, mapped so
that the examples of the separate categories are divided by a clear gap that is as
wide as possible
• New examples are then mapped into that same space and predicted to belong to
a category based on which side of the gap they fall on

56. SVM

57. Naive Bayes Classifier
• A family of simple probabilistic classifiers based on applying Bayes'
theorem with strong (naive) independence assumptions between the
features
• A popular method for text categorization, the problem of judging
documents as belonging to one category or the other such as spam or
legitimate, sports or politics etc with word frequencies as the features
• Highly scalable, requires a number of parameters linear in the number of
variables (features/predictors) in a learning problem

58. Conditional Probability Model

59. Naive Bayes - Example

60. UNSUPERVISED LEARNING

61. Clustering
• A technique to find similar groups in data clusters
• Groups data instances that are similar to (near) each other in one cluster and data
instances that are very different (far away) from each other into different clusters
• Called an unsupervised learning task - since no class values denoting an a priori
grouping of the data instances are given, which is the case in supervised learning
• One of the most utilized data mining techniques
• A long history and used in almost every field like medicine, psychology, botany,
sociology, biology, archeology, marketing, insurance, libraries and text clustering

62. Clustering

63. Applications
• Group people of similar sizes together to make small, medium and largeT-Shirts
– Tailor-made for each person - too expensive
– One-size-fits-all - does not fit all
• In marketing, segment customers according to their similarities
– Targeted marketing
• Given a collection of text documents, organize them according to their content
similarities
– To produce a topic hierarchy

64. Aspects of clustering
• Clustering algorithms
– Partitional clustering
– Hierarchical clustering
• A distance function - similarity or dissimilarity
• Clustering quality
– Inter-clusters distance  maximized
– Intra-clusters distance  minimized
• Quality of a clustering process depends on algorithm, distance function and
application

65. K-means Clustering
• A partitional clustering algorithm
• Classify a given data set through a certain number of k clusters (k is fixed)
• Let the set of data points D be {x1, x2, …, xn}
– xi = (xi1, xi2, …, xir) is a vector in a real-valued space X  Rr
– r = number of attributes (dimensions) in the data
• Algorithm partitions given data into k clusters
– Each cluster has a cluster center (centroid)
– K is user defined

66. K-means Clustering

67. K-Means Algorithm
1. Choose k
2. Randomly choose k data points (seeds) as initial centroids
3. Assign each data point to the closest centroid
4. Re-compute the centroids using the current cluster
memberships
5. If a convergence criterion is not met, go to 3

68. k initial means (in
this case k=3) are
randomly generated
within the data
domain
k clusters are
created by
associating every
observation with
the nearest mean
The centroid of
each of the k
clusters becomes
the new mean
Steps 2 and 3 are
repeated until
convergence has
been reached
K-Means Algorithm

69. Stopping / Convergence Criterion
1. No (or minimum) re-assignments of data points to different clusters
2. No (or minimum) change of centroids or minimum decrease in the sum
of squared error (SSE)
– Cj is the jth cluster, mj is the centroid of cluster Cj (the mean vector of all the data
points in Cj), and dist(x, mj) is the distance between data point x and centroid mj



k
j
C j
j
distSSE
1
2
),(x
mx

70. Example
+
+

71. Example

72. K Means - Strengths
• Simple to understand and implement
• Efficient:Time complexity O(tkn) where
– n is number of data points
– k is number of clusters
– t is number of iterations
• Since both k and t are small - a linear algorithm
• Most popular clustering algorithm
• Terminates at a local optimum if SSE is used
• The global optimum is hard to find due to complexity

73. K Means -Weaknesses
• Only applicable if mean is defined
– For categorical data, k-mode - the centroid is represented by most frequent
values
• User must specify k
• Sensitive to outliers
– Outliers are data points that are very far away from other data points
– Outliers could be errors in the data recording or some special data points with
very different values

74. K Means -Weaknesses

75. Handling Outliers
• Remove data points in the clustering process that are much further away
from the centroids than other data points
• Perform random sampling
– Since in sampling we only choose a small subset of the data points, the
chance of selecting an outlier is very small
– Assign the rest of the data points to the clusters by distance or similarity
comparison or classification

76. Sensitivity to Initial Seeds

77. Use of Different Seeds for Good Results
There are some methods to help choose good seeds

78. Weaknesses
+
Unsuitable for discovering clusters that are not hyper-ellipsoids (or hyper-spheres)

79. K-Means
• Still the most popular algorithm - simplicity, efficiency
• Other clustering algorithms have their own weaknesses
• No clear evidence that any other clustering algorithm performs better in
general
– although other algorithms could be more suitable for some specific types of
data or applications
• Comparing different clustering algorithms is a difficult task
• No one knows the correct clusters

80. Clusters Representation
• Use the centroid of each cluster to represent the cluster
• Compute the radius and standard deviation of the cluster to determine its
spread in each dimension
• Centroid representation alone works well if the clusters are of the hyper-
spherical shape
• If clusters are elongated or are of other shapes, centroids are not
sufficient

81. Cluster Classification
• All the points in a cluster have the same class label - the
cluster ID
• Run a supervised learning algorithm on the data to find a
classification model

82. Cluster Classification

83. Distance BetweenTwo Clusters
• Single link
• Complete link
• Average link
• Centroids
• …

84. Single Link Method
• The distance between two clusters is the distance between two closest
data points in the two clusters, one data point from each cluster
• It can find arbitrarily shaped clusters, but
– It may cause the undesirable chain effect by noisy points

85. Complete Link Method
• Distance between two clusters is the distance of two furthest data points
in the two clusters
• Sensitive to outliers because they are far away

86. Average Link Method
• Distance between two clusters is the average distance of all pair-wise
distances between the data points in two clusters
• A compromise between
– the sensitivity of complete-link clustering to outliers
– the tendency of single-link clustering to form long chains that do not
correspond to the intuitive notion of clusters as compact, spherical objects

87. Centroid Method
• Distance between two clusters is the distance
between their centroids

88. Algorithmic Complexity
• All the algorithms are at least O(n2)
– n is the number of data points
• Single link can be done in O(n2)
• Complete and average links can be done in O(n2logn)
• Due the complexity, hard to use for large data sets
– Sampling
– Scale-up methods (BIRCH)

89. Distance Functions
• Key to clustering
• similarity and dissimilarity are also commonly used terms
• Numerous distance functions
– Different types of data
• Numeric data
• Nominal data
– Different specific applications

90. Distance Functions - Numeric Attributes
• Euclidean distance
• Manhattan (city block) distance
• Denote distance with dist(xi, xj) where xi and xj are data points (vectors)
• They are special cases of Minkowski distance
• h is positive integer
hh
jrir
h
ji
h
jiji xxxxxxdist
1
2211 ))(...)()((),( xx

91. Distance Formulae
• If h = 2, it is the Euclidean distance
• If h = 1, it is the Manhattan distance
• Weighted Euclidean distance
22
22
2
11 )(...)()(),( jrirjijiji xxxxxxdist xx
||...||||),( 2211 jrirjijiji xxxxxxdist xx
22
222
2
111 )(...)()(),( jrirrjijiji xxwxxwxxwdist xx

92. Distance Formulae
• Squared Euclidean distance - to place progressively greater weight on
data points that are further apart
• Chebychev distance - one wants to define two data points as different if
they are different on any one of the attributes
22
22
2
11 )(...)()(),( jrirjijiji xxxxxxdist xx
|)|...,|,||,max(|),( 2211 jrirjijiji xxxxxxdist xx

93. Curse of Dimensionality
• Various problems that arise analyzing and organizing data in high
dimensional spaces do not occur in low dimensional space like 2D or 3D
• In the context of classification/function approximation, performance of
classification algorithm can improve by removing irrelevant features

94. Dimensionality Reduction - Applications
• Information Retrieval – web documents where dimensionality is
vocabulary of words
• Recommender systems – large scale of ratings matrix
• Social networks – social graph with large number of users
• Biology – gene expressions
• Image processing – facial recognition

95. Dimensionality Reduction
• Defying the curse of dimensionality - simpler models result in improved generalization
• Classification algorithm may not scale up to the size of the full feature set either in space or
time
• Improves understanding of domain
• Cheaper to collect and store data based on reduced feature set
• TwoTechniques
– FeatureConstruction
– Feature Selection

96. Dimensionality Reduction

97. Techniques
• Linear Discriminant Analysis – LDA
– Tries to identify attributes that account for the most variance between classes
– LDA compared to PCA is a supervised method using known labels
• Principal component analysis – PCA
– Identifies combination of linearly co-related attributes (principal components or directions in
feature space) that accounts for the most variance of data
– Plot the different samples on 2 first principal components
• Singular Value Decomposition – SVD
– Factorization of real or complex matrix
– Derived from PCA

98. Feature Construction
• Linear methods
– Principal component analysis (PCA)
– Independent component analysis (ICA)
– Fisher Linear Discriminant (LDA)
– ….
• Non-linear methods
– Kernel PCA
– Non linear component analysis (NLCA)
– Local linear embedding (LLE)
– ….

99. Principal component analysis - PCA
• A tool in exploratory data analysis and to create predictive models
• Involves calculating Eigenvalue decomposition of a data covariance matrix,
usually after mean centering the data for each attribute
• Mathematically defined as an orthogonal linear transformation to map data to a
new coordinate system such that the greatest variance by any projection of the
data comes to lie on the first coordinate (called the first principal component), the
second greatest variance on the second coordinate, and so on
• Theoretically the optimal linear scheme, in terms of least mean square error, for
compressing a set of high dimensional vectors into a set of lower dimensional
vectors and then reconstructing the original set

100. PCA

101. PCA

102. Fisher Linear Discriminant
• A classification method that projects high-dimensional data
onto a line and performs classification in this one-
dimensional space
• The projection maximizes the distance between the means of
the two classes while minimizing the variance within each
class

103. Fisher Linear Discriminant

104. Linear Discriminant Analysis - LDA
• A generalization of Fisher's linear
discriminant
• A method used to find a linear
combination of features that
characterizes or separates two or more
classes of objects or events
• The resulting combination may be used
as a linear classifier or more commonly
for dimensionality reduction before later
classification

105. Difference

106. Kernel PCA
• Classic PCA approach is a linear projection technique that works well if
the data is linearly separable
• In the case of linearly inseparable data, a nonlinear technique is required
if the task is to reduce the dimensionality of a dataset

107. Kernel PCA
• The basic idea to deal with linearly inseparable data is to project it onto a higher
dimensional space where it becomes linearly separable
• Consider a nonlinear mapping function ϕ so that the mapping of a sample xx can be written
as xx→ϕ(xx), which is called kernel function
• The term kernel describes a function that calculates the dot product of the images of the
samples xx under ϕ
• κ(xxi,xxj)=ϕ(xxi)Tϕ(xxj)
• Function ϕ maps the original d-dimensional features into a larger k-dimensional feature
space by creating nonlinear combinations of the original features

108. Kernel PCA

109. SingularValue Decomposition - SVD
• A mechanism to break a matrix into simpler meaningful pieces
• Used to detect groupings in data
• A factorization of a real or complex matrix
• A general rectangular M-by-N matrix A has a SVD into the product of an
M-by-N orthogonal matrix U, an N-by-N diagonal matrix of singular
values S and the transpose of an N-by-N orthogonal square matrixV
– A = U SV^T

110. SVD

111. Hidden Markov Models - HMM
• A statistical Markov model in which the system being modelled is
assumed to be a Markov process with unobserved (hidden) states
• Used in pattern recognition, such as handwriting and speech analysis
• In simpler Markov models like a Markov chain, the state is directly visible
to the observer, and therefore the state transition probabilities are the
only parameters
• In HMM, the state is not directly visible, but output, dependent on the
state, is visible

112. Hidden Markov Models - HMM
• Each state has a probability distribution over the possible output tokens
• Therefore the sequence of tokens generated by an HMM gives some
information about the sequence of states
• Adjective hidden refers to the state sequence through which the model
passes, not to the parameters of the model
• The model is still referred to as a 'hidden' Markov model even if these
parameters are known exactly

113. HMM

114. MODEL EVALUATION

115. Model Evaluation
• How accurate is the classifier?
• When the classifier is wrong, how is it wrong?
• Decide on which classifier (which parameters) to use
and to estimate what the performance of the system
will be

116. Testing Set
• Split the available data into a training set and a test set
• Train the classifier in the training set and evaluate based on
the test set

117. Classifier Accuracy
• The accuracy of a classifier on a given test set is the
percentage of test set tuples that are correctly classified by
the classifier
• Often also referred to as recognition rate
• Error rate (or misclassification rate) is the opposite of
accuracy

118. False PositiveVs. Negative
• When is the model wrong?
– False positives vs. false negatives
– Related to type I and type II errors in statistics
• Often there is a different cost associated with false
positives and false negatives
– Diagnosing diseases

119. Confusion Matrix
• Mechanism to illustrate how a model is performing in terms
of false positives and false negatives
• Provides more information than a single accuracy figure
• Allows thinking about the cost of mistakes
• Extendable to any number of classes

120. Confusion Matrix

121. More Accuracy Measures

122. Area Under ROC Curve - AUC
• ROC curves can be used to
compare models
• Bigger the AUC, the more
accurate the model
• ROC index is the area under
the ROC curve

123. Gini-Statistic
• Calculated from the ROC Curve
– Gini = 2 *AUC – 1
• Where the AUC is the area under the ROC curve

124. K-Fold CrossValidation
• Divide the entire data set into k folds
• For each of k experiments, use kth fold for testing and
everything else for training

125. K-Fold CrossValidation
• The accuracy of the system is calculated as the average error across the k
folds
• The main advantages of k-fold cross validation are that every example is
used in testing at some stage and the problem of an unfortunate split is
avoided
• Any value can be used for k
– 10 is most common
– Depends on the data set

126. References
1. W. L. Chao, J. J. Ding, “Integrated Machine Learning Algorithms for Human Age Estimation”, NTU, 2011
2. Phil Simon (March 18, 2013). Too Big to Ignore: The Business Case for Big Data. Wiley
3. Mitchell, T. (1997). Machine Learning, McGraw Hill. ISBN 0-07-042807-7
4. Harnad, Stevan (2008), "The Annotation Game: On Turing (1950) on Computing, Machinery, and Intelligence", in Epstein, Robert;
Peters, Grace, The Turing Test Sourcebook: Philosophical and Methodological Issues in the Quest for the Thinking Computer,
Kluwer
5. Russell, Stuart; Norvig, Peter (2003) [1995] Artificial Intelligence: A Modern Approach (2nd ed.) Prentice Hall
6. Langley, Pat (2011). "The changing science of machine learning". Machine Learning 82 (3): 275–279. doi:10.1007/s10994-011-5242-y
7. Le Roux, Nicolas; Bengio, Yoshua; Fitzgibbon, Andrew (2012). "Improving First and Second-Order Methods by Modeling
Uncertainty". In Sra, Suvrit; Nowozin, Sebastian; Wright, Stephen J. Optimization for Machine Learning. MIT Press. p. 404
8. MI Jordan (2014-09-10). "statistics and machine learning“ Cornell University Library. "Breiman : Statistical Modeling: The Two
Cultures (with comments and a rejoinder by the author)”
9. Gareth James; Daniela Witten; Trevor Hastie; Robert Tibshirani (2013). An Introduction to Statistical Learning. Springer
10. Yoshua Bengio (2009). Learning Deep Architectures for AI. Now Publishers Inc. pp. 1–3. ISBN 978-1-60198-294-0
11. A. M. Tillmann, "On the Computational Intractability of Exact and Approximate Dictionary Learning", IEEE Signal Processing
Letters 22(1), 2015: 45–49
12. Aharon, M, M Elad, and A Bruckstein. 2006. "K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse
Representation." Signal Processing, IEEE Transactions on 54 (11): 4311-4322
13. Goldberg, David E.; Holland, John H. (1988). "Genetic algorithms and machine learning". Machine Learning 3 (2): 95–99

127. ThankYou
Check Out My LinkedIn Profile at
https://in.linkedin.com/in/girishkhanzode