2. Motivation
Deep networks can be easily fooled … [1]
Evolution Algor.
generated images
99.99%
confidence
“It turns out some DNNs only
focus on discriminative
features in images.”
[1] Nguyen A, Yosinski J, Clune J. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In Computer Vision and
Pattern Recognition (CVPR '15), IEEE, 2015.
6. Big Picture
Are the modern learning systems really secure?
Training
dataset Model
Test
(Validation)
dataset
Training Test
Update
● Increase test error
● Reduce learning accuracy
● Fool the intelligent system
● Achieve personal gain
7. Big Picture
Are the modern learning systems really secure?
Training
dataset Model
Test
(Validation)
dataset
Training Test
Update
Causative Attack
Exploratory Attack
8. Attack’s capability
Access to Data Knowledge about
features
Knowledge about the
classifier
Limited Knowledge Partially Maybe Yes
Perfect Knowledge Yes Yes Yes
These are real inputs from users.
9. Basics
❏ Observations
❏ True signal:
❏ Polynomial curve fitting
❏ is unknown
❏ => learn the green curve
Observation
Original
signal
12. Overfitting
❏ Bad on unseen test set
❏ Central problem of ML.
❏ Generalization
❏ E.g., regularization, prior,
more data, model
selection
13. Bias-Variance
❏ Trade off
❏ Overfitting == low bias, high variance
❏ Underfitting == high bias, low variance
❏ Noise is dominating!
W is very
sensitive
Bias Variance Decomposition
15. Types of Adversaries
● Causative Attack (Poisoning)
○ Understanding how the learning algorithms work
○ Engineering on features or labels of training set
○ Change the discriminant function
● Exploratory Attack (Evasion)
○ Engineering features of a test point
○ Circumvent the legitimate detection
○ Change the discriminant result
16. Types of Adversaries
● Causative Attack (Poisoning)
○ Understanding how the learning algorithms work
○ Engineering on features or labels of training set
○ Change the discriminant function
● Exploratory Attack (Evasion)
○ Engineering features of a test point
○ Circumvent the legitimate detection
○ Change the discriminant result
17. Label Noises on SVM
● SVM: One of the state-of-art classifier
● Binary case: +1, -1
● Label flips attack under a certain budget
● Maximizing error on validation set
● Methods:
○ ALFA
○ Distance based: far-first, near-first, random
○ Continuous relaxation gradient ascend
○ Correlated cluster
18. Basics
We measure the error on a validation set using the function trained on
training set.
A training data set
A validation data set
Classifier trained on
Regularization coefficient
Risk measurement on validation set
21. Huang Xiao, B. Biggio, B. Nelson, Han Xiao, C. Eckert, and F. Roli, “Support Vector Machines under Adversarial Label
Contamination”, Neurocomputing, vol. Special Issue on Advances in Learning with Label Noise, In Press.
22. Poisoning Attack on SVM
● Noises on features, not on labels
● Design a malicious training point
● Maximizing the error (e.g., test error,
hinge loss, ...)
● Gradient ascend
23. How to?
Retrain the SVM after injecting a malicious point ,, , move the
point such that the classification error on validation set is maximized.
Validation data set with m samples
SVM trained on training set with a malicious point
26. B. Biggio, B. Nelson, and P. Laskov, “Poisoning attacks against support vector machines”, in 29th Int'l Conf. on Machine
Learning (ICML), 2012.
27. Walking example
B. Biggio, B. Nelson, and P. Laskov, “Poisoning attacks against support vector machines”, in 29th Int'l Conf. on Machine
Learning (ICML), 2012
You can:
● Mimic the ‘9’ as ‘8’ or,
● Label a ‘9’ as a ‘8’
28. Poisoning Lasso
● Lasso: feature selection, more generally,
L1 regularization
● Feature selection is often the first step
for many learning system
● Other targets: Rigid regression, elastic
network
● Gradient based method
29. Lasso
Capture the most relevant features in data set
automatically by shrinking the feature weights.
from:
Tibshirani, R. (1996). Regression shrinkage
and selection via the lasso. J. Royal.
Statist. Soc B., Vol. 58, No. 1, pages 267-
288).
35. Research goals
● Investigating robustness of feature
selection algorithms
● Design a multiple point attack method
● Warning: feature selection might not be
reliable
● A gradient based poisoning framework
36. Objective function
We inject a malicious point to form a
new compromised Data .
Variable: , we are maximising w.r.t
Remark that is learnt on contaminated data .
Maximise Generalization Error!
38. Demonstration
Error surface
Initial attack
point
on each (x, y)
Xiao, Huang, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is Feature Selection
Secure against Training Data Poisoning?. In ICML'15,Lille, France, July 2015.
39. Demonstration
Gradient ascend
path
Xiao, Huang, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is Feature Selection
Secure against Training Data Poisoning?. In ICML'15,Lille, France, July 2015.
40. Wrap up
● Don’t expect your algorithms too fancy
● Don’t expect adversaries too silly
● Setup objective and do the worst-case
study
● Machine learning needs to be more
robust
● There’s no innocent data