Applying Data Science to Your Business Problem

1 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
World®
’16
Applying Data Science to
Your Business Problem
Paul Dulany - VP Data Science - CA Technologies
SCX31S
SECURITY

© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

Abstract
For a while now, a number of industries have been interested in data science and
advanced analytics. But it isn’t always clear how best to use these within the
business context. In this session, we’ll discuss how to turn a business problem into
a data-science problem, and then back. We’ll use card-not-present payment fraud
and login attempts as examples of how to identify the problem, determine if data
science and advanced analytics can help (and if the situation warrants them), and
then follow through on developing a solution to the problem.
Paul Dulany,
PhD
CA Technologies
VP Data Science

Agenda
WHAT IS DATA SCIENCE?
DETERMINING A PROBLEM OF INTEREST
UNDERSTAND THE PRODUCTION ENVIRONMENT AND DEMANDS
MODEL CREATION AND EVALUATION
Q & A
1
2
3
4
5

Key Points for Applying Data Science
§ Identify a high-value Business Problem with High Quality Data
§ Determine the class of the problem to solve
§ Utilize business-domain knowledge
– Understand the "ecosystem"
– Define appropriate metrics
– Understand the data in full
§ Develop features and models / Evaluate / Iterate
§ Always keep the business problem in mind!

What is Data Science?
§ The application of analytical techniques to large and “big”
data
– A wide field encompassing many different aspects of analytics,
statistics, and data mining
– Fundamentally data driven
– Based upon the scientific method
– The goal is to use data and analytical techniques to solve problems
§ Requires knowledge in multiple domains
– Analytics
– Scientific computations
– Data formats
– Business domain
– Statistics

Steps in Applying Data Science
§ Identify a high-value business problem
– The business case is critical
§ Intelligent Mainframe Operations
– Need early detection of issues
§ Best is to predict and avoid issues
– Currently, false positives (false alarms) are too prevalent
– Expert-maintained systems of thresholds are hard to maintain
Business Problem

§ Payment Security:
– Fraud in eCommerce is a significant problem
§ 3-D Secure was developed to combat this
– Issuers incur the most pain from the current state
§ Fraud losses
§ Loss of income from interest and interchange fees
§ Customer experience and annoyance
§ Cost of inbound calls
– Merchants feel pain too…
Business Problem

Business Impact for a Client: 3 Year Impact
Credit Debit
Fraud savings (above current vendor) £13,441,843 £15,174,365
Interchange fees (above current) £466,462 £3,441,882
Interest income (above current) £3,674,803 N/A
Operational savings (not calculated) - -
Total £17,583,108 £18,616,247

§ Identify data related to the business problem
– The more data, the better!
– Is it categorical, ordinal, or numerical? What cardinality?
– Is there a unique advantage over the competition?
§ Payment Security
– 3DS data: PAReq message, device information, …
– Wide mix of types of data
– Time series is important
– SaaS Deployment allows quality data to be gathered
Identify Data (Results Proportional to Quality!)

– Multiple possible data sources
§ VSAM, DB2, IMS DB, IDMS, DATACOM, SMF, Syslogs, Vtape, CICS, …
– Utilize CA SYSVIEW’s excellence
– Embed analytics to detect abnormal patterns
Identify Data (Results Proportional to Quality!)

§ Determine the general class of problem
– Classification, regression, anomaly detection, etc.
– “Supervised” (teaching your children to read, teaching them manners)
– “Unsupervised” (university)
– “Semi-supervised” (school lunch room)

§ Payment Security
– Supervised classification
– Fraud information for losses is likely in good shape
– Complexities happen once you have a system in place
§ Censored problem, both in marking and in changing fraudster behavior
– Unsupervised to begin
– Need to develop baselines of normal behavior
§ But must provide results from day 0
– Possibility of semi-supervised in the future
Determine the general class of problem

§ Understand the ecosystem
– What actions can be taken?
– Is getting the result time sensitive?
– Predictive analytics needed
– Multiple possible actions, key is to inform the operator’s actions
– Different time-scales for problems
– Reaction-time is critical – real-time

Steps in applying Data Science
§ Payment Security
– Predictive and prescriptive analytics needed
– Multiple possible actions, at the transaction and the card level
– Timing is critical – real-time, i.e., < 50ms for vast majority
§ We must be able to take action now
Understand the Ecosystem

§ Determine the appropriate metrics
– How do we measure success?
§ Well defined measures are critical
– High Availability
– Problem avoidance
– Reduced MTTR
– Reduce SME dependence for issue detection

§ Payment Security
– A number of possibilities, based upon customer’s objectives
– Consider them all
§ Detection rates
§ “Outsort” rates
§ False-positive ratios
§ False-positive rates
§ Value-based / transaction based / card based
Metrics
TOR 𝑆 =
∑ 𝐹 𝑠*
+ 𝑁 𝑠*
./0.
∑ 𝐹 𝑠* + 𝑁 𝑠*
122 .*
TDR 𝑆 =
∑ 𝐹 𝑠*
./0.
∑ 𝐹 𝑠*
122 .*

What’s Next?
§ Now we have a well defined problem
§ So we spend a lot of time with the data!
– “Browse” the data
– Run some descriptive statistics
– See if you can surprise yourself
– If supervised, view the tagging data and the
production data separately, and then
together.

Review the Tagged Data
§ Browse the data again
§ Run many statistical test
– Beware of “Target Leaks”!!
– Begin getting a feel for the variations, correlations, idiosyncrasies
§ You never want perfectly clean data
– You want data that simulates production!
§ Beware of any changes to the data, especially non-causal changes
§ Model training is a numerical simulation of production

Data Partitioning: Beware What You Partition and How!
§ Partitioning
– Often need to use stratified
sampling
– When using multiple entities
for tracking behavior,
interactions are tricky!
§ Look for irreducibility
§ Go to out-of-time if needed
Historical Data
Training
Fraud
Non-Fraud
Validate
Fraud
Non-
Fraud
Holdout
Fraud
Non-
Fraud

Feature Development
§ “Features” pull the distinguishing
characteristics from the data
– Time series analysis techniques
– Digital Signal Processing techniques
– Statistical measures of differences
– Bayesian approaches
– Linear discriminants
– Non-linear transformations
– …

Feature Development: Simple Example
§ Determine a peer group from the historical data
– All transactions where there were four previous transactions in the last
week, at least two of which were on the same device, but in different
countries
§ Determine the distributions of classes for a continuous
variable
– Let’s say, the amount
– Use a discriminant calculation to
determine likelihood of belonging to either class.

Feature Development: More Complexity
§ At the other end of the scale are online-learning models to
determine behaviors
– Autocorrelation models, exponential weighting, KDE, etc.
– Many techniques –
§ but must keep in mind the CPU constraints, I/O constraints, etc.
§ Conversion of high-cardinality
categorical data into numerical
inputs
𝑥"(𝑡 𝑛 , 𝑡 𝑛−1, 𝑡0) = 𝛼(𝑡 𝑛 , 𝑡0)𝑥 𝑛 + 𝛽(𝑡 𝑛 , 𝑡 𝑛−1, 𝑡0)𝑥" 𝑛−1
𝛼(𝑡 𝑛 , 𝑡0) =
1 − 𝛾
1 − 𝛾(𝑡 𝑛 −𝑡0)

𝛽(𝑡 𝑛 , 𝑡 𝑛−1, 𝑡0) =
𝛾(𝑡 𝑛 −𝑡 𝑛−1)
11 − 𝛾(𝑡 𝑛−1−𝑡0)
2
1 − 𝛾(𝑡 𝑛 −𝑡0)

Example: Unsupervised Anomaly Detection
§ Utilize Historical data to define bands of different probabilities
– Map real time metric streams against system defined normal
– Multi-point alerts generated using industry-standard Western-Electric
rules
– Make static thresholds optional!
Unlikely
Most Likely
Metric
Time
Less Likely

Example: Supervised Model
§ Always remember Occam's Razor!
– Among competing hypotheses, the one with the fewest assumptions
should be selected.
– Avoid needless complexity
§ Start with simple models, and grow more complex as needed
– Linear regression
– Logistic regression
– Decision trees
– Neural Networks
– SVM …

Neural Network

§ There are many aspects of training a neural network
– different activation and error functions
– different training algorithms,
– variations of seeds, learning rate, momentum,
– self-regulation,
– number of hidden layers,
– number of nodes,
– boosting/bagging,
– preventing overfitting,
– etc.
Train the Model(s)

Review
§ Review the results of your training, and start all over again!
– Consider segmentation
– Try leaving out the variables with the highest sensitivity
– Subdivide the data to see if there are regions of instability
– Iterate as needed
§ Finally, select your model(s)!
§ But we’re not done…
– Now worry about calibration, upgrade/downgrade, priming time,
packaging, integration, model report, …

Model Performance Chart

Intelligent Mainframe Operations
Typical
Volatility
Anomaly
Tasks ready to be dispatched

Key Points for Applying Data Science
§ Identify a high-value Business Problem with High Quality Data
§ Determine the class of the problem to solve
§ Utilize business-domain knowledge
– Understand the "ecosystem"
– Define appropriate metrics
– Understand the data in full
§ Develop features and models / Evaluate / Iterate
§ Always keep the business problem in mind!

Recommended Sessions
SESSION # TITLE DATE/TIME
SCX50S
Convenience and Security for banking customers with CA
Advanced Authentication
11/17/2016 at 3:00 pm
SCX34S
Securing Mobile Payments: Applying Lessons Learned in
the Real World
11/17/2016 at 3:45 pm
SCT05T Threat Analytics for Privileged Access Management 11/17/2016 at 4:30 pm

Don’t Miss Our INTERACTIVE
Security Demo Experience!
SNEAK
PEEK!

We Want to Hear From You!
§ IT Central is a leading technology review site. CA has them to
help generate product reviews for our Security products.
§ ITCS staff may be at this session now! (look for their shirts). If
you would like to offer a product review, please ask them
after the class, or go by their booth.
Note:
§ Only takes 5-7 mins
§ You have total control over the review
§ It can be anonymous, if required

Questions?

Stay connected at communities.ca.com
Thank you.

Security
For more information on Security, please visit:
http://cainc.to/EtfYyw

Applying Data Science to Your Business Problem

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Applying Data Science to Your Business Problem

Similaire à Applying Data Science to Your Business Problem (20)

Plus de CA Technologies

Plus de CA Technologies (20)

Dernier

Dernier (20)

Applying Data Science to Your Business Problem