Teaching ML and AI in Education

#BigMLSchool
Agenda Machine Learning & AI* in Education:

• Objectives & Challenges

ML/AI* Industry Status:

• ML Adoption

• Scaling ML in the Enterprise

ML Platformization

AutoML

• Future Evolution

Conclusions & Recommendations

• For Business Schools

• For Technical Schools

Machine Learning

&

Education

ML Platforms

and

AutoML

#BigMLSchool
Agenda Machine Learning & AI* in Education:


ML/AI* Industry Status:

• ML Adoption


ML Platformization

AutoML





*Disclaimer: The term AI* (Artificial Intelligence) refers
specifically to the ability to build machine learning
driven applications which ultimately automate and/
or optimize business processes and SHOULD NOT BE
CONFUSED with robust or strong Artificial
Intelligence in the formal sense, ‘something not likely to
happen for a least this decade and/maybe next’ (emphasis
from the author)

Teaching & designing:
Master’s Degree Courses
Master in Enterprise Applied Intelligence & Master
in Data Analytics

• EAI6020 AI Systems & Technologies

• ALY6080 Experiential Learning - Machine
Learning & Analytics - Project Based

• ALY6983 Special Topics: Applied Machine
Learning

Corporate Learning
Machine Learning & AI - Enterprise Onboarding

#BigMLSchool
About your instructor:

• Nerd

(Engineer in the 90s - 1st PC: Commodore 64, 64Kb RAM)

• turned into Business

(Corporate Executive)

• turned Entrepreneur

(still shareholder)

• turned into VC

(Startups, VC and PE)

• turned into Board directorships

(Non-Exec Board Director)

• turned into Teaching

(Northeastern University - Silicon Valley, Berkeley Center for
Entrepreneurship & Technology, Headspring - IE Business
School)

Superskill: I can spin a [insert_object] in
the air on the tip of a finger
best way to learn anything,
teach it!
R. Feynman

#BigMLSchool
ALY6080/6983: Experiential Learning - Machine Learning
Motivation & Syllabus [6-12 weeks course]
Focus on learning by doing, real life - sponsored Capstone
project and theory, concepts and methods delivered with
examples and use cases
• Deep Learning vs Traditional ML
• Supervised Learning I: creating a ML app end to end, Linear
Regression - Decision Trees - Model Performance
• Supervised Learning II: Logistic Regression, Random Forest
& Ensembles, Bagging & Boosting, Neural Networks & DL
• Unsupervised Learning: Clustering, Association Discovery,
Anomaly Detection
• Feature Engineering, Dimensionality Reduction - PCA and
Automated ML
• Deploying ML models - Capstone Project
Tools & Technologies:
Python, R (legacy)
Tableau, PowerBI, BigML, AutoML (project/use case based)
Pic credit: BigML AutoML platform
https://github.com/whizzml/examples/tree/master/automl

#BigMLSchool
EAI6020: AI Systems & Technologies
Focus on Tools & Engineering for ML

• Machine Learning & AI*-Industry Overview

• ML/AI Engineering - Infrastructure & Tools (with Lab)

• Data Engineering

• Data Management

• ML Deployment & Prediction Serving (with Use Case &
Lab)

• Data Architecture Evolution & Business Rationale (with Use
Case)

• Capstone Project (Use Case & Lab)

Spark, SQL/NoSQL,

Databricks, BigML

References:

• Full Stack Deep Learning course - Berkeley (2019-2020/
Sergey Karayev)

• ML Systems Design - Stanford (2021/Chip Huyen)
Motivation & Syllabus [12 weeks core course]

#BigMLSchool
Objectives • Improve Practical Skills by exposure to Use
Cases and Sponsored - company projects
• Experiential Learning: Project Based, Practice
first - Theory/Math later (learning by doing)
• Improve Soft Skills: Communication, Synthesis
& Objectivity, Team Collaboration, Customer
Orientation, Sharing, working under pressure
• Objective & Outcomes focus: engineering vs
math, industry tools vs coding
• Focus on learning by doing, real life - company
sponsored Capstone project
• Close Industry gap: get (many more) ML
models to production

#BigMLSchool
The Missing
Course
in
Data Science
The Missing
Course
in
MBA
MBA
Data
Science
Technical
Knowledge:
Math
Statistics
Analytics
ML
Programming
Python/R
SQL/Databases
Business
Knowledge:
Soft Skills
Communication
Teamwork &
Collaboration
Data Driven Decisions
Digital Transformation
Finance
Leadership
ML Engineering:
Applied ML -
Project Based ML
MLOps
Data Engineering
Tools &
Infrastructure
Data Driven
Leadership:
ML Applications
ML Project
Management
Data Science/ML team
management
Tools & Infrastructure
Challenges: An Educational Gap
MBAs
&

#BigMLSchool
Challenges Hands-on experience and practical application of ML is relegated in
favor of theoretical and foundational knowledge (programming,
math, statistics) - The opposite is also true
• Select methods win over application oriented aspects
• Power shift in Curriculum: Syllabus must meet students
expectations
e.g demand for advanced DL methods (GANs, Transformers), despite
what ‘reality’ dictates (what you’ll need in a real job as Data Scientist
or MBA/exec of Data Driven projects)
• Students have (very) different backgrounds and levels of
experience:
Behavioral challenges due to diversity, cultural differences and
diverging attitudes
GOAL
• Find optimal balance between teaching hands-on best practices,
practical skills and technical skills/theory/concepts.
Why do students need to spend 6-9 months learning
to code before doing any ML?

#BigMLSchool
next Machine Learning & AI* in Education:


ML/AI Industry Status:

• ML Adoption


ML Platformization

AutoML





#BigMLSchool
Jensen Huang, CEO of Nvidia

#BigMLSchool
Machine Learning Adoption
Toward mainstream
source: courtesy of BigML Inc · http://bigml.com

#BigMLSchool
Adoption Cycle: Machine Learning Platforms
ML platforms: Custom Built vs Buy, crossing the chasm
source: adapted from BigML Inc materials · http://bigml.com
• Open Source
• Custom Built vs Buy
• Fragmented
• Proprietary
• Buy vs Build
• Consolidated

#BigMLSchool
credit: Full Stack Deep Learning Course - Infrastructure & Tools (*augmented with BigML & DataRobot Academic Programs)
* link to free Academic Programs:

#BigMLSchool
Internal &
External
ML modeling,
heuristics
AI assets:
ML platform
AI assets:
skills/expertise
ML Adoption
cross-function
Enterprise Roadmap for AI & ML at scale

#BigMLSchool
Scaling ML: dimensions
Use Cases
(sources of value)
ML Models
Generation
ML Models in
Production
• Volume, both by the number of Models in Production and the ability to validate new experiments/
hypothesis quickly determine success
• Significant number of models in production, complexity of ML workflows and model management call for
tools & platform approach (ML platforms)
• Rapid Model Prototyping driven by AutoML (Automated Machine Learning) for increased speed & efficiency
Key activities • Experiments & rapid
prototyping
• Validation & testing
• Model improvement/feature
engineering
• Model deployment
• Performance measurement &
monitoring
• Model drift/Model lifecycle
Management
Key technologies
/tools
AutoML ML Platforms

#BigMLSchool
How many ML models are too many models
Facebook ML platform (a.k.a FBlearner):
+1Mn ML models trained
+6 Mn predictions/sec
25% of engineering team using it
Source: ModelOps IBM research Waldemar Hummer et al

#BigMLSchool
Architecture of a ML Platform
ML at scale requires tooling and ultimately a platform approach
ML Platform architecture - Courtesy of BigML

#BigMLSchool
next Machine Learning & AI* in Education:



• ML Adoption


ML Platformization

AutoML

• Future Trends




#BigMLSchool
Amazon
Jeff Bezos’ letter to Amazon shareholders - May, 2017
“Machine learning and AI is a horizontal
enabling layer. It will empower and improve
every business, every government
organization, every philanthropy — basically
there’s no institution in the world that cannot
be improved with machine learning” .
Jeff Bezos

#BigMLSchool
Machine Learning Platforms
An Infrastructure & Service layer to drive ML at scale in the enterprise
Facebook FBlearner May 9, 2016
https://code.fb.com/core-data/
introducing-fblearner-flow-facebook-s-
ai-backbone/
Google TFX Tensorflow Aug 13, 2017
https://www.tensorflow.org/tfx/
https://dl.acm.org/ft_gateway.cfm?
id=3098021&ftid=1899117&dwn=1&CF
ID=81485403&CFTOKEN=79729647b
2ac491f-EAC34BCC-93F2-A3C5-
BE9311C722468452
Netflix
Notebook Data
Platform
Aug 16, 2018 https://medium.com/netflix-techblog/
notebook-innovation-591ee3221233
Uber Michelangelo Sept 5, 2017 https://eng.uber.com/michelangelo/
Twitter Cortex Sept, 2015
https://cortex.twitter.com/en.html
https://blog.twitter.com/engineering/
en_us/topics/insights/2018/ml-
workflows.html
Magic Pony acquisition - 2016:
https://www.bernardmarr.com/
default.asp?contentID=1373
AirBnB BigHead Feb, 2018
https://databricks.com/session/
bighead-airbnbs-end-to-end-machine-
learning-platform
LinkedIN Pro-ML Oct, 2018
https://engineering.linkedin.com/blog/
2018/10/an-introduction-to-ai-at-
linkedin

#BigMLSchool
an unfair ‘platform’ advantage

#BigMLSchool
eBay Krylov Dec 17, 2019
https://tech.ebayinc.com/engineering/
ebays-transformation-to-a-modern-ai-
platform/
Lyft Flyte Jan 20, 2020
https://eng.lyft.com/introducing-flyte-
cloud-native-machine-learning-and-
data-processing-platform-
fb2bb3046a59
AT&T Acumos Oct 30, 2017 https://medium.com/netflix-techblog/
notebook-innovation-591ee3221233
Spotify
Spotify ML
platform
Dec 13, 2019
https://labs.spotify.com/2019/12/13/the-
winding-road-to-better-machine-
learning-infrastructure-through-
tensorflow-extended-and-kubeflow/
Delta Airlines (licensed) Jan 8, 2020
https://www.aviationtoday.com/
2020/01/08/delta-develops-ai-tool-
address-weather-disruption-improve-
flight-operations/
GE
Predix (customer
IoT platform)
Feb, 2018
https://www.ge.com/digital/sites/
default/files/download_assets/Predix-
The-Industrial-Internet-Platform-
Brief.pdf
KT Telecom Neuroflow Jan, 2018 https://disruptive.asia/kt-ai-platform-
internal-use/
An Infrastructure & Service layer to drive ML at scale in the enterprise

#BigMLSchool
25
Increasing number of models & complexity
Facebook
Twitter
Linkedin
Google
SO PUT THE RIGHT ML PLATFORM IN PLACE
THESE COMPANIES ALREADY DID (Custom Built)
•e-commerce
•online/real time
transaccions
•consumer C2C services
•Predictions driven by
volume (millions) & models
•long term trends &
patterns
•B2B & Government
services
•consumer C2C services
•Predictions driven by
certainty vs speed
•rules based knowledge
AirBnB
Netflix
Spotify
GE
AT&T
Delta
eBay
Amazon
Lyft
Uber

MACHINE LEARNING AS A SERVICE MACHINE LEARNING PLATFORM & SOFTWARE
https://www.crisp-research.com/vendor-universe/machine-learning/#fndtn-mlaas
Vendor Landscape MLaaS: Machine Learning as a Service & On Premise
Source:

#BigMLSchool
“All Models are wrong, but some are useful”

#BigMLSchool
AutoML
Typical AutoML pipeline
AutoML
Feature
generation
Feature
selection
Model
selection
= + +
• Cluster Batch Centroids (Clustering)
• Anomaly Scores (Anomaly
Detection)
• Batch Association Sets (Association
Discovery): Using the objective field
from your dataset as consequent and
using leverage and lift as search_stra
tegy
• PCA Batch Projections (Principal
Component Analysis)
• Batch Topic Distributions (Topic
Model): Created only when the
dataset contains text fields.
• Recursive Feature Elimination
• automatically creating and
evaluating multiple models with
multiple configurations (decision
trees, ensembles, logistic
regressions, and deepnets) by
using Bayesian parameter
optimization.
The OptiML algorithm is split into two phases. The first, the “parameter
search” phase, uses a single holdout set to iteratively find promising sets of
parameters. The second, the “validation” phase is used to iteratively
perform Monte Carlo cross-validation on those parameters that are
somewhat close to the best.
References:
• Introduction to Automatic Model Selection - OptiML https://blog.bigml.com/2018/05/08/introduction-to-optiml-automatic-model-optimization/
• Recursive Feature Elimination - Github https://github.com/whizzml/examples/tree/master/recursive-feature-elimination
• Bayesian Parameter Optimization - Wikipedia https://en.wikipedia.org/wiki/Hyperparameter_optimization#Bayesian_optimization
• Automated Machine Learning - OptiML: https://blog.bigml.com/2018/05/16/optiml-the-nitty-gritty/

#BigMLSchool
AutoML
Automated Machine Learning
Problem Formulation
Data Acquisition
Feature Engineering
Modeling and Evaluations
Predictions
Measure Results
Data Transformations
5%
80%
• Data acquisition and transformation - semi
automated
• Feature Engineering, key to model
performance - semi automated
10% • Goal definition - Human driven
5% • Model Selection & Evaluation - automated
• Measuring & Monitoring - automated

#BigMLSchool
31
Enable knowledge workers (e.g., analysts, developers) to build
stable and insightful models quickly.

Scale the number of predictive use cases in collaboration with
non-technical peers through rapid prototyping.
Best AutoML approaches rely on automation of parts of the
Machine Learning process (e.g., hyper-parameter tuning)
without limiting the practitioners’ ability to control customization.

GDPR, data privacy, interpretability and prediction
explanations become critical concerns when deploying AutoML
AutoML
Automated Machine Learning
That feeling when your AutoML models are done

#BigMLSchool
32
AutoML DATAROBOT H2O BigML
Data Preparation
• Encoded categorical variables (one-hot);
Text n- grams; Missing values imputing;
Discretization (bins)

• limited manual transformations • Max. of
10 classes in the objective*
•Encoded categorical variables (one-hot); Missing
values handling; Date-time fields expansion; Bulk
interactions transformers; SVD numeric
transformer; CV target encoding; Cluster distance
transformer; Time lag

•Automatic feature engineering possible when
using AutoDL
• Encoded categorical variables (one-hot); Text
analysis; Missing values handling; Date-time fields
expansion

• Automatic Recursive Feature Selection & Feature
Engineering

• Multiple flexible manual transformations • Max of
1,000 classes in the objective
Optimization
Undisclosed optimization technique

(“expert data scientists preset
hyperparameter search space for models*)
Random Stacking

(a combination of random grid search and stacked
ensembles, plus early stopping)
Bayesian Parameter Optimization

(SMAC — Sequential Model-based Algorithm
Configuration) & DNN Metalearning
Models/Algorithms
•Open-source libraries: scikit-learn, R, H2O,
Tensorflow (not CNN or RNN), Spark,
XGBoost, DMTK, and Vowpal Wabbit  
•They also “blend” multiple models during
the optimization process.
•GBMs, Random Forests, XGBoost, deep neural
nets, and extreme random forests

•· Stacks of models can be learned. Best of family
stacks adopt the top model type from each of the
main algorithms.
•Decision trees, random decision forests, boosting,
logistic regression, deep neural networks  
•Customizable model ensembles with Fusions
leveraging the individually optimized models for
different classification, regression algorithms.
Speed It tests 30-40 different modeling
approaches and takes ~20 min.
Default time limit for AutoML is 1 hour. Can use
GPU or CPU. Can specify settings for accuracy,
time, and interpretability.
It tests 128 different modeling approaches
(creating more than 500 resources) and takes ~30
min.
Model
Visualizations &
Interpretability
• Limited model visualizations

• Feature importance for models • Predictions
explainability
• Dashboard: A single page with a global
interpretable model explanations plot, a feature
importance plot, a decision tree plot, and a partial
dependence plot.

• A machine learning interpretation tool (MLI) that
includes a KLIME or LIME-SUP graph.
• Multiple model visualizations to analyze the
impact of the variables on predictions:
sunburst, decision tree, partial dependence
plots, line chart (LR)
• Feature importance for models
• Predictions explainability
Model Evaluations
• Confusion matrix 
• ROC curve (only for binary classification) 
• Lift curve (only for binary classification) 
• Side-by-side evaluations comparison 
• Trade-off between complexity vs.
performance
• Models are ranked by cross-validation  
AUC by default.

• Return leaderboard sortable by deviance (mean
residual deviance), logloss, MSE, RMSE, MAE,
RMSLE, mean per class error
• Confusion matrix 
• ROC curve 
• Precision-Recall curve 
• Gain curve 
• Lift curve 
• Multiple evaluations comparison chart
Programmability &
Deployability
• Models can be used and created via API •
Export models 
• Cloud, VPC or on-premises
• H2O allows you to convert the models you have
built to either a Plain Old Java Object (POJO) or a
Model ObJect, Optimized (MOJO).

• H2O-generated MOJO and POJO models are
ieasily embeddable in Java environments
• Models can be used and created via API • Export
models 
• Cloud, VPC or on-premises
Source: Public Resources, Vendor Docs, BigML Analysis
Metalearning!

#BigMLSchool
33
AutoML - Metalearning
Automatic Network Hyperparameters Selection - DNNs (DeepNets)
We trained 296,748 deep neural networks
so you don’t have to!
• 296,748+ deep neural networks trained on 50 datasets
• For each one, recorded the optimum network structure for the
given dataset structure (number of fields, types of fields, etc)
• Trained a model to predict the optimum network structure for
any given dataset.
• This predicted network structure & hyper parameters can be
used directly or as a seed for a more intensive network search
Source: BigML - DeepNets https://blog.bigml.com/2017/10/04/deepnets-behind-the-scenes/
• Automated Machine Learning - OptiML: https://blog.bigml.com/2018/05/16/optiml-the-nitty-gritty/

#BigMLSchool
We are
here
(mostly)
Simplified* AI Landscape
* and imperfect
Future:

• Knowledge
representation
(symbolic/
Subsymbolic)

• Planning
(Reinforcement
Learning, Agents)

• Reasoning
(Causality, Logic,
Symbolic)

• Search &
Optimization
(evolutionary/
genetic algos)

#BigMLSchool
36
BigML, Inc
Private and Confidential
BigML Product Progression
5
AutoML, Linear
Regression, Node-
Red, Workflow
Report, Improved
Topic Modeling
Organizations,
Operating
Thresholds, OptiML,
Fusions, Data
Transformations, PCA
Boosted Trees,
ROC Analysis,
Time Series,
DeepNets
Scripts, Libraries,
Executions,
WhizzML, Logistic
Regression, Topic
Models
Association
Discovery,
Correlations,
Samples,
Statistical Tests
Anomaly Detection,
Clusters, Flatline
Evaluations, Batch
Predictions,
Ensembles,
Starbursts
Core ML Workflow:
Source, Dataset,
Model, Prediction
Prototyping and
Beta
2019
2018
2017
2016
2015
2014
2013
2012
2011
Automating Model Creation, Selection, Operation and Workflows = Making Machine Learning Easier
Reproducibility at the core:
Programmability, Interpretability, Explainability are
essential part of BigML's platform
Sophistication
Ease
of
Use
WE HAVE BEEN BUILDING A STRONG FOUNDATION TO DEVELOP, DEPLOY AND OPERATE MACHINE-LEARNING BASED APPLICATIONS OF UNPARALLELED QUALITY

#BigMLSchool
37
BigML, Inc
Private and Confidential
7
AI/ML
Market
Maturity
Automating Workflows for
Model Creation,
Selection, Operation
Extending the Platform to Build and Manage Smarter Predictive Applications End-to-End
Building the BEST End-
to-End Machine
Learning Platform
2020 2030
1980
BigML's Co-Founder
Participates in first University
Machine Learning
2011
BigML
Founded
BigML Future
EXTENDING THE PLATFORM TO BUILD AND MANAGE SMARTER PREDICTIVE APPLICATIONS END-TO-END
Reasoning
Knowledge
Representation
Planning Optimization
Principles
Machine Learning
ROBUST AI
Doing to Reasoning, Planning, Knowledge Representation
and Optimization what we have done to Machine Learning
and combining them to build Robust AI Applications
Machine Learning

#BigMLSchool
Recommendations
Context
The world is changing…. fast:
• Companies are adopting ML/AI* quickly, the build- vs-buy
paradigm is changing
• ML tools & platforms are spreading (buy vs build/open source)
• Success out there is measured by the ability to deploy models
rapidly and efficiently
• Technical debt in ML is an issue, MLOps and Engineering
becoming critical (time to model deployment => time to market)
• Students, technical or not, will confront a world where they’ll be
expected to understand and (somehow) master ML/AI end to end
• Tools & Platforms are here to help, coding necessary but not core
to problem and solution (code automation and scripting)

#BigMLSchool
Recommendations II
For Educators, Business Schools and Technical Schools
• MBAs and business leaders need to understand
tech/ML/AI
• Include ML in the Curriculum, industry approach,
key concepts and high level ML modeling (no hard
coding but use of scripting tools)
• Experiential learning, hands on project
assignments, tie ML models and use cases to
business value.
• Technical students need more Soft skills (comms,
teamwork, project management).
• MBAs need more ‘hard’ tech skills (tools,
applications & tech concepts)
AI

#BigMLSchool
End Machine Learning & AI* in Education:



• ML Adoption


ML Platformization

AutoML





Teaching ML and AI in Education

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Teaching ML and AI in Education

Similaire à Teaching ML and AI in Education (20)

Plus de BigML, Inc

Plus de BigML, Inc (20)

Dernier

Dernier (20)

Teaching ML and AI in Education