SlideShare une entreprise Scribd logo
1  sur  190
daniel martin katz
michael j bommarito
adjunct professor @ university of michigan
associate professor of law @ illinois tech - chicago kent
co-founder @ LexPredict
co-founder @ LexPredict
Legal Analytics, Machine Learning and
Some Comments on the Status of
Innovation in the Legal Industry
Forum on Legal Evolution
NYC
in the legal
industry
there already
is better corn
"We always overestimate the change that will
occur in the next two years and underestimate
the change that will occur in the next ten”
- bill gates
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
today’s focus is primarily
legal analytics + process engineering
© daniel martin katz michael j bommarito
before providing some
concrete examples -
some broad thoughts ...
© daniel martin katz michael j bommarito
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
three faces of innovation in legal
© daniel martin katz michael j bommarito
(1) lawyers for innovators / entrepreneurs
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
what most lawyers and law schools think of as “Law+Entrepreneurship"
(1) lawyers for innovators / entrepreneurs
(2) lawyers as innovators - substance
© daniel martin katz michael j bommarito
poison pill - “the most important innovation in corporate law
since Samuel Calvin Tate Dodd invented the trust
for John D. Rockefeller and Standard Oil in 1879”
© daniel martin katz michael j bommarito
(2) lawyers as innovators - substance
emerging areas - 3D Printing, Driverless Cars, Augmented Reality,
Data Breach, Big Data+Privacy, etc.
Drones, Internet of Things, CyberSecurity,
© daniel martin katz michael j bommarito
(2) lawyers as innovators - substance
(3) lawyers as innovators - business/process
© daniel martin katz michael j bommarito
innovation directed toward transforming the practice of law
© daniel martin katz michael j bommarito
(3) lawyers as innovators - business/process
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
there are different ways that
organizations are innovating
on the third face
{Law
Substantive
Legal
Expertise
Analytics
Platform
AI
Computing
Process Mapping
User Experience
Design Thinking
Business Models
Regulation
Marketing
+ Tech + Design
TM
+ Delivery}
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
some traditional law firms
have been very aggressive
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
but most of the innovation
is Lex.Startup
© daniel martin katz michael j bommarito
Lex.Startup
is beginning to take hold
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
15
2009
Lex.Startup
15
2009
Lex.Startup
15 425+
2009 2014
Law or Legal Related Companies*
as highlighted by Josh Kubicki @ ReInventLaw London 2013
Lex.Startup
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
© daniel martin katz michael j bommarito
So what are these folks doing?
R + D Function in the
Legal Industry
© daniel martin katz michael j bommarito
We Could Imagine a World
Where Law Firms Did the
R+D for the Industry
© daniel martin katz michael j bommarito
But That Has
(Mostly) Proven Illusive
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Lex.Startup
is undertaking that function
© daniel martin katz michael j bommarito
Here are the specific
approaches that are
being undertaken
© daniel martin katz michael j bommarito
Some organizations are
doing more than one
© daniel martin katz michael j bommarito
labor
arbitrage
© daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
© daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
© daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
© daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
predictive
analytics
© daniel martin katz michael j bommarito
could do an
individual talk on
any of these topics...
© daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
predictive
analytics
© daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
predictive
analytics
© daniel martin katz michael j bommarito
Predictive Analytics in Law
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
The Data Driven
Future of the
Legal Industry
© daniel martin katz michael j bommarito
is already here...
© daniel martin katz michael j bommarito
2011
© daniel martin katz michael j bommarito
2011
© daniel martin katz michael j bommarito
2012
© daniel martin katz michael j bommarito
2013
© daniel martin katz michael j bommarito
2013
© daniel martin katz michael j bommarito
2013
© daniel martin katz michael j bommarito
2013
2013
© daniel martin katz michael j bommarito
2013
© daniel martin katz michael j bommarito
2013
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
© daniel martin katz michael j bommarito
Quantitative Legal Prediction
- or -
How I Learned to Stop Worrying and
Start Preparing for the Data Driven
Future of the Legal Services Industry
Daniel Martin Katz
Associate Professor of Law
Michigan State University
62 Emory L. J. 909 (2013)
© daniel martin katz michael j bommarito
Cause
and
Effect
Quantitative
Legal
Prediction
vs.
© daniel martin katz michael j bommarito
Cause
and
Effect
Quantitative
Legal
Prediction
vs.
© daniel martin katz michael j bommarito
Machine Learning
is the heart of
predictive analytics
Legal Analytics
Professor Daniel Martin Katz
Professor Michael J Bommarito II
@MSU Law - Winter 2014
© daniel martin katz michael j bommarito
Supervised
Statistical models
Bayesian, e.g., Naïve Bayes Classification
Frequentist, e.g., Ordinary Least Squares
Neural Networks (NN)
Support Vector Machines (SVM)
Random Forests (RF)
Genetic Algorithms (GA)
Semi/Unsupervised
Neural Networks (NN)
Clustering
K-means
Hierarchical
Radial Basis (RBF)
Graph
Some Machine Learning Methods
© daniel martin katz michael j bommarito
http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html
© daniel martin katz michael j bommarito
classification
clustering
regression
dimension reduction
the family of machine learning methods © daniel martin katz michael j bommarito
Quick Example of
the Methods
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Adapted from Slides By
Victor Lavrenko and Nigel Goddard
@ University of Edinburgh
Take A LookThese 12
© daniel martin katz michael j bommarito
72
Female
Human
3
Female
Horse
36
Male
Human
21
Male
Human
67
Male
Human
29
Female
Human
54
Male
Human
44
Male
Human
50
Male
Human
42
Female
Human
6
Male
Dog
7
Female
Human
© daniel martin katz michael j bommarito
Classification
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?
© daniel martin katz michael j bommarito
Classification
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?
Regression
(Supervised Learning)
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4
© daniel martin katz michael j bommarito
Classification
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?
f( )
Loan
Application?
Yes
Multi Class Classification
(Supervised Learning)
No
Maybe
Yes
Perhaps
No
Multiclass =
Boundary
Hyperplane
Regression
(Supervised Learning)
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4
© daniel martin katz michael j bommarito
Classification
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?
f( )
Loan
Application?
Yes
Multi Class Classification
(Supervised Learning)
No
Maybe
Yes
Perhaps
No
Multiclass =
Boundary
Hyperplane
Regression
(Supervised Learning)
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4
Clustering
(Unsupervised
Learning)
Clusterf( )
Group?
© daniel martin katz michael j bommarito
Regression as a Prediction Tool
© daniel martin katz michael j bommarito
Regression as a Prediction Tool
© daniel martin katz michael j bommarito
Standard Linear Regression
Can Be Used to
Predict a Probability
(using LPM, Logit, etc.)
© daniel martin katz michael j bommarito
Standard Linear Regression
Can Be Used to
Predict a Quantity
© daniel martin katz michael j bommarito
Task = Predict the Expected Cost of
a Given Legal Service
f( )
Cost?
#
and/or
010
101
001
Regression (Supervised Learning)
© daniel martin katz michael j bommarito
http://reinventlawchannel.com/ron-gruner-were-on-a-mission/
© daniel martin katz michael j bommarito
Y = βo +/- β1 ( X1 ) +/- β2 ( X2 ) +/- β3 ( X3 ) +/- β4 ( X3 ) +/- β5 ( X3 ) + ε
Y = $151 + $15 ( ) + 161 ( ) + 95 ( ) + 34 ( ) +/- β5 ( ) + ε
Per
100
Lawyers
If Tier 1
Market
is True
Partner
Status
is True
Per
10
Years
Practice
Area
© daniel martin katz michael j bommarito
Turn Around and
Use This Model
To Predict Other Lawyers
(also Matters, etc.)
© daniel martin katz michael j bommarito
This Requires a Method to Deal
With Changes in Dynamics, etc.
© daniel martin katz michael j bommarito
This Requires a Method to Update
the Model as Time Moves Forward
© daniel martin katz michael j bommarito
Must Deal With Overfitting
to the Existing Data
© daniel martin katz michael j bommarito
Machine Learning and
the Future of E-Discovery
© daniel martin katz michael j bommarito
imagine your client is served
with a request for production
© daniel martin katz michael j bommarito
in random
order
assume
this is the
size
of the
hypothetical
document
set
(emails,
memos,
etc.)
we can
sample
a subset
of the
documents
we can
sample
a subset
of the
documents
© daniel martin katz michael j bommarito
classification
clustering
regression
dimension reduction
© daniel martin katz michael j bommarito
classification
© daniel martin katz michael j bommarito
predictive coding =
~ binary classification
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
LearningTask = Determine Whether a Given
Document is Relevant?
Relevant
Not Relevant
f( )
relevance?
Binary Classification (Supervised Learning)
and/or
010
101
001
take the sample set as
a training set and
use human experts
© daniel martin katz michael j bommarito
the use of the human
experts is called
“supervised learning”
© daniel martin katz michael j bommarito
in the simple binary case,
ask humans to assign
objects to two piles
© daniel martin katz michael j bommarito
Apply Human Coders
© daniel martin katz michael j bommarito
yellow = relevant
white = non-relevant
and return this
© daniel martin katz michael j bommarito
Non RelevantRelevant
© daniel martin katz michael j bommarito
Key Insight ...
© daniel martin katz michael j bommarito
What Allows A
Human To Separate
These Two Classes of
Documents?
© daniel martin katz michael j bommarito
that precise human
process is what
“predictive coding”
is trying to mimic
© daniel martin katz michael j bommarito
most vendors are selling a
largely undifferentiated product
© daniel martin katz michael j bommarito
Humans are selecting
upon some “features”
of the documents
© daniel martin katz michael j bommarito
to place those
documents in their
respective bins

(i.e. relevant, non-relevant)
© daniel martin katz michael j bommarito
features =?
text,
author,
date,
other metadata
© daniel martin katz michael j bommarito
machine learning task is
trying to recover (learn)
what separates the
relevant from the
non-relevant documents
© daniel martin katz michael j bommarito
once we learn the
rule / boundary
we can apply it to separate
the remain documents into
the two classes
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
we want to take what we learn here
© daniel martin katz michael j bommarito
we want to take what we learn here
© daniel martin katz michael j bommarito
we want to take what we learn here
and apply it here
© daniel martin katz michael j bommarito
the future of e-discovery
will follow the arc of
machine learning
© daniel martin katz michael j bommarito
Supervised Unsupervised
Predictive
Coding
(Classification)
The Long
Term Future
Machine
Learning
Methods
2 x 2
Informed
Naive
Basic
Clustering
Algorithm
© daniel martin katz michael j bommarito
there are different
forms of learning
by machines ...
© daniel martin katz michael j bommarito
There Is Learning
Within a Matter
(i.e. learning from a
specific training set)
© daniel martin katz michael j bommarito
In other words, it is
possible for the
machine to learn from
the experience of
having processed
documents in the past
© daniel martin katz michael j bommarito
both inside a given
company but also
across companies ...
© daniel martin katz michael j bommarito
this is how
data aggregation / reusing data
becomes very powerful
© daniel martin katz michael j bommarito
data aggregation / reusing data
make the naive into the informed
© daniel martin katz michael j bommarito
data aggregation / reusing data
help move from the supervised
to the semi/unsupervised
© daniel martin katz michael j bommarito
Supervised Unsupervised
Predictive
Coding
(Classification)
The Future
Machine
Learning
Methods
2 x 2
Informed
Naive
Basic
Clustering
Algorithm
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Machine Learning
Natural Language Processing
and Due Diligence
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
The system comes pre-trained 

for provisions including:
Title, Parties, Date, Term, Change of
Control, Assignment, Indemnity,
Confidentiality, Governing Law,
License Grant, Bankruptcy, Notice,
Amendment, Non-Solicit, and more.
Based on testing, we know our system finds
90% or more of the instances of nearly
every substantive provision it covers.
This 90% number is our system’s recall;
its precision differs by provision by
provision but is consistently very
manageable.
© daniel martin katz michael j bommarito
We are able to build custom provisions on
request. Thanks to our highly customized
training algorithms, this process is easy and
relatively automated. We are also engaged
in adding more provisions.
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Machine Learning
and
Judicial Behavior
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
2002 Prediction Tourney
and its limits
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Model Leverages
Classification Tree
(Tool from Machine Learning)
Standard Decision Tree
Often Not Generalizable,
Often Overfits the Data
© daniel martin katz michael j bommarito
Need a more complex approach
© daniel martin katz michael j bommarito
Predicting the Behavior of the
United States Supreme Court:
A General Approach
© daniel martin katz michael j bommarito
Black
Reed
Frankfurter
Douglas
Jackson
Burton
Clark
Minton
Warren
Harlan
Brennan
Whittaker
Stewart
White
Goldberg
Fortas
Marshall
Burger
Blackmun
Powell
Rehnquist
Stevens
OConnor
Scalia
Kennedy
Souter
Thomas
Ginsburg
Breyer
Roberts
Alito
Sotomayor
Kagan
1953 1963 1973 1983 1993 2003 2013
9-0 Reverse
8-1, 7-2, 6-3
19 19 19 19 19 20 20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
- Reverse
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
-
8-1, 7-2, 6-3
9-0
19 19 19 19 19 20 20
feature engineering
© daniel martin katz michael j bommarito
The real world gives us raw material, at best.
 Typically, you even have to dig the stuff raw
material out of your own unstructured data
similar approach can be
applied to other problems
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Case Prediction
and
Litigation Data
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
“John Dragseth, a principal at
Fish & Richardson (the most active
IP litigation firm in the United
States, according to Corporate
Counsel magazine), credits Lex
Machina’s database with helping
him spot meaningful but
otherwise hidden trends in IP
litigation—and he won’t give
details. “If you published it, then
people on the other side would
know,” he says.
© daniel martin katz michael j bommarito
Notice there is an
offloading of data but
it is up to the end user
to derive meaning
© daniel martin katz michael j bommarito
In general, the relevant
consumer market is not
yet mature when it
comes to data science
© daniel martin katz michael j bommarito
Difficult to sell machine
learning technology in
instances where the
end user does not have
the right assets in place
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Many other examples ...
just starting to come online
© daniel martin katz michael j bommarito
Attorney Quality
and Performance
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution-  NYC
© daniel martin katz michael j bommarito
Leveraging Public Data
for Legal Insight
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Change
Management
is the Hardest
Innovation of All
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
Bulls and Bears
~1984 - 2009 ~2009 - 2014
© daniel martin katz michael j bommarito
53 in 2009
58 in 2014
If you were
28 in 1984
than you were
© daniel martin katz michael j bommarito
before 2009 most of
the individuals in the
profession have only
known the bull market
© daniel martin katz michael j bommarito
it is a bear market now ...
and in a bear market you
need a serious strategy
© daniel martin katz michael j bommarito
analytics/data should be
part of that strategy
© daniel martin katz michael j bommarito
“data is the oil of the 21st Century”
So lets be wildcatters
© daniel martin katz michael j bommarito
law < > finance
many elements in law look
like finance did 25 years ago
© daniel martin katz michael j bommarito
© daniel martin katz michael j bommarito
When it comes to
innovation at the
level that is going
to be needed ...
© daniel martin katz michael j bommarito
Assigning a innovation partner
or an innovation committee is
probably not enough
© daniel martin katz michael j bommarito
Shunk Works
© daniel martin katz michael j bommarito
how many
organizations have a
full time data scientist
(data science team)?
© daniel martin katz michael j bommarito
need a full scale
and empowered
R+D team
(data science, tech, etc.)
© daniel martin katz michael j bommarito
Final Thought
Exit,
Voice
&
Loyalty
© daniel martin katz michael j bommarito
daniel martin katz
michael j bommarito ii
adjunct professor of Law @ michigan state university
associate professor of law @ illinois tech - chicago kent
co-founder @ LexPredict
director of research @ reInventLaw laboratory
co-founder @ LexPredict
Forum on Legal Evolution
NYC

Contenu connexe

Tendances

Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Ma...
Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Ma...Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Ma...
Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Ma...Daniel Katz
 
Can Law Librarians Help Law Become More Data Driven ? An Open Question in Ne...
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Ne...Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Ne...
Can Law Librarians Help Law Become More Data Driven ? An Open Question in Ne...Daniel Katz
 
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...Daniel Katz
 
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...Daniel Katz
 
Trends in legal tech 2018
Trends in legal tech 2018Trends in legal tech 2018
Trends in legal tech 2018Secure Privacy
 
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...Daniel Katz
 
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...Daniel Katz
 
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...Daniel Katz
 
Introduction to artificial intelligence and law
Introduction to artificial intelligence and lawIntroduction to artificial intelligence and law
Introduction to artificial intelligence and lawLawScienceTech
 
Innovation and Emerging Technology
Innovation and Emerging TechnologyInnovation and Emerging Technology
Innovation and Emerging TechnologyRon Dolin
 
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...Daniel Katz
 
Thoughts on Legal Prediction and Legal Metrics - Association of Corporate Cou...
Thoughts on Legal Prediction and Legal Metrics - Association of Corporate Cou...Thoughts on Legal Prediction and Legal Metrics - Association of Corporate Cou...
Thoughts on Legal Prediction and Legal Metrics - Association of Corporate Cou...Daniel Katz
 
Investment in Legal Technology
Investment in Legal TechnologyInvestment in Legal Technology
Investment in Legal TechnologyEvolve Law
 
Legal tech trends 2019 report
Legal tech trends 2019 reportLegal tech trends 2019 report
Legal tech trends 2019 reportDan Storbaek
 
Trends in legal tech 2018
Trends in legal tech 2018Trends in legal tech 2018
Trends in legal tech 2018Dan Storbaek
 
Legal Analytics - Introduction to the Course - Professor Daniel Martin Katz +...
Legal Analytics - Introduction to the Course - Professor Daniel Martin Katz +...Legal Analytics - Introduction to the Course - Professor Daniel Martin Katz +...
Legal Analytics - Introduction to the Course - Professor Daniel Martin Katz +...Daniel Katz
 
Partner event tibco patterns 2011-10-12
Partner event   tibco patterns 2011-10-12Partner event   tibco patterns 2011-10-12
Partner event tibco patterns 2011-10-12Lakshmi Srikanth Reddy
 
Vintage group-newsletter xbrl-final-lr-1017
Vintage group-newsletter xbrl-final-lr-1017Vintage group-newsletter xbrl-final-lr-1017
Vintage group-newsletter xbrl-final-lr-1017genein19
 
IQPC NY Financial Conference on eDiscovery: Legal Speaks Greek and IT Speaks ...
IQPC NY Financial Conference on eDiscovery: Legal Speaks Greek and IT Speaks ...IQPC NY Financial Conference on eDiscovery: Legal Speaks Greek and IT Speaks ...
IQPC NY Financial Conference on eDiscovery: Legal Speaks Greek and IT Speaks ...J. David Morris
 

Tendances (19)

Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Ma...
Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Ma...Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Ma...
Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Ma...
 
Can Law Librarians Help Law Become More Data Driven ? An Open Question in Ne...
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Ne...Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Ne...
Can Law Librarians Help Law Become More Data Driven ? An Open Question in Ne...
 
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
 
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
 
Trends in legal tech 2018
Trends in legal tech 2018Trends in legal tech 2018
Trends in legal tech 2018
 
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
 
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
 
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
 
Introduction to artificial intelligence and law
Introduction to artificial intelligence and lawIntroduction to artificial intelligence and law
Introduction to artificial intelligence and law
 
Innovation and Emerging Technology
Innovation and Emerging TechnologyInnovation and Emerging Technology
Innovation and Emerging Technology
 
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
 
Thoughts on Legal Prediction and Legal Metrics - Association of Corporate Cou...
Thoughts on Legal Prediction and Legal Metrics - Association of Corporate Cou...Thoughts on Legal Prediction and Legal Metrics - Association of Corporate Cou...
Thoughts on Legal Prediction and Legal Metrics - Association of Corporate Cou...
 
Investment in Legal Technology
Investment in Legal TechnologyInvestment in Legal Technology
Investment in Legal Technology
 
Legal tech trends 2019 report
Legal tech trends 2019 reportLegal tech trends 2019 report
Legal tech trends 2019 report
 
Trends in legal tech 2018
Trends in legal tech 2018Trends in legal tech 2018
Trends in legal tech 2018
 
Legal Analytics - Introduction to the Course - Professor Daniel Martin Katz +...
Legal Analytics - Introduction to the Course - Professor Daniel Martin Katz +...Legal Analytics - Introduction to the Course - Professor Daniel Martin Katz +...
Legal Analytics - Introduction to the Course - Professor Daniel Martin Katz +...
 
Partner event tibco patterns 2011-10-12
Partner event   tibco patterns 2011-10-12Partner event   tibco patterns 2011-10-12
Partner event tibco patterns 2011-10-12
 
Vintage group-newsletter xbrl-final-lr-1017
Vintage group-newsletter xbrl-final-lr-1017Vintage group-newsletter xbrl-final-lr-1017
Vintage group-newsletter xbrl-final-lr-1017
 
IQPC NY Financial Conference on eDiscovery: Legal Speaks Greek and IT Speaks ...
IQPC NY Financial Conference on eDiscovery: Legal Speaks Greek and IT Speaks ...IQPC NY Financial Conference on eDiscovery: Legal Speaks Greek and IT Speaks ...
IQPC NY Financial Conference on eDiscovery: Legal Speaks Greek and IT Speaks ...
 

En vedette

Legal Language Explorer .com Tutorial
Legal Language Explorer .com  TutorialLegal Language Explorer .com  Tutorial
Legal Language Explorer .com TutorialDaniel Katz
 
Schellingcodemap
SchellingcodemapSchellingcodemap
SchellingcodemapDaniel Katz
 
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Pre...
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Pre...Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Pre...
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Pre...Daniel Katz
 
Complex Systems Computing - Webscraping - Bonus Module
Complex Systems Computing - Webscraping - Bonus ModuleComplex Systems Computing - Webscraping - Bonus Module
Complex Systems Computing - Webscraping - Bonus ModuleDaniel Katz
 
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010Daniel Katz
 
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Daniel Katz
 
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...Daniel Katz
 

En vedette (7)

Legal Language Explorer .com Tutorial
Legal Language Explorer .com  TutorialLegal Language Explorer .com  Tutorial
Legal Language Explorer .com Tutorial
 
Schellingcodemap
SchellingcodemapSchellingcodemap
Schellingcodemap
 
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Pre...
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Pre...Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Pre...
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Pre...
 
Complex Systems Computing - Webscraping - Bonus Module
Complex Systems Computing - Webscraping - Bonus ModuleComplex Systems Computing - Webscraping - Bonus Module
Complex Systems Computing - Webscraping - Bonus Module
 
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
 
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
 
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
 

Similaire à Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC

{Law, Tech, Design, Delivery} Observations Regarding Innovation in the Legal ...
{Law, Tech, Design, Delivery} Observations Regarding Innovation in the Legal ...{Law, Tech, Design, Delivery} Observations Regarding Innovation in the Legal ...
{Law, Tech, Design, Delivery} Observations Regarding Innovation in the Legal ...Daniel Katz
 
About Scott Goldstein: MortgageOrb Person of the Week
About Scott Goldstein: MortgageOrb Person of the WeekAbout Scott Goldstein: MortgageOrb Person of the Week
About Scott Goldstein: MortgageOrb Person of the WeekNDeXTech
 
The Importance Of The Internet
The Importance Of The InternetThe Importance Of The Internet
The Importance Of The InternetLisa Thompson
 
Essay On How To Make The Malahat Safer
Essay On How To Make The Malahat SaferEssay On How To Make The Malahat Safer
Essay On How To Make The Malahat SaferErica Smith
 
2014 Tech M&A Monthly - Deal Structures Today
2014 Tech M&A Monthly - Deal Structures Today2014 Tech M&A Monthly - Deal Structures Today
2014 Tech M&A Monthly - Deal Structures TodayCorum Group
 
Automation of Loan Origination using Process and Decision Services
Automation of Loan Origination using Process and Decision ServicesAutomation of Loan Origination using Process and Decision Services
Automation of Loan Origination using Process and Decision ServicesDenis Gagné
 
ICPSR - Complex Systems Models in the Social Sciences - Lecture 6 - Professor...
ICPSR - Complex Systems Models in the Social Sciences - Lecture 6 - Professor...ICPSR - Complex Systems Models in the Social Sciences - Lecture 6 - Professor...
ICPSR - Complex Systems Models in the Social Sciences - Lecture 6 - Professor...Daniel Katz
 
A Brief Note On The Uk Multinational Company
A Brief Note On The Uk Multinational CompanyA Brief Note On The Uk Multinational Company
A Brief Note On The Uk Multinational CompanySandra Ahn
 
Advantages And Disadvantages Of BPO
Advantages And Disadvantages Of BPOAdvantages And Disadvantages Of BPO
Advantages And Disadvantages Of BPOHeather Harvey
 
CrimsonLogic World Bank_IADB_Washington DC_30 Sep 2009_eGovernance to yield g...
CrimsonLogic World Bank_IADB_Washington DC_30 Sep 2009_eGovernance to yield g...CrimsonLogic World Bank_IADB_Washington DC_30 Sep 2009_eGovernance to yield g...
CrimsonLogic World Bank_IADB_Washington DC_30 Sep 2009_eGovernance to yield g...Ceski
 
Principle of mobile privacy - Privacy in data era
Principle of mobile privacy - Privacy in data eraPrinciple of mobile privacy - Privacy in data era
Principle of mobile privacy - Privacy in data eramohammad alsalamin
 
Crypto Regulation Fight Begins
Crypto Regulation Fight BeginsCrypto Regulation Fight Begins
Crypto Regulation Fight BeginsInvestingTips
 
The Democratization Of Real Estate
The Democratization Of Real EstateThe Democratization Of Real Estate
The Democratization Of Real EstateDonald Teel
 
Blockchain in practice
Blockchain in practice Blockchain in practice
Blockchain in practice SettleMint
 
Defining Digital Transformation - the research
Defining Digital Transformation - the researchDefining Digital Transformation - the research
Defining Digital Transformation - the researchTom Rieger
 
Change acceptance - Catherine Hellebaut
Change acceptance - Catherine HellebautChange acceptance - Catherine Hellebaut
Change acceptance - Catherine HellebautAGILEMinds
 
SYMCTranscript4304
SYMCTranscript4304SYMCTranscript4304
SYMCTranscript4304finance40
 
Software impact of gdpr
Software impact of gdprSoftware impact of gdpr
Software impact of gdprIRIS
 
SYMCTranscript2608
SYMCTranscript2608SYMCTranscript2608
SYMCTranscript2608finance40
 

Similaire à Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC (20)

{Law, Tech, Design, Delivery} Observations Regarding Innovation in the Legal ...
{Law, Tech, Design, Delivery} Observations Regarding Innovation in the Legal ...{Law, Tech, Design, Delivery} Observations Regarding Innovation in the Legal ...
{Law, Tech, Design, Delivery} Observations Regarding Innovation in the Legal ...
 
About Scott Goldstein: MortgageOrb Person of the Week
About Scott Goldstein: MortgageOrb Person of the WeekAbout Scott Goldstein: MortgageOrb Person of the Week
About Scott Goldstein: MortgageOrb Person of the Week
 
The Importance Of The Internet
The Importance Of The InternetThe Importance Of The Internet
The Importance Of The Internet
 
Essay On How To Make The Malahat Safer
Essay On How To Make The Malahat SaferEssay On How To Make The Malahat Safer
Essay On How To Make The Malahat Safer
 
2014 Tech M&A Monthly - Deal Structures Today
2014 Tech M&A Monthly - Deal Structures Today2014 Tech M&A Monthly - Deal Structures Today
2014 Tech M&A Monthly - Deal Structures Today
 
Automation of Loan Origination using Process and Decision Services
Automation of Loan Origination using Process and Decision ServicesAutomation of Loan Origination using Process and Decision Services
Automation of Loan Origination using Process and Decision Services
 
ICPSR - Complex Systems Models in the Social Sciences - Lecture 6 - Professor...
ICPSR - Complex Systems Models in the Social Sciences - Lecture 6 - Professor...ICPSR - Complex Systems Models in the Social Sciences - Lecture 6 - Professor...
ICPSR - Complex Systems Models in the Social Sciences - Lecture 6 - Professor...
 
A Brief Note On The Uk Multinational Company
A Brief Note On The Uk Multinational CompanyA Brief Note On The Uk Multinational Company
A Brief Note On The Uk Multinational Company
 
Cost Plus History
Cost Plus HistoryCost Plus History
Cost Plus History
 
Advantages And Disadvantages Of BPO
Advantages And Disadvantages Of BPOAdvantages And Disadvantages Of BPO
Advantages And Disadvantages Of BPO
 
CrimsonLogic World Bank_IADB_Washington DC_30 Sep 2009_eGovernance to yield g...
CrimsonLogic World Bank_IADB_Washington DC_30 Sep 2009_eGovernance to yield g...CrimsonLogic World Bank_IADB_Washington DC_30 Sep 2009_eGovernance to yield g...
CrimsonLogic World Bank_IADB_Washington DC_30 Sep 2009_eGovernance to yield g...
 
Principle of mobile privacy - Privacy in data era
Principle of mobile privacy - Privacy in data eraPrinciple of mobile privacy - Privacy in data era
Principle of mobile privacy - Privacy in data era
 
Crypto Regulation Fight Begins
Crypto Regulation Fight BeginsCrypto Regulation Fight Begins
Crypto Regulation Fight Begins
 
The Democratization Of Real Estate
The Democratization Of Real EstateThe Democratization Of Real Estate
The Democratization Of Real Estate
 
Blockchain in practice
Blockchain in practice Blockchain in practice
Blockchain in practice
 
Defining Digital Transformation - the research
Defining Digital Transformation - the researchDefining Digital Transformation - the research
Defining Digital Transformation - the research
 
Change acceptance - Catherine Hellebaut
Change acceptance - Catherine HellebautChange acceptance - Catherine Hellebaut
Change acceptance - Catherine Hellebaut
 
SYMCTranscript4304
SYMCTranscript4304SYMCTranscript4304
SYMCTranscript4304
 
Software impact of gdpr
Software impact of gdprSoftware impact of gdpr
Software impact of gdpr
 
SYMCTranscript2608
SYMCTranscript2608SYMCTranscript2608
SYMCTranscript2608
 

Plus de Daniel Katz

Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...Daniel Katz
 
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...Daniel Katz
 
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...Daniel Katz
 
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...Daniel Katz
 
Legal Analytics Course - Class 9 - Clustering Algorithms (K-Means & Hierarch...
Legal Analytics Course - Class 9 -  Clustering Algorithms (K-Means & Hierarch...Legal Analytics Course - Class 9 -  Clustering Algorithms (K-Means & Hierarch...
Legal Analytics Course - Class 9 - Clustering Algorithms (K-Means & Hierarch...Daniel Katz
 
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...Daniel Katz
 
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...Daniel Katz
 
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Daniel Katz
 
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...Daniel Katz
 
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...Daniel Katz
 
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...Daniel Katz
 
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...Daniel Katz
 
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5Daniel Katz
 

Plus de Daniel Katz (13)

Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
 
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
 
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
 
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
 
Legal Analytics Course - Class 9 - Clustering Algorithms (K-Means & Hierarch...
Legal Analytics Course - Class 9 -  Clustering Algorithms (K-Means & Hierarch...Legal Analytics Course - Class 9 -  Clustering Algorithms (K-Means & Hierarch...
Legal Analytics Course - Class 9 - Clustering Algorithms (K-Means & Hierarch...
 
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
 
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
 
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
 
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
 
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
 
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
 
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
 
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
 

Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC

  • 1. daniel martin katz michael j bommarito adjunct professor @ university of michigan associate professor of law @ illinois tech - chicago kent co-founder @ LexPredict co-founder @ LexPredict Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry
  • 2. Forum on Legal Evolution NYC
  • 3. in the legal industry there already is better corn
  • 4. "We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten” - bill gates
  • 6. today’s focus is primarily legal analytics + process engineering © daniel martin katz michael j bommarito
  • 7. before providing some concrete examples - some broad thoughts ... © daniel martin katz michael j bommarito
  • 9. three faces of innovation in legal © daniel martin katz michael j bommarito
  • 10. (1) lawyers for innovators / entrepreneurs © daniel martin katz michael j bommarito
  • 11. © daniel martin katz michael j bommarito what most lawyers and law schools think of as “Law+Entrepreneurship" (1) lawyers for innovators / entrepreneurs
  • 12. (2) lawyers as innovators - substance © daniel martin katz michael j bommarito
  • 13. poison pill - “the most important innovation in corporate law since Samuel Calvin Tate Dodd invented the trust for John D. Rockefeller and Standard Oil in 1879” © daniel martin katz michael j bommarito (2) lawyers as innovators - substance
  • 14. emerging areas - 3D Printing, Driverless Cars, Augmented Reality, Data Breach, Big Data+Privacy, etc. Drones, Internet of Things, CyberSecurity, © daniel martin katz michael j bommarito (2) lawyers as innovators - substance
  • 15. (3) lawyers as innovators - business/process © daniel martin katz michael j bommarito
  • 16. innovation directed toward transforming the practice of law © daniel martin katz michael j bommarito (3) lawyers as innovators - business/process
  • 17. © daniel martin katz michael j bommarito
  • 18. © daniel martin katz michael j bommarito there are different ways that organizations are innovating on the third face
  • 19. {Law Substantive Legal Expertise Analytics Platform AI Computing Process Mapping User Experience Design Thinking Business Models Regulation Marketing + Tech + Design TM + Delivery} © daniel martin katz michael j bommarito
  • 20. © daniel martin katz michael j bommarito
  • 21. © daniel martin katz michael j bommarito some traditional law firms have been very aggressive
  • 22. © daniel martin katz michael j bommarito
  • 23. © daniel martin katz michael j bommarito but most of the innovation is Lex.Startup
  • 24. © daniel martin katz michael j bommarito Lex.Startup is beginning to take hold
  • 28. 15 425+ 2009 2014 Law or Legal Related Companies* as highlighted by Josh Kubicki @ ReInventLaw London 2013 Lex.Startup
  • 42. © daniel martin katz michael j bommarito So what are these folks doing?
  • 43. R + D Function in the Legal Industry © daniel martin katz michael j bommarito
  • 44. We Could Imagine a World Where Law Firms Did the R+D for the Industry © daniel martin katz michael j bommarito
  • 45. But That Has (Mostly) Proven Illusive © daniel martin katz michael j bommarito
  • 46. © daniel martin katz michael j bommarito Lex.Startup is undertaking that function
  • 47. © daniel martin katz michael j bommarito Here are the specific approaches that are being undertaken
  • 48. © daniel martin katz michael j bommarito Some organizations are doing more than one
  • 49. © daniel martin katz michael j bommarito labor arbitrage
  • 50. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage
  • 51. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage regulatory arbitrage
  • 52. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage regulatory arbitrage design as the ultimate bespoke
  • 53. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage regulatory arbitrage design as the ultimate bespoke predictive analytics
  • 54. © daniel martin katz michael j bommarito could do an individual talk on any of these topics...
  • 55. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage regulatory arbitrage design as the ultimate bespoke predictive analytics
  • 56. © daniel martin katz michael j bommarito labor arbitrage process/tech arbitrage regulatory arbitrage design as the ultimate bespoke predictive analytics
  • 57. © daniel martin katz michael j bommarito
  • 58. Predictive Analytics in Law © daniel martin katz michael j bommarito
  • 59. © daniel martin katz michael j bommarito The Data Driven Future of the Legal Industry
  • 60. © daniel martin katz michael j bommarito is already here...
  • 61. © daniel martin katz michael j bommarito 2011
  • 62. © daniel martin katz michael j bommarito 2011
  • 63. © daniel martin katz michael j bommarito 2012
  • 64. © daniel martin katz michael j bommarito 2013
  • 65. © daniel martin katz michael j bommarito 2013
  • 66. © daniel martin katz michael j bommarito 2013
  • 67. © daniel martin katz michael j bommarito 2013
  • 68. 2013
  • 69. © daniel martin katz michael j bommarito 2013
  • 70. © daniel martin katz michael j bommarito 2013
  • 72. © daniel martin katz michael j bommarito Quantitative Legal Prediction - or - How I Learned to Stop Worrying and Start Preparing for the Data Driven Future of the Legal Services Industry Daniel Martin Katz Associate Professor of Law Michigan State University 62 Emory L. J. 909 (2013)
  • 73. © daniel martin katz michael j bommarito Cause and Effect Quantitative Legal Prediction vs.
  • 74. © daniel martin katz michael j bommarito Cause and Effect Quantitative Legal Prediction vs.
  • 75. © daniel martin katz michael j bommarito Machine Learning is the heart of predictive analytics
  • 76. Legal Analytics Professor Daniel Martin Katz Professor Michael J Bommarito II @MSU Law - Winter 2014 © daniel martin katz michael j bommarito
  • 77. Supervised Statistical models Bayesian, e.g., Naïve Bayes Classification Frequentist, e.g., Ordinary Least Squares Neural Networks (NN) Support Vector Machines (SVM) Random Forests (RF) Genetic Algorithms (GA) Semi/Unsupervised Neural Networks (NN) Clustering K-means Hierarchical Radial Basis (RBF) Graph Some Machine Learning Methods © daniel martin katz michael j bommarito
  • 79. classification clustering regression dimension reduction the family of machine learning methods © daniel martin katz michael j bommarito
  • 80. Quick Example of the Methods © daniel martin katz michael j bommarito
  • 81. © daniel martin katz michael j bommarito Adapted from Slides By Victor Lavrenko and Nigel Goddard @ University of Edinburgh Take A LookThese 12
  • 82. © daniel martin katz michael j bommarito 72 Female Human 3 Female Horse 36 Male Human 21 Male Human 67 Male Human 29 Female Human 54 Male Human 44 Male Human 50 Male Human 42 Female Human 6 Male Dog 7 Female Human
  • 83. © daniel martin katz michael j bommarito Classification (Supervised Learning) decision boundary female male f( ) Gender?
  • 84. © daniel martin katz michael j bommarito Classification (Supervised Learning) decision boundary female male f( ) Gender? Regression (Supervised Learning) #f( ) Age? 723 2 3 67 54 29 42 44 50 7 6 27 44 53 3 68 2 48 10 6 743 4 4
  • 85. © daniel martin katz michael j bommarito Classification (Supervised Learning) decision boundary female male f( ) Gender? f( ) Loan Application? Yes Multi Class Classification (Supervised Learning) No Maybe Yes Perhaps No Multiclass = Boundary Hyperplane Regression (Supervised Learning) #f( ) Age? 723 2 3 67 54 29 42 44 50 7 6 27 44 53 3 68 2 48 10 6 743 4 4
  • 86. © daniel martin katz michael j bommarito Classification (Supervised Learning) decision boundary female male f( ) Gender? f( ) Loan Application? Yes Multi Class Classification (Supervised Learning) No Maybe Yes Perhaps No Multiclass = Boundary Hyperplane Regression (Supervised Learning) #f( ) Age? 723 2 3 67 54 29 42 44 50 7 6 27 44 53 3 68 2 48 10 6 743 4 4 Clustering (Unsupervised Learning) Clusterf( ) Group?
  • 87. © daniel martin katz michael j bommarito Regression as a Prediction Tool
  • 88. © daniel martin katz michael j bommarito Regression as a Prediction Tool
  • 89. © daniel martin katz michael j bommarito Standard Linear Regression Can Be Used to Predict a Probability (using LPM, Logit, etc.)
  • 90. © daniel martin katz michael j bommarito Standard Linear Regression Can Be Used to Predict a Quantity
  • 91. © daniel martin katz michael j bommarito Task = Predict the Expected Cost of a Given Legal Service f( ) Cost? # and/or 010 101 001 Regression (Supervised Learning)
  • 92. © daniel martin katz michael j bommarito http://reinventlawchannel.com/ron-gruner-were-on-a-mission/
  • 93. © daniel martin katz michael j bommarito Y = βo +/- β1 ( X1 ) +/- β2 ( X2 ) +/- β3 ( X3 ) +/- β4 ( X3 ) +/- β5 ( X3 ) + ε Y = $151 + $15 ( ) + 161 ( ) + 95 ( ) + 34 ( ) +/- β5 ( ) + ε Per 100 Lawyers If Tier 1 Market is True Partner Status is True Per 10 Years Practice Area
  • 94. © daniel martin katz michael j bommarito Turn Around and Use This Model To Predict Other Lawyers (also Matters, etc.)
  • 95. © daniel martin katz michael j bommarito This Requires a Method to Deal With Changes in Dynamics, etc.
  • 96. © daniel martin katz michael j bommarito This Requires a Method to Update the Model as Time Moves Forward
  • 97. © daniel martin katz michael j bommarito Must Deal With Overfitting to the Existing Data
  • 98. © daniel martin katz michael j bommarito
  • 99. Machine Learning and the Future of E-Discovery © daniel martin katz michael j bommarito
  • 100. imagine your client is served with a request for production © daniel martin katz michael j bommarito
  • 101. in random order assume this is the size of the hypothetical document set (emails, memos, etc.)
  • 102. we can sample a subset of the documents
  • 103. we can sample a subset of the documents
  • 104. © daniel martin katz michael j bommarito classification clustering regression dimension reduction
  • 105. © daniel martin katz michael j bommarito classification
  • 106. © daniel martin katz michael j bommarito
  • 107. predictive coding = ~ binary classification © daniel martin katz michael j bommarito
  • 108. © daniel martin katz michael j bommarito LearningTask = Determine Whether a Given Document is Relevant? Relevant Not Relevant f( ) relevance? Binary Classification (Supervised Learning) and/or 010 101 001
  • 109. take the sample set as a training set and use human experts © daniel martin katz michael j bommarito
  • 110. the use of the human experts is called “supervised learning” © daniel martin katz michael j bommarito
  • 111. in the simple binary case, ask humans to assign objects to two piles © daniel martin katz michael j bommarito
  • 112. Apply Human Coders © daniel martin katz michael j bommarito
  • 113. yellow = relevant white = non-relevant and return this © daniel martin katz michael j bommarito
  • 114. Non RelevantRelevant © daniel martin katz michael j bommarito
  • 115. Key Insight ... © daniel martin katz michael j bommarito
  • 116. What Allows A Human To Separate These Two Classes of Documents? © daniel martin katz michael j bommarito
  • 117. that precise human process is what “predictive coding” is trying to mimic © daniel martin katz michael j bommarito
  • 118. most vendors are selling a largely undifferentiated product © daniel martin katz michael j bommarito
  • 119. Humans are selecting upon some “features” of the documents © daniel martin katz michael j bommarito
  • 120. to place those documents in their respective bins
 (i.e. relevant, non-relevant) © daniel martin katz michael j bommarito
  • 121. features =? text, author, date, other metadata © daniel martin katz michael j bommarito
  • 122. machine learning task is trying to recover (learn) what separates the relevant from the non-relevant documents © daniel martin katz michael j bommarito
  • 123. once we learn the rule / boundary we can apply it to separate the remain documents into the two classes © daniel martin katz michael j bommarito
  • 124. © daniel martin katz michael j bommarito we want to take what we learn here
  • 125. © daniel martin katz michael j bommarito we want to take what we learn here
  • 126. © daniel martin katz michael j bommarito we want to take what we learn here and apply it here
  • 127. © daniel martin katz michael j bommarito
  • 128. the future of e-discovery will follow the arc of machine learning © daniel martin katz michael j bommarito
  • 129. Supervised Unsupervised Predictive Coding (Classification) The Long Term Future Machine Learning Methods 2 x 2 Informed Naive Basic Clustering Algorithm © daniel martin katz michael j bommarito
  • 130. there are different forms of learning by machines ... © daniel martin katz michael j bommarito
  • 131. There Is Learning Within a Matter (i.e. learning from a specific training set) © daniel martin katz michael j bommarito
  • 132. In other words, it is possible for the machine to learn from the experience of having processed documents in the past © daniel martin katz michael j bommarito
  • 133. both inside a given company but also across companies ... © daniel martin katz michael j bommarito
  • 134. this is how data aggregation / reusing data becomes very powerful © daniel martin katz michael j bommarito
  • 135. data aggregation / reusing data make the naive into the informed © daniel martin katz michael j bommarito
  • 136. data aggregation / reusing data help move from the supervised to the semi/unsupervised © daniel martin katz michael j bommarito
  • 137. Supervised Unsupervised Predictive Coding (Classification) The Future Machine Learning Methods 2 x 2 Informed Naive Basic Clustering Algorithm © daniel martin katz michael j bommarito
  • 138. © daniel martin katz michael j bommarito
  • 139. Machine Learning Natural Language Processing and Due Diligence © daniel martin katz michael j bommarito
  • 140. © daniel martin katz michael j bommarito
  • 141. © daniel martin katz michael j bommarito The system comes pre-trained 
 for provisions including: Title, Parties, Date, Term, Change of Control, Assignment, Indemnity, Confidentiality, Governing Law, License Grant, Bankruptcy, Notice, Amendment, Non-Solicit, and more.
  • 142. Based on testing, we know our system finds 90% or more of the instances of nearly every substantive provision it covers. This 90% number is our system’s recall; its precision differs by provision by provision but is consistently very manageable. © daniel martin katz michael j bommarito
  • 143. We are able to build custom provisions on request. Thanks to our highly customized training algorithms, this process is easy and relatively automated. We are also engaged in adding more provisions. © daniel martin katz michael j bommarito
  • 144. © daniel martin katz michael j bommarito
  • 145. Machine Learning and Judicial Behavior © daniel martin katz michael j bommarito
  • 146. © daniel martin katz michael j bommarito
  • 147. © daniel martin katz michael j bommarito
  • 148. © daniel martin katz michael j bommarito
  • 149. © daniel martin katz michael j bommarito
  • 150. 2002 Prediction Tourney and its limits © daniel martin katz michael j bommarito
  • 151. © daniel martin katz michael j bommarito Model Leverages Classification Tree (Tool from Machine Learning)
  • 152. Standard Decision Tree Often Not Generalizable, Often Overfits the Data © daniel martin katz michael j bommarito
  • 153. Need a more complex approach © daniel martin katz michael j bommarito
  • 154. Predicting the Behavior of the United States Supreme Court: A General Approach © daniel martin katz michael j bommarito Black Reed Frankfurter Douglas Jackson Burton Clark Minton Warren Harlan Brennan Whittaker Stewart White Goldberg Fortas Marshall Burger Blackmun Powell Rehnquist Stevens OConnor Scalia Kennedy Souter Thomas Ginsburg Breyer Roberts Alito Sotomayor Kagan 1953 1963 1973 1983 1993 2003 2013 9-0 Reverse 8-1, 7-2, 6-3 19 19 19 19 19 20 20 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - Reverse 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 - 8-1, 7-2, 6-3 9-0 19 19 19 19 19 20 20
  • 155. feature engineering © daniel martin katz michael j bommarito The real world gives us raw material, at best.  Typically, you even have to dig the stuff raw material out of your own unstructured data
  • 156. similar approach can be applied to other problems © daniel martin katz michael j bommarito
  • 157. © daniel martin katz michael j bommarito
  • 158. © daniel martin katz michael j bommarito Case Prediction and Litigation Data
  • 159. © daniel martin katz michael j bommarito
  • 160. © daniel martin katz michael j bommarito
  • 161. © daniel martin katz michael j bommarito
  • 162. © daniel martin katz michael j bommarito “John Dragseth, a principal at Fish & Richardson (the most active IP litigation firm in the United States, according to Corporate Counsel magazine), credits Lex Machina’s database with helping him spot meaningful but otherwise hidden trends in IP litigation—and he won’t give details. “If you published it, then people on the other side would know,” he says.
  • 163. © daniel martin katz michael j bommarito Notice there is an offloading of data but it is up to the end user to derive meaning
  • 164. © daniel martin katz michael j bommarito In general, the relevant consumer market is not yet mature when it comes to data science
  • 165. © daniel martin katz michael j bommarito Difficult to sell machine learning technology in instances where the end user does not have the right assets in place
  • 166. © daniel martin katz michael j bommarito
  • 167. © daniel martin katz michael j bommarito Many other examples ... just starting to come online
  • 168. © daniel martin katz michael j bommarito Attorney Quality and Performance
  • 170. © daniel martin katz michael j bommarito Leveraging Public Data for Legal Insight
  • 171. © daniel martin katz michael j bommarito
  • 172. © daniel martin katz michael j bommarito
  • 173. Change Management is the Hardest Innovation of All © daniel martin katz michael j bommarito
  • 174. © daniel martin katz michael j bommarito Bulls and Bears ~1984 - 2009 ~2009 - 2014
  • 175. © daniel martin katz michael j bommarito 53 in 2009 58 in 2014 If you were 28 in 1984 than you were
  • 176. © daniel martin katz michael j bommarito before 2009 most of the individuals in the profession have only known the bull market
  • 177. © daniel martin katz michael j bommarito it is a bear market now ... and in a bear market you need a serious strategy
  • 178. © daniel martin katz michael j bommarito analytics/data should be part of that strategy
  • 179. © daniel martin katz michael j bommarito “data is the oil of the 21st Century”
  • 180. So lets be wildcatters
  • 181. © daniel martin katz michael j bommarito law < > finance many elements in law look like finance did 25 years ago
  • 182. © daniel martin katz michael j bommarito
  • 183. © daniel martin katz michael j bommarito When it comes to innovation at the level that is going to be needed ...
  • 184. © daniel martin katz michael j bommarito Assigning a innovation partner or an innovation committee is probably not enough
  • 185. © daniel martin katz michael j bommarito Shunk Works
  • 186. © daniel martin katz michael j bommarito how many organizations have a full time data scientist (data science team)?
  • 187. © daniel martin katz michael j bommarito need a full scale and empowered R+D team (data science, tech, etc.)
  • 188. © daniel martin katz michael j bommarito Final Thought
  • 189. Exit, Voice & Loyalty © daniel martin katz michael j bommarito
  • 190. daniel martin katz michael j bommarito ii adjunct professor of Law @ michigan state university associate professor of law @ illinois tech - chicago kent co-founder @ LexPredict director of research @ reInventLaw laboratory co-founder @ LexPredict Forum on Legal Evolution NYC