SlideShare une entreprise Scribd logo
1  sur  160
legal analytics vs.
empirical legal studies
daniel martin katz
blog | ComputationalLegalStudies.com
corp | LexPredict.com
page | DanielMartinKatz.com
edu | chicago-kent college of law
lab | theLawLab.com
-or- causal inference vs prediction redux
ELS and Legal Analytics
Never the Twain
Shall Meet?
Partners in the
Same Pursuit
-OR-
I thought I might offer
a quick landscape
orientation regarding
terminology, methods, etc.
The ‘Empirical Turn’
in Legal Scholarship
Legal Scholarship
Has become far more ‘empirical’ in nature
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Goal:
Develop optimal
(better) legal rules for
various areas of
human endeavor
Tools:
Use Traditional
Social Science Methods
Instrumental Variables, Propensity Score
Matching, Rubin Causal Model, Regression
Discontinuity, Difference in Differences, etc.
(typically econometric) tools
Outcome:
Determine (as best
possible) whether a
particular policy
intervention achieves
the desired ends
relative to the alternative …
(where the alternative is a
normative essay writing contest)
This represents a material
improvement in the state
of affairs …
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
The Diversity of Tasks
that Lawyers Undertake
There are a diverse
set of tasks which
lawyers undertake …
Lawyer as
Policy Maker,
Appellate
Judge
Lets us divide the space
Lawyer as
Strategist,
Predictor,
Master of
Process
Lawyer as
Policy Maker,
Appellate
Judge
Causal Inference is at the core
of the ‘empirical turn’ that has
taken hold in law as well as the
social sciences
Such Approaches are best for
Appropriate Problems/Questions
where identifying / linking cause
and effect are key
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Some Epistemological
Issues / Questions
Some would make the
epistemological /
methodological
case to be made
that prediction >
causal inference
Part of that case comes from
finance, trading, etc.
where causal inference tools
are generally not used
Andrew D. Martin, Kevin M. Quinn, Theodore W. Ruger & Pauline T. Kim,
Competing Approaches to Predicting Supreme Court Decision Making,
2 Perspectives on Politics 761 (2004).
“the best test of an explanatory theory is its
ability to predict future events. To the extent
that scholars in both disciplines (social
science and law) seek to explain court
behavior, they ought to test their theories
not only against cases already decided, but
against future outcomes as well.”
Andrew D. Martin, Kevin M. Quinn, Theodore W. Ruger & Pauline T. Kim,
Competing Approaches to Predicting Supreme Court Decision Making,
2 Perspectives on Politics 761 (2004).
“the best test of an explanatory theory
is its ability to predict future
events. To the extent that scholars in both
disciplines (social science and law) seek to
explain court behavior, they ought to
test their theories not only against cases
already decided, but against future
outcomes as well.”
Andrew D. Martin, Kevin M. Quinn, Theodore W. Ruger & Pauline T. Kim,
Competing Approaches to Predicting Supreme Court Decision Making,
2 Perspectives on Politics 761 (2004).
“the best test of an explanatory theory is its
ability to predict future events. To the extent
that scholars in both disciplines (social
science and law) seek to explain court
behavior behavior, they ought to test their
theories not only against cases already
decided, but against future outcomes as
well.”
Other folks are starting
to ask similar questions …
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
So I believe that we will
see more efforts in the coming years
to do both backward and ‘forward
causal inference’ in the policy sphere
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
The Other Type
of Work
That Lawyers Do
Lawyer as
Policy Maker,
Appellate
Judge
Lets us divide the space
Lawyer as
Strategist,
Predictor,
Master of
Process
Lawyer as
Strategist,
Predictor,
Master of
Process
This version of the
lawyer taskset is often
directed at trying to
forecast / predict
future events
When you hear prediction
you should think …
#AI #LegalTech
#Machine Learning
#LegalAnalytics
Goal:
Predict the behavior
of some form of legal,
regulatory institution
Tools:
Use Some Blend of
Experts, Crowds, Algorithms
to Forecast Outcomes
Craft Optimal Strategies, etc.
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Quantitative
Legal
Prediction
Historically speaking,
there were practically
zero papers in law
that used any form of
machine learning
There has been
growing interest in
rigorous
There has been
growing interest in
rigorous
There has been
growing interest in
out of sample
rigorous
There has been
growing interest in
prediction in law
out of sample
rigorous
#AI #LegalTech
#Machine Learning
#LegalAnalytics
There has been
growing interest in
prediction in law
out of sample
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
http://journals.plos.org/
plosone/article?id=10.1371/
journal.pone.0174698
available at
RESEARCH ARTICLE
A general approach for predicting the
behavior of the Supreme Court of the United
States
Daniel Martin Katz1,2
*, Michael J. Bommarito II1,2
, Josh Blackman3
1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford
Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston,
Houston, TX, United States of America
* dkatz3@kentlaw.iit.edu
Abstract
Building on developments in machine learning and prior work in the science of judicial pre-
diction, we construct a model designed to predict the behavior of the Supreme Court of the
United States in a generalized, out-of-sample context. To do so, we develop a time-evolving
random forest classifier that leverages unique feature engineering to predict more than
240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015).
Using only data available prior to decision, our model outperforms null (baseline) models at
both the justice and case level under both parametric and non-parametric tests. Over nearly
two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus-
tice vote level. More recently, over the past century, we outperform an in-sample optimized
null model by nearly 5%. Our performance is consistent with, and improves on the general
level of prediction demonstrated by prior work; however, our model is distinctive because it
can be applied out-of-sample to the entire past and future of the Court, not a single term.
Our results represent an important advance for the science of quantitative legal prediction
and portend a range of other potential applications.
Introduction
As the leaves begin to fall each October, the first Monday marks the beginning of another term
for the Supreme Court of the United States. Each term brings with it a series of challenging,
important cases that cover legal questions as diverse as tax law, freedom of speech, patent law,
administrative law, equal protection, and environmental law. In many instances, the Court’s
decisions are meaningful not just for the litigants per se, but for society as a whole.
Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal
and political observers. Every year, newspapers, television and radio pundits, academic jour-
nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular
case. Will the Justices vote based on the political preferences of the President who appointed
them or form a coalition along other dimensions? Will the Court counter expectations with an
unexpected ruling?
PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18
a1111111111
a1111111111
a1111111111
a1111111111
a1111111111
OPEN ACCESS
Citation: Katz DM, Bommarito MJ, II, Blackman J
(2017) A general approach for predicting the
behavior of the Supreme Court of the United
States. PLoS ONE 12(4): e0174698. https://doi.
org/10.1371/journal.pone.0174698
Editor: Luı´s A. Nunes Amaral, Northwestern
University, UNITED STATES
Received: January 17, 2017
Accepted: March 13, 2017
Published: April 12, 2017
Copyright: © 2017 Katz et al. This is an open
access article distributed under the terms of the
Creative Commons Attribution License, which
permits unrestricted use, distribution, and
reproduction in any medium, provided the original
author and source are credited.
Data Availability Statement: Data and replication
code are available on Github at the following URL:
https://github.com/mjbommar/scotus-predict-v2/.
Funding: The author(s) received no specific
funding for this work.
Competing interests: All Authors are Members of
a LexPredict, LLC which provides consulting
services to various legal industry stakeholders. We
received no financial contributions from LexPredict
or anyone else for this paper. This does not alter
our adherence to PLOS ONE policies on sharing
data and materials.
conducting analysis
of legal system(s)
at scale
There has been
growing interest in
“…I study choice of law by
analyzing the nearly 1,000,000
contracts that have been disclosed
to the Securities and Exchange
Commission between 1996–2012.”
In this paper, we analyze over 4.5 million references to
U.S. Federal Acts and Agencies contained within these
10-K reports to build a mean-field measurement of
temperature and diversity in this regulatory ecosystem
There has also been
a significant amount
of commercial interest
linked to legal analytics
For example,
here are just a few
predictions
that lawyers are trying to
accomplish on a daily basis
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
#Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
Data Driven Compliance
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
#Predict Rogue Behavior
#Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Rogue Behavior
Data Driven Compliance
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
#Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
Data Driven Compliance
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
#Predict Regulatory Outcomes
Data Driven Lobbying, etc.
#Predict Rogue Behavior
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Not only law firms but also
the large enterprise clients …
35!
“From!se)lement!informa0on!and!
contracts! to! sensi0ve! client! data!
and! beyond,! Liberty! Mutual!
creates! and! stores! ever:growing!
volumes! of! unorganized! data!
across! its! worldwide! offices! and!
databases.”!
“I've!seen!a!real!transforma0on!in!
the! legal! department! just! having!
t h a t! i n f o r m a 0 o n! v i s u a l l y!
available."!
“The' legal' department' is' now'
w o r k i n g' p r e d i c 7 v e' a n d'
prescrip7ve' analy7cs,"' i.e.' ways'
to' analyze' data' that' enable'
forecas7ng'for'legal'issues.”'
34!
36!
“I"believe"strongly"that"data"analy2cs"is"
a"new"fron2er"in"the"legal"space.”"
Susie!Lees!
General!Counsel!!
Allstate!!
“Leveraging" data," not" only" that" we"
possess" but" that" our" law" firms" have"
amassed"over"the"years,"offers"a"plethora"
of" un<tapped" opportuni=es—not" simply"
to" help" us" forecast" and" manage" legal"
expenses," but" also" to" help" our" clients"
make"more"informed"business"decisions.”"
33!
“Now! we! have! program! managers,!
data! analysts,! business! analysts,!
data! scien9sts,! opera9ons!
managers,!I!mean,!we!have!a!ton!of!
stuff.! That's! the! key! for! me,! is!
thinking! about! the! right! people!
doing! the! right! tasks.! That's! the!
people!part.!And!then!how!they!do!
them,! is! the! process,! and! then,!
automa9ng! parts,! is! kind! of! that!
next,!final!step.!!
"
And$ all$ of$ that$ is$ underpinned$ by$
d a t a ." Y o u$ c a n ' t$ d o$ a n y$
improvements$ unless$ you$ have$
data.$ You$ can't$ automate$ unless$
you$have$good$data.”!
In so much as prediction is
the task in question …
#LegalTech #FinTech
#Fin(Legal)Tech
“The real roll-up of all this isn’t robot lawyers,
it’s financialization, with law becoming an
applied branch of finance and insurance.”
Daniel Martin Katz, professor, Illinois Tech’s Chicago Kent College of Law
http://www.ozy.com/fast-forward/why-artificial-intelligence-might-replace-your-lawyer/75435
#Fin(Legal)Tech
https://computationallegalstudies.com/2016/02/27/fin-legal-tech-
laws-future-from-finances-past-an-expanded-version-of-the-deck/
GO HERE FOR A DETAILED TREATMENT OF THE QUESTION
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
The Three Forms
of (Legal) Prediction
www.legalanalyticscourse.com
In so much as prediction is
the task in question …
#MachineLearing
is the method du jour
It is not necessarily ML
alone but rather some
ensemble of
experts, crowds + algorithms
http://www.sciencemag.org/news/
2017/05/artificial-intelligence-prevails-
predicting-supreme-court-decisions
Professor Katz noted
that in the long term
…“We believe the
blend of experts,
crowds, and
algorithms is the
secret sauce for the
whole thing.”
May 2nd 2017
example from our own work
predicting the decisions of the
Supreme Court of the United States
#SCOTUS
Experts
Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political
Science Approaches to
Predicting Supreme
Court Decision Making
The Supreme Court
Forecasting Project:
experts
Case Level Prediction
Justice Level Prediction
67.4% experts
58% experts
From the 68
Included
Cases
for the
2002-2003
Supreme
Court Term
these experts probably
overfit
they fit to the noise
and
not the signal
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
if this were
finance this
would be
trading
worse than
S&P500
#NoiseTrading
#BuffetChallenge
like many other forms
human endeavor
law is full of 

noise predictors …
we need to
evaluate
legal experts
and
somehow
benchmark
their
expertise
from a pure
forecasting
standpoint
the best
known
SCOTUS
predictor is
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
the law
version of
superforecasting
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Crowds
crowds
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
https://fantasyscotus.lexpredict.com/case/list/
We can
generate
Crowd
Sourced
Predictions
not all
members of
crowd are
made equal
we maintain
a ‘supercrowd’
which is the top n
of predictors
up to time t-1
the
‘supercrowd’
outperforms
the overall
crowd
(and even the
best single player)
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
not
enough
crowd
based
decision
making in
institutions
(law included)
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
“Software developers were asked on
two separate days to estimate the
completion time for a given task, the
hours they projected differed by 71%,
on average.
W h e n p a t h o l o g i s t s m a d e t wo
assessments of the severity of biopsy
results, the correlation between their
ratings was only .61 (out of a perfect
1.0), indicating that they made
inconsistent diagnoses quite frequently.
Judgments made by different people
are even more likely to diverge.”
Brief Aside
About
Crowd
Sourced
Prediction
#LegalCrowdSourcing
(most pundits did not
identify as a serious
candidate him until
mid-January 2017)
Neil Gorsuch was #1
o n o u r F a n t a s y
Platform 12 Days after
Donald Trump was
elected President
(i.e Nov 20)
#FantasySCOTUS
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Algorithms
Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political Science
Approaches to Predicting
Supreme Court Decision
Making
The Supreme Court
Forecasting Project:
Ruger, et al (2004)
relied upon
Brieman(1984)
(as partially shown below)
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
Leo Brieman moved away from
CART in Brieman (2001)
Breiman, L.(2001). Random forests.
Machine learning, 45(1), 5-32.
Published in Machine Learning
(A Springer Science Journal)
One well-known problem with
standard classification trees is
their tendency toward overfitting
http://machinelearning202.pbworks.com/w/file/fetch/37597425/
performanceCompSupervisedLearning-caruana.pdf
Random
Forest
(particularly
with special
config/
optimization)
have proven to
be unreasonably
effective
Random forest is an approach to
aggregate weak learners into
collective strong learners
(using a combo of bagging and random substrates)
(think of it as crowd sourcing of models)
Our algorithm is a special version
of random forest (time evolving)
http://journals.plos.org/
plosone/article?id=10.1371/
journal.pone.0174698
available at
RESEARCH ARTICLE
A general approach for predicting the
behavior of the Supreme Court of the United
States
Daniel Martin Katz1,2
*, Michael J. Bommarito II1,2
, Josh Blackman3
1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford
Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston,
Houston, TX, United States of America
* dkatz3@kentlaw.iit.edu
Abstract
Building on developments in machine learning and prior work in the science of judicial pre-
diction, we construct a model designed to predict the behavior of the Supreme Court of the
United States in a generalized, out-of-sample context. To do so, we develop a time-evolving
random forest classifier that leverages unique feature engineering to predict more than
240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015).
Using only data available prior to decision, our model outperforms null (baseline) models at
both the justice and case level under both parametric and non-parametric tests. Over nearly
two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus-
tice vote level. More recently, over the past century, we outperform an in-sample optimized
null model by nearly 5%. Our performance is consistent with, and improves on the general
level of prediction demonstrated by prior work; however, our model is distinctive because it
can be applied out-of-sample to the entire past and future of the Court, not a single term.
Our results represent an important advance for the science of quantitative legal prediction
and portend a range of other potential applications.
Introduction
As the leaves begin to fall each October, the first Monday marks the beginning of another term
for the Supreme Court of the United States. Each term brings with it a series of challenging,
important cases that cover legal questions as diverse as tax law, freedom of speech, patent law,
administrative law, equal protection, and environmental law. In many instances, the Court’s
decisions are meaningful not just for the litigants per se, but for society as a whole.
Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal
and political observers. Every year, newspapers, television and radio pundits, academic jour-
nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular
case. Will the Justices vote based on the political preferences of the President who appointed
them or form a coalition along other dimensions? Will the Court counter expectations with an
unexpected ruling?
PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18
a1111111111
a1111111111
a1111111111
a1111111111
a1111111111
OPEN ACCESS
Citation: Katz DM, Bommarito MJ, II, Blackman J
(2017) A general approach for predicting the
behavior of the Supreme Court of the United
States. PLoS ONE 12(4): e0174698. https://doi.
org/10.1371/journal.pone.0174698
Editor: Luı´s A. Nunes Amaral, Northwestern
University, UNITED STATES
Received: January 17, 2017
Accepted: March 13, 2017
Published: April 12, 2017
Copyright: © 2017 Katz et al. This is an open
access article distributed under the terms of the
Creative Commons Attribution License, which
permits unrestricted use, distribution, and
reproduction in any medium, provided the original
author and source are credited.
Data Availability Statement: Data and replication
code are available on Github at the following URL:
https://github.com/mjbommar/scotus-predict-v2/.
Funding: The author(s) received no specific
funding for this work.
Competing interests: All Authors are Members of
a LexPredict, LLC which provides consulting
services to various legal industry stakeholders. We
received no financial contributions from LexPredict
or anyone else for this paper. This does not alter
our adherence to PLOS ONE policies on sharing
data and materials.
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
We call this a ‘general’ model
of #SCOTUS Prediction
available at
https://arxiv.org/pdf/1612.03473
Not just interested in accuracy
over a short time window
available at
https://arxiv.org/pdf/1612.03473
A locally tuned model will
typically lead to overfitting
as the dynamics shift
available at
https://arxiv.org/pdf/1612.03473
We want a model that is
robust to a large number
of known dynamics …
available at
https://arxiv.org/pdf/1612.03473
Version 2.02
January 16, 2017
243,882
28,009
Case Outcomes
JusticeVotes
Current Version of #PredictSCOTUS
1816-2015
Version 2.02
January 16, 2017
Current Version of #PredictSCOTUS
1816-2015
case accuracy
70.2%
71.9%
justice accuracy
But are these results ‘good’ ?
What constitutes ‘good’
performance in this context?
We Craft
Three
Alternative
‘Null’
Models
Our Model Against the Null Models
Some commentators had suggested using a heuristic rule of

‘always guess reverse’ as a baseline
(Null Model 1 ) the always guess Reverse model
Turns out it is a lousy
model prior to ~1950
Because the reversal rate
is not stable over time
Our Model Against the Null Models
(Null Model 2 ) memory window = inf
This is our model against Null Model 2
What about memory window that selects the most frequent
historical outcome?
(Green = our model out performs)
Our Model Against the Null Models
(Null Model 3 ) finite memory window = 10
We in-sample optimize using future information to select a
null model that is among the best performing of all null models
as it is using in-sample info this is a deeply unfair null
Over past century, we outperform
M=10 by nearly 5% and have
significant temporal stability at both
the justice, case, term level
Experts, Crowds, Algorithms
For most problems ...
ensembles of these streams
outperform any single stream
the non-trivial question
is how to optimally assemble
such streams for particular problems
Humans
+
Machines
Humans
+
Machines
>
Humans
+
Machines
Humans
or
Machines
>
Here is what we are
working on right now …
expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ensemble method
ensemble model
expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ensemble method
ensemble model
via back testing we can learn the weights
to apply for particular problems
By the way, you
might ask why does
one care about
marginal improvements
in prediction ?
#Fin(Legal)Tech
Given our ability to offer
forecasts of judicial
outcomes, we wondered
if this information could
inform an event driven
trading strategy ?
Revise + Resubmit @
http://arxiv.org/abs/1508.05751
available at
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux
In Summary …
The Three Forms
of (Legal) Prediction
The ‘Empirical Turn’
in Legal Scholarship
The Diversity of Tasks
that Lawyers Undertake
Some Epistemological
Issues / Questions
The Other Type
of Work
That Lawyers Do
Quantitative
Legal
Prediction
thelawlab.com
LexPredict.com
ComputationalLegalStudies.com
BLOG
@ computational
Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chicago kent college of law@
thelawlab.com

Contenu connexe

Tendances

Information technology Act with Cyber offences .pptx
Information technology Act with Cyber offences .pptxInformation technology Act with Cyber offences .pptx
Information technology Act with Cyber offences .pptxRahul Bharati
 
Artificial Intelligence and Intellectual Property
Artificial Intelligence and Intellectual PropertyArtificial Intelligence and Intellectual Property
Artificial Intelligence and Intellectual PropertyArul Scaria
 
Cyber laws - Ritu Gautam
Cyber laws - Ritu GautamCyber laws - Ritu Gautam
Cyber laws - Ritu GautamRitu Gautam
 
Intellectual property rights in cyberspace
Intellectual property rights in cyberspaceIntellectual property rights in cyberspace
Intellectual property rights in cyberspaceRistya Anditha
 
Ai and the Practice of Law
Ai and the Practice of LawAi and the Practice of Law
Ai and the Practice of LawJay Deragon
 
AI on the Case: Legal and Ethical Issues
AI on the Case:  Legal and Ethical IssuesAI on the Case:  Legal and Ethical Issues
AI on the Case: Legal and Ethical IssuesRichard Austin
 
20CS2024 Ethics in Information Technology
20CS2024 Ethics in Information Technology20CS2024 Ethics in Information Technology
20CS2024 Ethics in Information TechnologyKathirvel Ayyaswamy
 
Data Privacy Compliance (Series: Corporate & Regulatory Compliance Boot Camp)
Data Privacy Compliance (Series: Corporate & Regulatory Compliance Boot Camp)Data Privacy Compliance (Series: Corporate & Regulatory Compliance Boot Camp)
Data Privacy Compliance (Series: Corporate & Regulatory Compliance Boot Camp)Financial Poise
 
Trademarks in Cyberspace: Domain name disputes, cybersquatting and internet i...
Trademarks in Cyberspace: Domain name disputes, cybersquatting and internet i...Trademarks in Cyberspace: Domain name disputes, cybersquatting and internet i...
Trademarks in Cyberspace: Domain name disputes, cybersquatting and internet i...garypierson
 
E contracts and validity of e contracts in India
E contracts and validity of e contracts in IndiaE contracts and validity of e contracts in India
E contracts and validity of e contracts in IndiaKajalRandhawa
 
General Data Protection Regulation (GDPR) Compliance
General Data Protection Regulation (GDPR) ComplianceGeneral Data Protection Regulation (GDPR) Compliance
General Data Protection Regulation (GDPR) Complianceaccenture
 
Cyber crime - and digital device.pptx
Cyber crime - and digital device.pptxCyber crime - and digital device.pptx
Cyber crime - and digital device.pptxAlAsad4
 
Need And Importance Of Cyber Law
Need And Importance Of Cyber LawNeed And Importance Of Cyber Law
Need And Importance Of Cyber LawPoonam Bhasin
 
Data & Privacy: Striking the Right Balance - Jonny Leroy
Data & Privacy: Striking the Right Balance - Jonny LeroyData & Privacy: Striking the Right Balance - Jonny Leroy
Data & Privacy: Striking the Right Balance - Jonny LeroyThoughtworks
 

Tendances (20)

Trade Secrets Law
Trade Secrets LawTrade Secrets Law
Trade Secrets Law
 
The Meaning of Patent Infringement and Patent Litigation
The Meaning of Patent Infringement and Patent LitigationThe Meaning of Patent Infringement and Patent Litigation
The Meaning of Patent Infringement and Patent Litigation
 
Information technology Act with Cyber offences .pptx
Information technology Act with Cyber offences .pptxInformation technology Act with Cyber offences .pptx
Information technology Act with Cyber offences .pptx
 
Artificial Intelligence and Intellectual Property
Artificial Intelligence and Intellectual PropertyArtificial Intelligence and Intellectual Property
Artificial Intelligence and Intellectual Property
 
Cyber laws - Ritu Gautam
Cyber laws - Ritu GautamCyber laws - Ritu Gautam
Cyber laws - Ritu Gautam
 
Intellectual property rights in cyberspace
Intellectual property rights in cyberspaceIntellectual property rights in cyberspace
Intellectual property rights in cyberspace
 
Ai and the Practice of Law
Ai and the Practice of LawAi and the Practice of Law
Ai and the Practice of Law
 
AI on the Case: Legal and Ethical Issues
AI on the Case:  Legal and Ethical IssuesAI on the Case:  Legal and Ethical Issues
AI on the Case: Legal and Ethical Issues
 
20CS2024 Ethics in Information Technology
20CS2024 Ethics in Information Technology20CS2024 Ethics in Information Technology
20CS2024 Ethics in Information Technology
 
Data Privacy Compliance (Series: Corporate & Regulatory Compliance Boot Camp)
Data Privacy Compliance (Series: Corporate & Regulatory Compliance Boot Camp)Data Privacy Compliance (Series: Corporate & Regulatory Compliance Boot Camp)
Data Privacy Compliance (Series: Corporate & Regulatory Compliance Boot Camp)
 
Trademarks in Cyberspace: Domain name disputes, cybersquatting and internet i...
Trademarks in Cyberspace: Domain name disputes, cybersquatting and internet i...Trademarks in Cyberspace: Domain name disputes, cybersquatting and internet i...
Trademarks in Cyberspace: Domain name disputes, cybersquatting and internet i...
 
Intellectual property audits
Intellectual property auditsIntellectual property audits
Intellectual property audits
 
Cyber law
Cyber lawCyber law
Cyber law
 
E contracts and validity of e contracts in India
E contracts and validity of e contracts in IndiaE contracts and validity of e contracts in India
E contracts and validity of e contracts in India
 
General Data Protection Regulation (GDPR) Compliance
General Data Protection Regulation (GDPR) ComplianceGeneral Data Protection Regulation (GDPR) Compliance
General Data Protection Regulation (GDPR) Compliance
 
Cyber crime - and digital device.pptx
Cyber crime - and digital device.pptxCyber crime - and digital device.pptx
Cyber crime - and digital device.pptx
 
Need And Importance Of Cyber Law
Need And Importance Of Cyber LawNeed And Importance Of Cyber Law
Need And Importance Of Cyber Law
 
Cyber law
Cyber lawCyber law
Cyber law
 
Data & Privacy: Striking the Right Balance - Jonny Leroy
Data & Privacy: Striking the Right Balance - Jonny LeroyData & Privacy: Striking the Right Balance - Jonny Leroy
Data & Privacy: Striking the Right Balance - Jonny Leroy
 
It act ppt ( 1111)
It act ppt ( 1111)It act ppt ( 1111)
It act ppt ( 1111)
 

En vedette

Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...Daniel Katz
 
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...Daniel Katz
 
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010Daniel Katz
 
Complex Systems Computing - Webscraping - Bonus Module
Complex Systems Computing - Webscraping - Bonus ModuleComplex Systems Computing - Webscraping - Bonus Module
Complex Systems Computing - Webscraping - Bonus ModuleDaniel Katz
 
Legal Language Explorer .com Tutorial
Legal Language Explorer .com  TutorialLegal Language Explorer .com  Tutorial
Legal Language Explorer .com TutorialDaniel Katz
 
Legal Analytics, Machine Learning and Some Comments on the Status of Innovat...
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovat... Legal Analytics, Machine Learning and Some Comments on the Status of Innovat...
Legal Analytics, Machine Learning and Some Comments on the Status of Innovat...Daniel Katz
 
Building Your Personal (Legal) Brand - Some Thoughts for Law Students and Oth...
Building Your Personal (Legal) Brand - Some Thoughts for Law Students and Oth...Building Your Personal (Legal) Brand - Some Thoughts for Law Students and Oth...
Building Your Personal (Legal) Brand - Some Thoughts for Law Students and Oth...Daniel Katz
 
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Daniel Katz
 
Machine Learning as a Service: #MLaaS, Open Source and the Future of (Legal) ...
Machine Learning as a Service: #MLaaS, Open Source and the Future of (Legal) ...Machine Learning as a Service: #MLaaS, Open Source and the Future of (Legal) ...
Machine Learning as a Service: #MLaaS, Open Source and the Future of (Legal) ...Daniel Katz
 
Schellingcodemap
SchellingcodemapSchellingcodemap
SchellingcodemapDaniel Katz
 
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...Daniel Katz
 
Innovation in the Legal Services Industry - "The Future is Already Here, It i...
Innovation in the Legal Services Industry - "The Future is Already Here, It i...Innovation in the Legal Services Industry - "The Future is Already Here, It i...
Innovation in the Legal Services Industry - "The Future is Already Here, It i...Daniel Katz
 
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...Daniel Katz
 

En vedette (13)

Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
Exploring the Physical Properties of Regulatory Ecosystems - Professors Danie...
 
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
Measuring the Complexity of the Law: The United States Code ( Slides by Danie...
 
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
Sinks Method Paper Presentation @ Duke Political Networks Conference 2010
 
Complex Systems Computing - Webscraping - Bonus Module
Complex Systems Computing - Webscraping - Bonus ModuleComplex Systems Computing - Webscraping - Bonus Module
Complex Systems Computing - Webscraping - Bonus Module
 
Legal Language Explorer .com Tutorial
Legal Language Explorer .com  TutorialLegal Language Explorer .com  Tutorial
Legal Language Explorer .com Tutorial
 
Legal Analytics, Machine Learning and Some Comments on the Status of Innovat...
 Legal Analytics, Machine Learning and Some Comments on the Status of Innovat... Legal Analytics, Machine Learning and Some Comments on the Status of Innovat...
Legal Analytics, Machine Learning and Some Comments on the Status of Innovat...
 
Building Your Personal (Legal) Brand - Some Thoughts for Law Students and Oth...
Building Your Personal (Legal) Brand - Some Thoughts for Law Students and Oth...Building Your Personal (Legal) Brand - Some Thoughts for Law Students and Oth...
Building Your Personal (Legal) Brand - Some Thoughts for Law Students and Oth...
 
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
Quantitative Methods for Lawyers - Class #14 - Power Laws, Hypothesis Testing...
 
Machine Learning as a Service: #MLaaS, Open Source and the Future of (Legal) ...
Machine Learning as a Service: #MLaaS, Open Source and the Future of (Legal) ...Machine Learning as a Service: #MLaaS, Open Source and the Future of (Legal) ...
Machine Learning as a Service: #MLaaS, Open Source and the Future of (Legal) ...
 
Schellingcodemap
SchellingcodemapSchellingcodemap
Schellingcodemap
 
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
Thomas Schelling Segregation Model - An Exercise in Mapping the Dependencies ...
 
Innovation in the Legal Services Industry - "The Future is Already Here, It i...
Innovation in the Legal Services Industry - "The Future is Already Here, It i...Innovation in the Legal Services Industry - "The Future is Already Here, It i...
Innovation in the Legal Services Industry - "The Future is Already Here, It i...
 
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
Law + Complexity & Prediction: Toward a Characterization of Legal Systems as ...
 

Similaire à Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux

The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...Daniel Katz
 
The Constitutional Principle Of The Rule Of Law
The Constitutional Principle Of The Rule Of LawThe Constitutional Principle Of The Rule Of Law
The Constitutional Principle Of The Rule Of LawMegan Moore
 
The Role Of Lawyers And Interest Of The Judicial Process
The Role Of Lawyers And Interest Of The Judicial ProcessThe Role Of Lawyers And Interest Of The Judicial Process
The Role Of Lawyers And Interest Of The Judicial ProcessElaine Ake
 
3222020 Prediction, persuasion, and the jurisprudence of beh.docx
3222020 Prediction, persuasion, and the jurisprudence of beh.docx3222020 Prediction, persuasion, and the jurisprudence of beh.docx
3222020 Prediction, persuasion, and the jurisprudence of beh.docxlorainedeserre
 
3222020 Prediction, persuasion, and the jurisprudence of beh.docx
3222020 Prediction, persuasion, and the jurisprudence of beh.docx3222020 Prediction, persuasion, and the jurisprudence of beh.docx
3222020 Prediction, persuasion, and the jurisprudence of beh.docxpriestmanmable
 
ACTL_Journal_Issue_76
ACTL_Journal_Issue_76ACTL_Journal_Issue_76
ACTL_Journal_Issue_76Eliza Gano
 
A View from the Sky A General Overview about Civil Litigation in.pdf
A View from the Sky A General Overview about Civil Litigation in.pdfA View from the Sky A General Overview about Civil Litigation in.pdf
A View from the Sky A General Overview about Civil Litigation in.pdfCarrie Tran
 
Can Law Librarians Help Law Become More Data Driven ? An Open Question in Ne...
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Ne...Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Ne...
Can Law Librarians Help Law Become More Data Driven ? An Open Question in Ne...Daniel Katz
 
Prior to beginning work on this assignment, please review the articl.docx
Prior to beginning work on this assignment, please review the articl.docxPrior to beginning work on this assignment, please review the articl.docx
Prior to beginning work on this assignment, please review the articl.docxkeilenettie
 
Legal Validity Of Natural Law
Legal Validity Of Natural LawLegal Validity Of Natural Law
Legal Validity Of Natural LawChristi Miller
 
A survey of electronic research alternatives to lexis and westlaw in law firms
A survey of electronic research alternatives to lexis and westlaw in law firmsA survey of electronic research alternatives to lexis and westlaw in law firms
A survey of electronic research alternatives to lexis and westlaw in law firmsDillard University Library
 
Cognitive Legal Science V5
Cognitive Legal Science  V5Cognitive Legal Science  V5
Cognitive Legal Science V5Howard Moskowitz
 
Advantages Of Adversarial Legal System
Advantages Of Adversarial Legal SystemAdvantages Of Adversarial Legal System
Advantages Of Adversarial Legal SystemDearney Wartenbee
 
What is the Psychoanalysis of Law
What is the Psychoanalysis of LawWhat is the Psychoanalysis of Law
What is the Psychoanalysis of Lawiosrjce
 
Why Should The Uk Constitution Uncodified And Should...
Why Should The Uk Constitution Uncodified And Should...Why Should The Uk Constitution Uncodified And Should...
Why Should The Uk Constitution Uncodified And Should...Sherry Bailey
 
Court Decisions - Law and Public PolicyFor the Discussion this wee.docx
Court Decisions - Law and Public PolicyFor the Discussion this wee.docxCourt Decisions - Law and Public PolicyFor the Discussion this wee.docx
Court Decisions - Law and Public PolicyFor the Discussion this wee.docxmarilucorr
 

Similaire à Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux (20)

13Th Amendment
13Th Amendment13Th Amendment
13Th Amendment
 
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
The Three Forms of (Legal) Prediction: Experts, Crowds and Algorithms -- Prof...
 
The Constitutional Principle Of The Rule Of Law
The Constitutional Principle Of The Rule Of LawThe Constitutional Principle Of The Rule Of Law
The Constitutional Principle Of The Rule Of Law
 
The Role Of Lawyers And Interest Of The Judicial Process
The Role Of Lawyers And Interest Of The Judicial ProcessThe Role Of Lawyers And Interest Of The Judicial Process
The Role Of Lawyers And Interest Of The Judicial Process
 
3222020 Prediction, persuasion, and the jurisprudence of beh.docx
3222020 Prediction, persuasion, and the jurisprudence of beh.docx3222020 Prediction, persuasion, and the jurisprudence of beh.docx
3222020 Prediction, persuasion, and the jurisprudence of beh.docx
 
3222020 Prediction, persuasion, and the jurisprudence of beh.docx
3222020 Prediction, persuasion, and the jurisprudence of beh.docx3222020 Prediction, persuasion, and the jurisprudence of beh.docx
3222020 Prediction, persuasion, and the jurisprudence of beh.docx
 
Francis X. Shen, "The Neurolaw Revolution"
Francis X. Shen, "The Neurolaw Revolution"Francis X. Shen, "The Neurolaw Revolution"
Francis X. Shen, "The Neurolaw Revolution"
 
ACTL_76Journal_FINALONLINE
ACTL_76Journal_FINALONLINEACTL_76Journal_FINALONLINE
ACTL_76Journal_FINALONLINE
 
ACTL_Journal_Issue_76
ACTL_Journal_Issue_76ACTL_Journal_Issue_76
ACTL_Journal_Issue_76
 
A View from the Sky A General Overview about Civil Litigation in.pdf
A View from the Sky A General Overview about Civil Litigation in.pdfA View from the Sky A General Overview about Civil Litigation in.pdf
A View from the Sky A General Overview about Civil Litigation in.pdf
 
Can Law Librarians Help Law Become More Data Driven ? An Open Question in Ne...
Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Ne...Can Law Librarians Help Law Become More Data Driven ?  An Open Question in Ne...
Can Law Librarians Help Law Become More Data Driven ? An Open Question in Ne...
 
Prior to beginning work on this assignment, please review the articl.docx
Prior to beginning work on this assignment, please review the articl.docxPrior to beginning work on this assignment, please review the articl.docx
Prior to beginning work on this assignment, please review the articl.docx
 
Legal Validity Of Natural Law
Legal Validity Of Natural LawLegal Validity Of Natural Law
Legal Validity Of Natural Law
 
A survey of electronic research alternatives to lexis and westlaw in law firms
A survey of electronic research alternatives to lexis and westlaw in law firmsA survey of electronic research alternatives to lexis and westlaw in law firms
A survey of electronic research alternatives to lexis and westlaw in law firms
 
How to Do a Legal Research: Definition, Types, Examples, Methodology - Legodesk
How to Do a Legal Research: Definition, Types, Examples, Methodology - LegodeskHow to Do a Legal Research: Definition, Types, Examples, Methodology - Legodesk
How to Do a Legal Research: Definition, Types, Examples, Methodology - Legodesk
 
Cognitive Legal Science V5
Cognitive Legal Science  V5Cognitive Legal Science  V5
Cognitive Legal Science V5
 
Advantages Of Adversarial Legal System
Advantages Of Adversarial Legal SystemAdvantages Of Adversarial Legal System
Advantages Of Adversarial Legal System
 
What is the Psychoanalysis of Law
What is the Psychoanalysis of LawWhat is the Psychoanalysis of Law
What is the Psychoanalysis of Law
 
Why Should The Uk Constitution Uncodified And Should...
Why Should The Uk Constitution Uncodified And Should...Why Should The Uk Constitution Uncodified And Should...
Why Should The Uk Constitution Uncodified And Should...
 
Court Decisions - Law and Public PolicyFor the Discussion this wee.docx
Court Decisions - Law and Public PolicyFor the Discussion this wee.docxCourt Decisions - Law and Public PolicyFor the Discussion this wee.docx
Court Decisions - Law and Public PolicyFor the Discussion this wee.docx
 

Plus de Daniel Katz

Why We Are Open Sourcing ContraxSuite and Some Thoughts About Legal Tech and ...
Why We Are Open Sourcing ContraxSuite and Some Thoughts About Legal Tech and ...Why We Are Open Sourcing ContraxSuite and Some Thoughts About Legal Tech and ...
Why We Are Open Sourcing ContraxSuite and Some Thoughts About Legal Tech and ...Daniel Katz
 
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...Daniel Katz
 
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Tog...
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Tog...Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Tog...
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Tog...Daniel Katz
 
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...Daniel Katz
 
LexPredict - Empowering the Future of Legal Decision Making
LexPredict - Empowering the Future of Legal Decision MakingLexPredict - Empowering the Future of Legal Decision Making
LexPredict - Empowering the Future of Legal Decision MakingDaniel Katz
 
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...Daniel Katz
 
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...Daniel Katz
 
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...Daniel Katz
 
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...Daniel Katz
 
Legal Analytics Course - Class 9 - Clustering Algorithms (K-Means & Hierarch...
Legal Analytics Course - Class 9 -  Clustering Algorithms (K-Means & Hierarch...Legal Analytics Course - Class 9 -  Clustering Algorithms (K-Means & Hierarch...
Legal Analytics Course - Class 9 - Clustering Algorithms (K-Means & Hierarch...Daniel Katz
 
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...Daniel Katz
 
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...Daniel Katz
 
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Daniel Katz
 
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...Daniel Katz
 
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...Daniel Katz
 
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...Daniel Katz
 
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...Daniel Katz
 
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5Daniel Katz
 

Plus de Daniel Katz (18)

Why We Are Open Sourcing ContraxSuite and Some Thoughts About Legal Tech and ...
Why We Are Open Sourcing ContraxSuite and Some Thoughts About Legal Tech and ...Why We Are Open Sourcing ContraxSuite and Some Thoughts About Legal Tech and ...
Why We Are Open Sourcing ContraxSuite and Some Thoughts About Legal Tech and ...
 
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
Fin (Legal) Tech – Law’s Future from Finance’s Past (Some Thoughts About the ...
 
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Tog...
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Tog...Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Tog...
Measure Twice, Cut Once - Solving the Legal Profession Biggest Challenges Tog...
 
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
Technology, Data and Computation Session @ The World Bank - Law, Justice, and...
 
LexPredict - Empowering the Future of Legal Decision Making
LexPredict - Empowering the Future of Legal Decision MakingLexPredict - Empowering the Future of Legal Decision Making
LexPredict - Empowering the Future of Legal Decision Making
 
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
Legal Analytics Course - Class 11 - Network Analysis and Law - Professors Dan...
 
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
Legal Analytics Course - Class 12 - Data Preprocessing using dPlyR - Professo...
 
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
Legal Analytics Course - Class 10 - Information Visualization + DataViz in R ...
 
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
Legal Analytics Course - Class #4 - Github and RMarkdown Tutorial - Professor...
 
Legal Analytics Course - Class 9 - Clustering Algorithms (K-Means & Hierarch...
Legal Analytics Course - Class 9 -  Clustering Algorithms (K-Means & Hierarch...Legal Analytics Course - Class 9 -  Clustering Algorithms (K-Means & Hierarch...
Legal Analytics Course - Class 9 - Clustering Algorithms (K-Means & Hierarch...
 
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
Legal Analytics Course - Class 8 - Introduction to Random Forests and Ensembl...
 
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
Legal Analytics Course - Class 7 - Binary Classification with Decision Tree L...
 
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
 
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
Legal Analytics Course - Class 5 - Quantitative Legal Prediction + Data Drive...
 
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
Legal Analytics Course - Class #2 - Introduction to Machine Learning for Lawy...
 
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
Quantitative Methods for Lawyers - R Boot Camp Bonus Module - Professor Danie...
 
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
Quantitative Methods for Lawyers - Class #15 - Chi Square Distribution and Ch...
 
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
Quantitative Methods for Lawyers - Class #22 - Regression Analysis - Part 5
 

Dernier

Classification of Contracts in Business Regulations
Classification of Contracts in Business RegulationsClassification of Contracts in Business Regulations
Classification of Contracts in Business RegulationsSyedaAyeshaTabassum1
 
Women and the World of Climate Change- A Conceptual Foundation by Shraddha Pa...
Women and the World of Climate Change- A Conceptual Foundation by Shraddha Pa...Women and the World of Climate Change- A Conceptual Foundation by Shraddha Pa...
Women and the World of Climate Change- A Conceptual Foundation by Shraddha Pa...SHRADDHA PANDIT
 
Islamabad High Court Judges wrote a letter to Supreme Judicial Council.pdf
Islamabad High Court Judges wrote a letter to Supreme Judicial Council.pdfIslamabad High Court Judges wrote a letter to Supreme Judicial Council.pdf
Islamabad High Court Judges wrote a letter to Supreme Judicial Council.pdfNo One
 
ArtificiaI Intelligence based Cyber Forensic Tools: Relevancy and Admissibili...
ArtificiaI Intelligence based Cyber Forensic Tools: Relevancy and Admissibili...ArtificiaI Intelligence based Cyber Forensic Tools: Relevancy and Admissibili...
ArtificiaI Intelligence based Cyber Forensic Tools: Relevancy and Admissibili...Anadi Tewari
 
An introduction to Indian Contract Act, 1872 by Shraddha Pandit
An introduction to Indian Contract Act, 1872 by Shraddha PanditAn introduction to Indian Contract Act, 1872 by Shraddha Pandit
An introduction to Indian Contract Act, 1872 by Shraddha PanditSHRADDHA PANDIT
 
Law-on-Partnership-and-Corporation business org
Law-on-Partnership-and-Corporation business orgLaw-on-Partnership-and-Corporation business org
Law-on-Partnership-and-Corporation business orgAnonymousUKTzN2ggtG
 
Embed-7.pdfp;kpokipppedoioediouedooedijed
Embed-7.pdfp;kpokipppedoioediouedooedijedEmbed-7.pdfp;kpokipppedoioediouedooedijed
Embed-7.pdfp;kpokipppedoioediouedooedijedbhavenpr
 
Corporate Sustainability Due Diligence Directive (CSDDD or the EU Supply Chai...
Corporate Sustainability Due Diligence Directive (CSDDD or the EU Supply Chai...Corporate Sustainability Due Diligence Directive (CSDDD or the EU Supply Chai...
Corporate Sustainability Due Diligence Directive (CSDDD or the EU Supply Chai...Dr. Oliver Massmann
 
The Ultimate Guide to Drafting Your Separation Agreement with a Template
The Ultimate Guide to Drafting Your Separation Agreement with a TemplateThe Ultimate Guide to Drafting Your Separation Agreement with a Template
The Ultimate Guide to Drafting Your Separation Agreement with a TemplateBTL Law P.C.
 

Dernier (9)

Classification of Contracts in Business Regulations
Classification of Contracts in Business RegulationsClassification of Contracts in Business Regulations
Classification of Contracts in Business Regulations
 
Women and the World of Climate Change- A Conceptual Foundation by Shraddha Pa...
Women and the World of Climate Change- A Conceptual Foundation by Shraddha Pa...Women and the World of Climate Change- A Conceptual Foundation by Shraddha Pa...
Women and the World of Climate Change- A Conceptual Foundation by Shraddha Pa...
 
Islamabad High Court Judges wrote a letter to Supreme Judicial Council.pdf
Islamabad High Court Judges wrote a letter to Supreme Judicial Council.pdfIslamabad High Court Judges wrote a letter to Supreme Judicial Council.pdf
Islamabad High Court Judges wrote a letter to Supreme Judicial Council.pdf
 
ArtificiaI Intelligence based Cyber Forensic Tools: Relevancy and Admissibili...
ArtificiaI Intelligence based Cyber Forensic Tools: Relevancy and Admissibili...ArtificiaI Intelligence based Cyber Forensic Tools: Relevancy and Admissibili...
ArtificiaI Intelligence based Cyber Forensic Tools: Relevancy and Admissibili...
 
An introduction to Indian Contract Act, 1872 by Shraddha Pandit
An introduction to Indian Contract Act, 1872 by Shraddha PanditAn introduction to Indian Contract Act, 1872 by Shraddha Pandit
An introduction to Indian Contract Act, 1872 by Shraddha Pandit
 
Law-on-Partnership-and-Corporation business org
Law-on-Partnership-and-Corporation business orgLaw-on-Partnership-and-Corporation business org
Law-on-Partnership-and-Corporation business org
 
Embed-7.pdfp;kpokipppedoioediouedooedijed
Embed-7.pdfp;kpokipppedoioediouedooedijedEmbed-7.pdfp;kpokipppedoioediouedooedijed
Embed-7.pdfp;kpokipppedoioediouedooedijed
 
Corporate Sustainability Due Diligence Directive (CSDDD or the EU Supply Chai...
Corporate Sustainability Due Diligence Directive (CSDDD or the EU Supply Chai...Corporate Sustainability Due Diligence Directive (CSDDD or the EU Supply Chai...
Corporate Sustainability Due Diligence Directive (CSDDD or the EU Supply Chai...
 
The Ultimate Guide to Drafting Your Separation Agreement with a Template
The Ultimate Guide to Drafting Your Separation Agreement with a TemplateThe Ultimate Guide to Drafting Your Separation Agreement with a Template
The Ultimate Guide to Drafting Your Separation Agreement with a Template
 

Legal Analytics versus Empirical Legal Studies - or - Causal Inference vs Prediction Redux

  • 1. legal analytics vs. empirical legal studies daniel martin katz blog | ComputationalLegalStudies.com corp | LexPredict.com page | DanielMartinKatz.com edu | chicago-kent college of law lab | theLawLab.com -or- causal inference vs prediction redux
  • 2. ELS and Legal Analytics Never the Twain Shall Meet? Partners in the Same Pursuit -OR-
  • 3. I thought I might offer a quick landscape orientation regarding terminology, methods, etc.
  • 4. The ‘Empirical Turn’ in Legal Scholarship
  • 5. Legal Scholarship Has become far more ‘empirical’ in nature
  • 8. Goal: Develop optimal (better) legal rules for various areas of human endeavor
  • 9. Tools: Use Traditional Social Science Methods Instrumental Variables, Propensity Score Matching, Rubin Causal Model, Regression Discontinuity, Difference in Differences, etc. (typically econometric) tools
  • 10. Outcome: Determine (as best possible) whether a particular policy intervention achieves the desired ends
  • 11. relative to the alternative … (where the alternative is a normative essay writing contest)
  • 12. This represents a material improvement in the state of affairs …
  • 14. The Diversity of Tasks that Lawyers Undertake
  • 15. There are a diverse set of tasks which lawyers undertake …
  • 16. Lawyer as Policy Maker, Appellate Judge Lets us divide the space Lawyer as Strategist, Predictor, Master of Process
  • 18. Causal Inference is at the core of the ‘empirical turn’ that has taken hold in law as well as the social sciences
  • 19. Such Approaches are best for Appropriate Problems/Questions where identifying / linking cause and effect are key
  • 22. Some would make the epistemological / methodological case to be made that prediction > causal inference
  • 23. Part of that case comes from finance, trading, etc. where causal inference tools are generally not used
  • 24. Andrew D. Martin, Kevin M. Quinn, Theodore W. Ruger & Pauline T. Kim, Competing Approaches to Predicting Supreme Court Decision Making, 2 Perspectives on Politics 761 (2004). “the best test of an explanatory theory is its ability to predict future events. To the extent that scholars in both disciplines (social science and law) seek to explain court behavior, they ought to test their theories not only against cases already decided, but against future outcomes as well.”
  • 25. Andrew D. Martin, Kevin M. Quinn, Theodore W. Ruger & Pauline T. Kim, Competing Approaches to Predicting Supreme Court Decision Making, 2 Perspectives on Politics 761 (2004). “the best test of an explanatory theory is its ability to predict future events. To the extent that scholars in both disciplines (social science and law) seek to explain court behavior, they ought to test their theories not only against cases already decided, but against future outcomes as well.”
  • 26. Andrew D. Martin, Kevin M. Quinn, Theodore W. Ruger & Pauline T. Kim, Competing Approaches to Predicting Supreme Court Decision Making, 2 Perspectives on Politics 761 (2004). “the best test of an explanatory theory is its ability to predict future events. To the extent that scholars in both disciplines (social science and law) seek to explain court behavior behavior, they ought to test their theories not only against cases already decided, but against future outcomes as well.”
  • 27. Other folks are starting to ask similar questions …
  • 32. So I believe that we will see more efforts in the coming years to do both backward and ‘forward causal inference’ in the policy sphere
  • 34. The Other Type of Work That Lawyers Do
  • 35. Lawyer as Policy Maker, Appellate Judge Lets us divide the space Lawyer as Strategist, Predictor, Master of Process
  • 37. This version of the lawyer taskset is often directed at trying to forecast / predict future events
  • 38. When you hear prediction you should think … #AI #LegalTech #Machine Learning #LegalAnalytics
  • 39. Goal: Predict the behavior of some form of legal, regulatory institution
  • 40. Tools: Use Some Blend of Experts, Crowds, Algorithms to Forecast Outcomes Craft Optimal Strategies, etc.
  • 43. Historically speaking, there were practically zero papers in law that used any form of machine learning
  • 44. There has been growing interest in
  • 46. rigorous There has been growing interest in out of sample
  • 47. rigorous There has been growing interest in prediction in law out of sample
  • 48. rigorous #AI #LegalTech #Machine Learning #LegalAnalytics There has been growing interest in prediction in law out of sample
  • 54. http://journals.plos.org/ plosone/article?id=10.1371/ journal.pone.0174698 available at RESEARCH ARTICLE A general approach for predicting the behavior of the Supreme Court of the United States Daniel Martin Katz1,2 *, Michael J. Bommarito II1,2 , Josh Blackman3 1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston, Houston, TX, United States of America * dkatz3@kentlaw.iit.edu Abstract Building on developments in machine learning and prior work in the science of judicial pre- diction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus- tice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications. Introduction As the leaves begin to fall each October, the first Monday marks the beginning of another term for the Supreme Court of the United States. Each term brings with it a series of challenging, important cases that cover legal questions as diverse as tax law, freedom of speech, patent law, administrative law, equal protection, and environmental law. In many instances, the Court’s decisions are meaningful not just for the litigants per se, but for society as a whole. Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal and political observers. Every year, newspapers, television and radio pundits, academic jour- nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular case. Will the Justices vote based on the political preferences of the President who appointed them or form a coalition along other dimensions? Will the Court counter expectations with an unexpected ruling? PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Katz DM, Bommarito MJ, II, Blackman J (2017) A general approach for predicting the behavior of the Supreme Court of the United States. PLoS ONE 12(4): e0174698. https://doi. org/10.1371/journal.pone.0174698 Editor: Luı´s A. Nunes Amaral, Northwestern University, UNITED STATES Received: January 17, 2017 Accepted: March 13, 2017 Published: April 12, 2017 Copyright: © 2017 Katz et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: Data and replication code are available on Github at the following URL: https://github.com/mjbommar/scotus-predict-v2/. Funding: The author(s) received no specific funding for this work. Competing interests: All Authors are Members of a LexPredict, LLC which provides consulting services to various legal industry stakeholders. We received no financial contributions from LexPredict or anyone else for this paper. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
  • 55. conducting analysis of legal system(s) at scale There has been growing interest in
  • 56. “…I study choice of law by analyzing the nearly 1,000,000 contracts that have been disclosed to the Securities and Exchange Commission between 1996–2012.”
  • 57. In this paper, we analyze over 4.5 million references to U.S. Federal Acts and Agencies contained within these 10-K reports to build a mean-field measurement of temperature and diversity in this regulatory ecosystem
  • 58. There has also been a significant amount of commercial interest linked to legal analytics
  • 59. For example, here are just a few predictions that lawyers are trying to accomplish on a daily basis
  • 60. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding)
  • 61. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Contract Terms/Outcomes Data Driven Transactional Work
  • 62. #Predict Relevant Documents Data Driven EDiscovery/Due Diligence (Predictive Coding) Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work #Predict Rogue Behavior
  • 63. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Rogue Behavior Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work
  • 64. #Predict Relevant Documents #Predict Case Outcomes Data Driven Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work #Predict Regulatory Outcomes Data Driven Lobbying, etc. #Predict Rogue Behavior
  • 66. Not only law firms but also the large enterprise clients …
  • 67. 35! “From!se)lement!informa0on!and! contracts! to! sensi0ve! client! data! and! beyond,! Liberty! Mutual! creates! and! stores! ever:growing! volumes! of! unorganized! data! across! its! worldwide! offices! and! databases.”! “I've!seen!a!real!transforma0on!in! the! legal! department! just! having! t h a t! i n f o r m a 0 o n! v i s u a l l y! available."! “The' legal' department' is' now' w o r k i n g' p r e d i c 7 v e' a n d' prescrip7ve' analy7cs,"' i.e.' ways' to' analyze' data' that' enable' forecas7ng'for'legal'issues.”'
  • 68. 34!
  • 69. 36! “I"believe"strongly"that"data"analy2cs"is" a"new"fron2er"in"the"legal"space.”" Susie!Lees! General!Counsel!! Allstate!! “Leveraging" data," not" only" that" we" possess" but" that" our" law" firms" have" amassed"over"the"years,"offers"a"plethora" of" un<tapped" opportuni=es—not" simply" to" help" us" forecast" and" manage" legal" expenses," but" also" to" help" our" clients" make"more"informed"business"decisions.”"
  • 70. 33! “Now! we! have! program! managers,! data! analysts,! business! analysts,! data! scien9sts,! opera9ons! managers,!I!mean,!we!have!a!ton!of! stuff.! That's! the! key! for! me,! is! thinking! about! the! right! people! doing! the! right! tasks.! That's! the! people!part.!And!then!how!they!do! them,! is! the! process,! and! then,! automa9ng! parts,! is! kind! of! that! next,!final!step.!! " And$ all$ of$ that$ is$ underpinned$ by$ d a t a ." Y o u$ c a n ' t$ d o$ a n y$ improvements$ unless$ you$ have$ data.$ You$ can't$ automate$ unless$ you$have$good$data.”!
  • 71. In so much as prediction is the task in question … #LegalTech #FinTech #Fin(Legal)Tech
  • 72. “The real roll-up of all this isn’t robot lawyers, it’s financialization, with law becoming an applied branch of finance and insurance.” Daniel Martin Katz, professor, Illinois Tech’s Chicago Kent College of Law http://www.ozy.com/fast-forward/why-artificial-intelligence-might-replace-your-lawyer/75435
  • 76. The Three Forms of (Legal) Prediction
  • 78. In so much as prediction is the task in question … #MachineLearing is the method du jour
  • 79. It is not necessarily ML alone but rather some ensemble of experts, crowds + algorithms
  • 80. http://www.sciencemag.org/news/ 2017/05/artificial-intelligence-prevails- predicting-supreme-court-decisions Professor Katz noted that in the long term …“We believe the blend of experts, crowds, and algorithms is the secret sauce for the whole thing.” May 2nd 2017
  • 81. example from our own work
  • 82. predicting the decisions of the Supreme Court of the United States #SCOTUS
  • 84. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  • 86. Case Level Prediction Justice Level Prediction 67.4% experts 58% experts From the 68 Included Cases for the 2002-2003 Supreme Court Term
  • 88. they fit to the noise and not the signal
  • 90. if this were finance this would be trading worse than S&P500
  • 93. like many other forms human endeavor law is full of 
 noise predictors …
  • 94. we need to evaluate legal experts and somehow benchmark their expertise
  • 100. Crowds
  • 101. crowds
  • 106. not all members of crowd are made equal
  • 107. we maintain a ‘supercrowd’ which is the top n of predictors up to time t-1
  • 112. “Software developers were asked on two separate days to estimate the completion time for a given task, the hours they projected differed by 71%, on average. W h e n p a t h o l o g i s t s m a d e t wo assessments of the severity of biopsy results, the correlation between their ratings was only .61 (out of a perfect 1.0), indicating that they made inconsistent diagnoses quite frequently. Judgments made by different people are even more likely to diverge.”
  • 114. (most pundits did not identify as a serious candidate him until mid-January 2017) Neil Gorsuch was #1 o n o u r F a n t a s y Platform 12 Days after Donald Trump was elected President (i.e Nov 20)
  • 118. Columbia Law Review October, 2004 Theodore W. Ruger, Pauline T. Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
  • 119. Ruger, et al (2004) relied upon Brieman(1984) (as partially shown below)
  • 121. Leo Brieman moved away from CART in Brieman (2001)
  • 122. Breiman, L.(2001). Random forests. Machine learning, 45(1), 5-32. Published in Machine Learning (A Springer Science Journal)
  • 123. One well-known problem with standard classification trees is their tendency toward overfitting
  • 125. Random forest is an approach to aggregate weak learners into collective strong learners (using a combo of bagging and random substrates) (think of it as crowd sourcing of models)
  • 126. Our algorithm is a special version of random forest (time evolving) http://journals.plos.org/ plosone/article?id=10.1371/ journal.pone.0174698 available at RESEARCH ARTICLE A general approach for predicting the behavior of the Supreme Court of the United States Daniel Martin Katz1,2 *, Michael J. Bommarito II1,2 , Josh Blackman3 1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston, Houston, TX, United States of America * dkatz3@kentlaw.iit.edu Abstract Building on developments in machine learning and prior work in the science of judicial pre- diction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus- tice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications. Introduction As the leaves begin to fall each October, the first Monday marks the beginning of another term for the Supreme Court of the United States. Each term brings with it a series of challenging, important cases that cover legal questions as diverse as tax law, freedom of speech, patent law, administrative law, equal protection, and environmental law. In many instances, the Court’s decisions are meaningful not just for the litigants per se, but for society as a whole. Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal and political observers. Every year, newspapers, television and radio pundits, academic jour- nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular case. Will the Justices vote based on the political preferences of the President who appointed them or form a coalition along other dimensions? Will the Court counter expectations with an unexpected ruling? PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Katz DM, Bommarito MJ, II, Blackman J (2017) A general approach for predicting the behavior of the Supreme Court of the United States. PLoS ONE 12(4): e0174698. https://doi. org/10.1371/journal.pone.0174698 Editor: Luı´s A. Nunes Amaral, Northwestern University, UNITED STATES Received: January 17, 2017 Accepted: March 13, 2017 Published: April 12, 2017 Copyright: © 2017 Katz et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: Data and replication code are available on Github at the following URL: https://github.com/mjbommar/scotus-predict-v2/. Funding: The author(s) received no specific funding for this work. Competing interests: All Authors are Members of a LexPredict, LLC which provides consulting services to various legal industry stakeholders. We received no financial contributions from LexPredict or anyone else for this paper. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
  • 128. We call this a ‘general’ model of #SCOTUS Prediction available at https://arxiv.org/pdf/1612.03473
  • 129. Not just interested in accuracy over a short time window available at https://arxiv.org/pdf/1612.03473
  • 130. A locally tuned model will typically lead to overfitting as the dynamics shift available at https://arxiv.org/pdf/1612.03473
  • 131. We want a model that is robust to a large number of known dynamics … available at https://arxiv.org/pdf/1612.03473
  • 132. Version 2.02 January 16, 2017 243,882 28,009 Case Outcomes JusticeVotes Current Version of #PredictSCOTUS 1816-2015
  • 133. Version 2.02 January 16, 2017 Current Version of #PredictSCOTUS 1816-2015 case accuracy 70.2% 71.9% justice accuracy
  • 134. But are these results ‘good’ ?
  • 137. Our Model Against the Null Models Some commentators had suggested using a heuristic rule of
 ‘always guess reverse’ as a baseline (Null Model 1 ) the always guess Reverse model Turns out it is a lousy model prior to ~1950 Because the reversal rate is not stable over time
  • 138. Our Model Against the Null Models (Null Model 2 ) memory window = inf This is our model against Null Model 2 What about memory window that selects the most frequent historical outcome? (Green = our model out performs)
  • 139. Our Model Against the Null Models (Null Model 3 ) finite memory window = 10 We in-sample optimize using future information to select a null model that is among the best performing of all null models as it is using in-sample info this is a deeply unfair null
  • 140. Over past century, we outperform M=10 by nearly 5% and have significant temporal stability at both the justice, case, term level
  • 142. For most problems ... ensembles of these streams outperform any single stream
  • 143. the non-trivial question is how to optimally assemble such streams for particular problems
  • 147. Here is what we are working on right now …
  • 148. expert forecast crowd forecast learning problem is to discover how to blend streams of intelligence algorithm forecast ensemble method ensemble model
  • 149. expert forecast crowd forecast learning problem is to discover how to blend streams of intelligence algorithm forecast ensemble method ensemble model via back testing we can learn the weights to apply for particular problems
  • 150. By the way, you might ask why does one care about marginal improvements in prediction ? #Fin(Legal)Tech
  • 151. Given our ability to offer forecasts of judicial outcomes, we wondered if this information could inform an event driven trading strategy ?
  • 152. Revise + Resubmit @ http://arxiv.org/abs/1508.05751 available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
  • 155. The Three Forms of (Legal) Prediction The ‘Empirical Turn’ in Legal Scholarship The Diversity of Tasks that Lawyers Undertake Some Epistemological Issues / Questions The Other Type of Work That Lawyers Do Quantitative Legal Prediction
  • 160. Daniel Martin Katz @ computational computationallegalstudies.com lexpredict.com danielmartinkatz.com illinois tech - chicago kent college of law@ thelawlab.com