Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Martin Katz

Quantitative Legal Prediction
Professor Daniel Martin Katz
Illinois Tech - Chicago Kent College of Law
-Or- How I Learned to Stop Worrying and Start Preparing
for the Future of the Legal Services Industry)
@computationaldanielmartinkatz.com computationallegalstudies.com

My is Lab Focused Upon
Support the R&D for the
Legal Services Industry

There are many
Research & Development
Questions in the Legal
Services Industry

My Research Group is
Actively Engaged in
Research that is Relevant
to Developing
Future
Legal/Govt
Information
Products

Here Are a Few
Things from Our Lab
3D HD Visualization of Supreme
Court Citation Network
Campaign Contributions and
Legislative Ecosystems
Six Degrees
of
Marbury
v.
Madison
Electronic
World
Treaty
Index
The United States Code

Here Are a Few
Things from Our Lab
American
Federal
Judiciary
American
Law Professoriate
Building New Algorithms
Large Scale
Judicial Studies

Legal Language Explorer
Indexing 450,000+ Cases

ComputationalLegalStudies.com
BLOG

before providing some
concrete examples -
some broad thoughts ...

three faces of innovation in legal

(1) lawyers for innovators / entrepreneurs

(1) lawyers for innovators / entrepreneurs
what most lawyers and law schools think of as “Law+Entrepreneurship"

(2) lawyers as innovators - substance

poison pill - “the most important innovation in corporate law since
Samuel Calvin Tate Dodd invented the trust for John
D. Rockefeller and Standard Oil in 1879”

emerging areas - 3D Printing, Driverless Cars, Augmented Reality,
Data Breach, Big Data+Privacy, etc.
Drones, Internet of Things, CyberSecurity,

(3) lawyers as innovators - business/process

innovation directed toward transforming the practice of law
(3) lawyers as innovators - business/process

there are different ways
that organizations are
innovating on the third face

{Law
Substantive
Legal
Expertise
Analytics
Platform
AI
Computing
Process Mapping
User Experience
Design Thinking
Business Models
Regulation
Marketing
+ Tech + Design
TM
+ Delivery}

some traditional law firms
have been very aggressive

but most of the innovation
is Lex.Startup

Lex.Startup
is beginning to take hold

15 425+
2009 2014
Law or Legal Related Companies
as highlighted by Josh Kubicki @ ReInventLaw London 2013
Lex.Startup

So what are these
folks doing?

R + D Function in the
Legal Industry

We Could Imagine a World
Where Law Firms Did the
R+D for the Industry

But That Has
(Mostly) Proven Illusive

Lex.Startup
is undertaking that function

Here are the specific
approaches that are
being undertaken

Some organizations are
doing more than one

labor
arbitrage
process/
tech arbitrage

labor
arbitrage
process/
tech arbitrage
regulatory
arbitrage

labor
arbitrage
process/
tech arbitrage
regulatory
arbitrage
design as the
ultimate bespoke

labor
arbitrage
process/
tech arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
predictive
analytics

could do an
individual talk on
any of these topics...

Daniel Martin Katz
Michigan State University - College of Law
-Or- How I Learned to Stop Worrying and Start Preparing
for the Future of the Legal Services Industry)

Today I Would Like to
Sketch (In Part)
Where I Believe the
Legal Industry
is Heading

Before Talking
About the
Law Business

This is the Era of “Big Data”
Decreasing Data Storage Costs
Increasing Computing Power
Fundamentally Altering the Scope of Scientific Inquiry
and Technical Possibility

Highlighting the Data Deluge
2008 2009 2010

What is Driving
the Big Data
Revolution?

Erik Brynjolfsson is the Schussel Family
Professor at the MIT Sloan School of
Management , Director of the MIT
Center for Digital Business, Chair of
the MIT Sloan Management Review ,
and the Editor of the Information
Systems Network
Andrew McAfee, a principal research
scientist at MIT’s Center for Digital
Business, studies the ways that
information technology (IT) affects
business.

. .. ....
..
......
....
............ 128
256 512 1024 2048 4096 327688192 16384
65536 131k 262k 524k 1M 2M 4M
................................................................
8M
................................
16M 33M 67M 134M 268M 536M 1B 2B
4B 8B 17B 34B 68B 137B 274B 549B
1T 2T 4T 8T 17T 35T 70T 140T
281T 562T 1Q 2Q 4Q 9Q 18Q 36Q
72Q 144Q 288Q 576Q 1QT 2QT 4QT 9QT

Data is only half the story...

Computation
and
Artificial Intelligence

The Artificial Intelligence
Revolution is On

The Artificial Intelligence
Revolution is On ....
But it is not what we thought

‘Soft’ Artificial Intelligence

“Practically every financial transaction, from
someone buying a cup of coffee to someone
trading a trillion dollars of credit default
derivatives, is done in software ....
Health care and education, in my view, are
next up for fundamental software-based
transformation. My venture capital firm is
backing aggressive start-ups in both of these
gigantic and critical industries.
We believe both of these industries, which
historically have been highly resistant to
entrepreneurial change, are primed for tipping
by great new software-centric entrepreneurs ...
Companies in every industry need to assume
that a software revolution is coming.”

The First Response
I Typically Encounter

You Cannot Replace
The Things I Do ...

Where Qualitative
Human Reasoning

http://www.cbsnews.com/video/watch/?id=6945451n

Goal:
Build a Driverless Car that
Could Travel 150 miles

Winning Vehicle
Traveled
only
Eight Miles

Remember Industry
It is Not
A Binary Proposition

Computers CANNOT
Do Everything

Displacing 20%-30%
of the Work Load is
Damn Pretty
Significant

100
Lawyers
70
Lawyers

10
Law
+
Tech

5

Tech
+
Law

70
Lawyers
in
‘Safe’
Employment

30
Lawyers
in
Employment

Susceptible
to
Automation
85
Lawyers/Legal
Service
Jobs
30%
Reduction
in
Traditional
Law
Jobs

15%
Reduction
in
Law
Related
Employment

Arbitrage
Opportunities
For
Helping
Move
Across
the
Spectrum

30%
Reduction
in
Traditional
Law
Jobs

15%
Reduction
in
Law
Related
Employment

Arbitrage
Opportunities
For
Helping
Move
Across
the
Spectrum
100
Lawyers
70
Lawyers

10
Law
+
Tech

5

Tech
+
Law

70
Lawyers
in
‘Safe’
Employment

30
Lawyers
in
Employment

Susceptible
to
Automation
85
Lawyers/Legal
Service
Jobs

There is Potential For
Growing Other Parts
of the Market ...

Technology Aided
Access to Justice

Some Applicable Terms
That Will Drive The
Future of the Industry

And Will Be Part of
What it Means to
“Think Like a Lawyer”

Natural Language Processing
Clustering
Knowledge Representation
Dimension Reduction
Feature Selection
Feature Extraction
Classification

http://www.drewconway.com/zia/?p=2378

So What is the
Next Big Thing... ?

2011
The Age of

2012
The Age of

2013
The Age of

Three Key Ideas
About
Prediction

(1) Inverse Problem
(2) System Dynamics
(3) How Machines Learn

Hypothesis Testing
is the Core of
Mainstream Science

This is the Age of Aspirational Spelling
(Spelling is 1.0 Thinking)

(a) Induce a Plausible Model
from Existing Data

Either:
Out of Sample
Forward Prediction
Or Both

Imagine Two Different
Complex Systems

vs.
Easy/ Predictable Difficult / Chaotic
TIDES ALMANAC

Formal Treatment of the
question of prediction in
alternative Domains

(3)
Machine Learning is the
Heart of Predictive Analytics

Cause
and
Effect
Quantitative
Legal
Prediction
vs.

Quantitative Methods for Lawyers

Legal Analytics
Professor Michael J Bommarito II
http://www.legalanalyticscourse.com/

Supervised
Statistical models
Bayesian, e.g., Naïve Bayes Classification
Frequentist, e.g., Ordinary Least Squares
Neural Networks (NN)
Support Vector Machines (SVM)
Random Forests (RF)
Genetic Algorithms (GA)
Semi/Unsupervised
Neural Networks (NN)
Clustering
K-means
Hierarchical
Radial Basis (RBF)
Graph
Some Machine Learning Methods

http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html

classification
clustering
regression
dimension reduction
the family of machine learning methods

Quick Example of
Some of the Methods

Adapted from Slides By
Victor Lavrenko and Nigel Goddard
@ University of Edinburgh
Take A LookThese 12

72
Female
Human
3
Female
Horse
36
Male
Human
21
Male
Human
67
Male
Human
29
Female
Human
54
Male
Human
44
Male
Human
50
Male
Human
42
Female
Human
6
Male
Dog
7
Female
Human

Classiﬁcation
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?

Classiﬁcation
decision
boundary
female
male
f( )
Gender?
Regression
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4

Classiﬁcation
decision
boundary
female
male
f( )
Gender?
f( )
Loan
Application?
Yes
Multi Class Classiﬁcation
No
Maybe
Yes
Perhaps
No
Multiclass =
Boundary
Hyperplane
Regression
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4

Classiﬁcation
decision
boundary
female
male
f( )
Gender?
f( )
Loan
Application?
Yes
Multi Class Classiﬁcation
No
Maybe
Yes
Perhaps
No
Multiclass =
Boundary
Hyperplane
Regression
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4
Clustering
(Unsupervised
Learning)
Clusterf( )
Group?

Human Prediction is
One Hallmark of the Legal
Services Industry

The Race for the
Future of this Industry

To Use Applied Machine
Learning to Mimic the Behavior
“Expert Reasoners”

but do so
at highly aggregated scale

Humans are
Amazing
Pattern Detectors

But Aggregation is
the Question ...

How Does a Human
Reasoner Evaluate
10,000
100,000
1,000,000
Data
Points?

Truth is They Don’t
#Heuristics

And It Will Continue to Move
Up The Value Chain ...

Some Pieces of the
Previous
Entry Level
BigLaw Jobs

Have Been Displaced
In Part By

Just Remember there
was a Day When
E-Discovery was
not mainstream

And Now It Dominates
Our Industry

And Now It
Dominates The
Industry

But Another Form of
Predictive Technology...

Predictive Coding
Will Move Across the
Machine Learning
Spectrum
Supervised UnsupervisedSemi-
Supervised
Present Future

Supervised Unsupervised
Predictive
Coding
(Classification)
The Future
Machine
Learning
Methods
2 x 2
Informed
Naive
Basic
Clustering
Algorithm
© daniel martin katz michael j bommarito

Predictive Coding
as an example of applied
machine learning ...

predictive coding =
~ binary classification

LearningTask = Determine Whether a Given
Document is Relevant?
Relevant
Not Relevant
f( )
relevance?
Binary Classiﬁcation (Supervised Learning)
and/or
010
101
001

take the sample set as
a training set and
use human experts

the use of the human
experts is called
“supervised learning”

in the simple binary case,
ask humans to assign
objects to two piles

Apply Human Coders

yellow = relevant
white = non-relevant
and return this

Non RelevantRelevant

Key Insight ...

What Allows A
Human To Separate
These Two Classes of
Documents?

that precise human
process is what
“predictive coding”
is trying to mimic

most vendors are selling a
largely undifferentiated product

Humans are selecting
upon some “features”
of the documents

to place those
documents in their
respective bins 
(i.e. relevant, non-relevant)

features =?
text,
author,
date,
other metadata

machine learning task is
trying to recover (learn)
what separates the
relevant from the
non-relevant documents

once we learn the
rule / boundary
we can apply it to separate
the remain documents into
the two classes

we want to take what we learn here

we want to take what we learn here
and apply it here

Legal Procurement
&
Legal Supply Chain Mgmt.

Legal Supply Chain Mgmt.
(High End of Market)
Data and Logistics =
General Counsels as
Maestros managing the
global legal supply chain

General Counsels as Legal
Procurement Specialists
TyMetrix -
Using $50 billion+ in Legal
Spend Data to Help GC’s
Look for Arbitrage
Opportunities, Value
Propositions in Hiring Law
Firms
Legal Procurement

Driving Down your
Legal Bills
Yeah there is an
App for That
City
Firm Size
Partner
Experience
Calculate
Legal Procurement
http://tymetrix.com/mobile_apps/

Model Leverages
Classification Tree
(Tool from
Machine Learning)

Here is a Technical Paper
that I Am Currently Finishing

Predicting the Behavior of the
United States Supreme Court:
A General Approach
By Daniel Martin Katz
Michael J. Bommarito II
Josh Blackman

From
Classification Trees
to
Random Forest

Random forest is an approach to
aggregate weak learners into
collective strong learners
(think of it as “the wisdom of the statistical crowds”)

Affirm or Reverse
Lower Court
Decision ?
Left Hand Side (Y)

Right
Hand
Side
( X’s)
court_direction_mean
court_direction_std
justice_direction_mean
justice_direction_std
justice_court_difference_z
lcDisposition_Direction
lcDispositionDirection_difference_abs
lcDispositionDirection_difference
lcDispositionDirection_difference_z
Case
Information
issue
issueArea
lawType
certReason
respondent
respondent_dk
petitioner
petitioner_dk
caseOrigin
caseSource
monthArgument
timesince_arg
Court / Justice
Information
party_president
segal_cover
year_of_birth
naturalCourt
Historical Justice
/ Court
Information

But This is Merely the
Tip of the Iceberg

Predicting
Outcomes
in Disputes

Bargaining in the
Shadow of the Law

What is the Client’s
First Question?

How is that
assessment generated?

How Does the Human
Reasoner Arrive at
Their Conclusion?

Pattern Detection
High Dimensional
Similarity Matching
Analogical Reasoning
etc.

Mental Models
{ vs or + }
Aggregated Data

Humans
+
Machines
>
Humans
or
Machines

Standard Client Memo
+
Statistical Portrait of
Lots of ‘similar’ cases

IP LITIGATION
/
M & A
Valuation

https://lexmachina.com/
June 2012

Predictive Model of
Securities Fraud Class
Action Lawsuits

Predicts both the likelihood of
settlement and the expected
settlement amount

Uses only variables
that are known at the
day of filing

There Are Many Other
Additional Examples ...

“The software
identiﬁes standard
and terms in
contracts, and its
benchmarking
tools show
lawyers how their
current document
compares to the
standard.”

The system comes pre-trained  
for provisions including:
Title, Parties, Date, Term, Change of
Control, Assignment, Indemnity,
Confidentiality, Governing Law,
License Grant, Bankruptcy, Notice,
Amendment, Non-Solicit, and more.

Based on testing, we know our system finds
90% or more of the instances of nearly
every substantive provision it covers.
This 90% number is our system’s recall;
its precision differs by provision by
provision but is consistently very
manageable.

We are able to build custom provisions on
request. Thanks to our highly customized
training algorithms, this process is easy and
relatively automated. We are also engaged
in adding more provisions.

The VC Community
Is Turning to Legal

R e p o r t e d s a l e
price between $35
million and $40
million.
Final Number was
likely between
$80 - $100 million
A n u m b e r o f
venture capitalists
have invested in
t h e c o m p a n y ,
including Silicon
Valley’s Sequoia
C a p i t a l w h i c h
invested $7 million
in 2007 ....

And There is
Lots More in this Space ...

Okay There are Serious
Technical Questions
In Here ...

But this is Also
Very Practical

Lawyers are in the
Prediction Business
(in Part)

Technology Has Already
Disrupted Law ...

There Is Going to Be
Lots More

Welcome To
Law’s
Information
Revolution

And Yes There Is Going
to Be Math (& Computing)
on the Exam

Daniel Martin Katz
Illinois Institute of Technology
Associate Professor of Law
@ computational
computationallegalstudies.com
danielmartinkatz.com

Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Martin Katz

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

En vedette

En vedette (17)

Similaire à Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Martin Katz

Similaire à Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Martin Katz (20)

Plus de Daniel Katz

Plus de Daniel Katz (10)

Dernier

Dernier (20)

Quantitative Legal Prediction - Presentation @ Santa Clara Law - By Daniel Martin Katz