38 min

Legal Analytics

Quantitative methods applied to legal materials. The three things people mean by prediction and why only one is forecasting, the leakage result that reframed the field, precedent as a citation network, and the selection effect that makes litigated cases the worst possible sample.

Listen

0. What it is, and three things it is not

Legal analytics is the quantitative study of legal materials and legal institutions — judgments, dockets, citations, filings, outcomes — treated as data. It is the closest named field to what ATLANTIS's computational strand would be doing, and it has a twenty-year methodological literature that has already found most of the traps. Knowing that literature is cheap credibility, because it means you arrive with the field's known failure modes rather than rediscovering them.

Not this	Why the distinction matters
Not legal tech	Contract-review products and e-discovery platforms are tools sold to firms. Legal analytics is a research method. Overlapping techniques, different purpose, and conflating them makes you sound like a vendor
Not AI judging	Nobody serious proposes replacing judges, and the press coverage of this field has been consistently worse than the field. If you bring this up, bring it up to dismiss it
Not natural-language processing with a legal dataset	The method is NLP, graph analysis and statistics. What makes it legal analytics is taking the legal process seriously as a data-generating process — which is the whole content of sections 2 and 4

1. The four data objects

Legal materials are not one kind of data, and each kind supports different questions. Being able to say which object you would need for a given question is a strong, concrete answer.

What a single case actually offers as data

  ONE CASE
     |
     +-- 1. THE TEXT  unstructured prose
     |      facts, reasoning, holding, dissent
     |      --> classification, topic modelling,
     |          text reuse, doctrinal change
     |
     +-- 2. THE DOCKET  the procedural record
     |      filing dates, parties, counsel, motions,
     |      which judge, how long it took
     |      --> duration models, procedural
     |          behaviour, forum effects
     |      --> AND it is the only object available
     |          BEFORE the judgment exists
     |
     +-- 3. THE CITATIONS  what it cites, what cites it
     |      --> network analysis, precedent
     |          authority, doctrinal lineage
     |
     '-- 4. THE OUTCOME  who won, what remedy, what fine
            --> the label for any supervised model
            --> and the hardest one to define well

The split that matters most is between the docket and the text. The docket exists before the decision; the text is written afterwards by the institution that decided. That single fact is what section 3 is about, and it is the most consequential methodological point in the field.

2. The three things people call prediction — and only one is forecasting

This is the spine of the chapter. Medvedeva, Wieling and Vols argued, in a paper on rethinking automatic prediction of court decisions, that the field had been lumping together three genuinely different tasks under one word — and that most published *prediction* results are not prediction in the sense a reader assumes. The distinction is simple, it is devastating when applied, and it is the single most useful thing in this chapter.

Task	What information the model gets	What it actually demonstrates
Outcome identification	The published judgment, including the court's own account of the facts	That the outcome is recoverable from a document written after, and by, the deciding body. Useful for automatic annotation of large archives. It is not prediction of anything
Outcome-based categorisation	Some part of the judgment, with the explicit result removed	That textual features correlate with outcome. Interesting as a study of judicial reasoning and framing. Still not forecasting, because the document postdates the decision
Outcome forecasting	Only information that existed before the decision — the application, the docket, the parties, the date	Genuine prediction. This is the only one that would be usable prospectively, and it is much harder and much less accurate

Why this is worth saying in the interview

Because the same confusion is about to happen in competition law, and nobody has named it there yet. A screen trained on the Commission's decided cases, using features drawn from the decisions, is doing outcome identification — it learns what a decided infringement looks like once the Commission has written it up. A tool that tells an agency where to look next has to do forecasting, from pre-investigation data only. These are different products with different accuracy ceilings, and an agency procuring the first while believing it bought the second is a real and specific governance failure. *I think the most useful thing I could do early is insist on the distinction between identifying an outcome in a decided case and forecasting one in an undecided case, because the published accuracy figures in adjacent fields almost all come from the first and get read as the second.* That is a research contribution stated as a sentence.

3. The ECtHR story, worked through — the field's cautionary tale

This sequence is the best-documented example of the whole problem and it is worth knowing in order, because it runs from a headline result to a careful rebuttal and ends somewhere more honest.

Step	The finding
Aletras, Tsarapatsanis, Preoţiuc-Pietro and Lampos (2016), PeerJ Computer Science	Textual features from roughly 600 European Court of Human Rights cases predicted whether the Court found a violation, at about 79 percent average accuracy. The paper reports that the formal facts section was the most predictive part, and reads that as consistent with legal realism — that judicial decisions are substantially shaped by the facts
The press	Reported as an AI predicting human-rights trials. The paper's own authors pushed back on the AI-judge framing in interviews at the time
A reviewer on the published peer-review record	Noted that the acknowledged limitations leave the paper reliant on crude proxies, and that its arguments would struggle in a legal setting. The caveat was visible from the start
Medvedeva, Vols and Wieling (2020), Artificial Intelligence and Law	Reported about 75 percent average accuracy across nine Convention articles — and then did the test that mattered: training on past cases and predicting future ones dropped performance to roughly 58 to 68 percent. They also found they could classify outcomes at relatively high accuracy using only the surnames of the judges hearing the case
Medvedeva and colleagues (2021)	Moved to forecasting for pending applications, using only information available before the judgment — the methodologically correct framing, and much harder

Why the 79 percent is not forecasting

  TIME ------------------------------------------>

  application     deliberation     JUDGMENT PUBLISHED
  filed           and decision     |
    |                   |          |
    |                   |          +-- "THE FACTS"  <-- written
    |                   |          |   section          HERE
    |                   |          |
    |                   |          +-- reasoning
    |                   |          +-- the holding
    |                   |
    |                   '-- the outcome is
    |                       determined here
    |
    '-- this is the ONLY point at which a
        genuine forecast could be made

  The 2016 study drew its features from the facts
  section -- which sits to the RIGHT of the decision
  on this timeline. It is written by the same body
  that decided, after deciding, and its emphasis
  and selection already reflect the conclusion.

  So the model is not predicting the outcome.
  It is reading the court's own framing of a case
  it has already resolved.

  And the judges'-surnames result is the proof:
  if WHO HEARD IT predicts the result well, the
  signal is not purely in the law.

This is information leakage, and it is the most common fatal flaw in applied machine learning generally. The fix is a strict temporal split and a feature set restricted to pre-decision information, which is exactly what the later work did — and performance fell by ten to twenty points, which is the honest number.

The transfer, and it is direct

A Commission decision has exactly the same structure. It contains a statement of facts, written by the Commission, after the Commission decided, selected and emphasised to support the conclusion it reached. **So a model trained on decision text to *predict infringement* is reading the Commission's framing, not forecasting conduct. And the surnames result has an uncomfortable analogue: if the identity of the case team or the decision period predicts the outcome, you have measured institutional variation rather than law.** *That is not a reason not to build these systems — it is the reason to build them with a temporal split and a pre-decision feature set, and to report both numbers.* Both numbers. The identification figure and the forecasting figure, side by side, because the gap between them is itself the finding.

4. Precedent as a network

Citation analysis is the most mature and least controversial part of legal analytics, because it does not predict anything — it measures a structure that genuinely exists. A judgment citing another judgment is a directed edge, and a body of case law is therefore a graph.

A(c) = \frac{1-d}{N} + d \sum_{c' \to c} \frac{A(c')}{L(c')}

The authority of a case equals a small baseline, plus a share of the authority of every case that cites it, divided by how many cases each of those cites in turn.This is the PageRank idea applied to precedent, and the recursion is the point: being cited by an authoritative case counts for more than being cited by an obscure one. Dividing by the number of outgoing citations stops a case that cites two hundred others from passing on full weight to each. Plain-language version: a precedent is authoritative if important cases rely on it, and importance is defined the same way all the way down.

Measure	Legal reading	Honest caveat
In-degree — how many cases cite it	Crude popularity	Scales with age. A 1950 case has had seventy years to accumulate citations
Recursive authority (above)	Cited by cases that matter	Better, and still age-dependent
Betweenness	A case that bridges two doctrinal areas	Often the genuinely interesting one — bridges are where doctrine transfers between fields
Community detection	Clusters of mutually citing cases = doctrinal areas, discovered rather than assumed	The appealing use: watching a doctrinal area split or merge over time
Citation decay	A case stops being cited	Could be overruled, superseded, or simply the question stopped arising. The graph cannot tell you which

The three caveats, and the third is the deep one

One — a citation is not an endorsement. Courts cite cases to distinguish them, to narrow them, and to overrule them. **An unsigned citation graph treats *we decline to follow X* identically to *we follow X*. Serious work signs the edges, which requires reading or a classifier, and the classifier needs validating. Two — age bias, as above: any accumulation measure rewards being old, so compare within cohorts or normalise by time at risk. Three — preferential attachment. If citation is substantially rich-get-richer, then centrality partly measures accumulated attention rather than legal importance — a case is cited because it is cited. That means centrality is a measure of *salience in the citation practice*, not of *legal correctness or quality*, and the two can diverge. Say that distinction and you have said the thing most citation-analysis papers skate over.**

5. The selection effect — and why litigated cases are the worst available sample

This is the catchment problem, and it has a classic formal statement. Priest and Klein, in *The Selection of Disputes for Litigation* (1984), argued that the disputes which reach adjudication are neither a random nor a representative sample of the disputes that exist — because parties settle the cases whose outcomes they can both predict.

(P_{p} - P_{d})\, J \;>\; C_{p} + C_{d} \qquad\Longleftrightarrow\qquad P_{p} - P_{d} \;>\; \frac{C_{p} + C_{d}}{J}

A case goes to trial rather than settling when the gap between the plaintiff's and the defendant's estimates of the plaintiff's chance of winning, multiplied by the amount at stake, exceeds the combined cost of litigating.**Read the right-hand form: trial requires the two sides to *disagree* by more than the cost ratio. If both sides agree the plaintiff will probably win, there is a settlement both prefer. So litigation selects for cases where the parties disagree — which means cases that are genuinely close. Priest and Klein's famous corollary is a tendency toward 50 percent plaintiff victories among litigated cases, with the party having more at stake winning more often. Note the status of that result: later formal work (Lee and Klerman, 2016) shows the exact 50 percent figure is sensitive to the bargaining and distributional assumptions. So treat it as a strong tendency with known conditions, not a law — and saying that is the correct level of confidence.**

The funnel — and your dataset is the last layer

  ALL DISPUTES that exist in the world
  ####################################################
        |  most are never articulated as grievances
        v
  GRIEVANCES RAISED
  ##########################
        |  most are abandoned or informally resolved
        v
  CLAIMS FILED
  ##############
        |  the large majority SETTLE
        |  (estimates for civil litigation
        |   commonly put this around 95%)
        v
  LITIGATED TO JUDGMENT
  ###
        |  only some are appealed
        v
  APPEALED
  ##
        |  only some are reported / published
        v
  IN YOUR DATASET
  #

   Each layer filters NON-RANDOMLY, and the
   filter at the settlement layer selects
   specifically for CLOSE cases.

   So: a finding about published appellate
   judgments is a finding about the least
   representative slice of the funnel.

Every layer of this funnel is a selection step, and the settlement layer selects on predictability itself. This is why a plaintiff win rate computed from litigated cases cannot be read as the strength of plaintiffs' claims in general, and it is the single most important thing to understand before building any dataset of decided cases.

The competition-law version, and it is your own point

The enforcement funnel has exactly this shape. All anticompetitive conduct, then conduct someone notices, then complaints or leniency applications, then opened investigations, then decisions, then published decisions. And the filter at the detection layer selects on concealment competence — the cartels in your dataset are the ones that were bad at hiding. This is the same structure as the dark figure in your crime paper, and you should connect them explicitly: *the published-decisions corpus is to collusion what recorded offences are to crime. In both cases the selection mechanism is the thing I would most want to measure, and in both cases it is the thing that determined what I could see.* That sentence joins Priest and Klein, your own paper, and the project's accuracy strand in one move.

6. Judicial behaviour analytics — where the law has already drawn a line

Analysing how individual judges decide is the most legally fraught application in the field, and it is the one place where a jurisdiction has banned the research outright.

France, and this is the fact to have exactly right

Article 33 of French Law number 2019-222 of 23 March 2019 prohibits the reuse of judges' and registry members' identity data where the purpose or effect is to evaluate, analyse, compare or predict their professional practices. Breach carries penalties up to five years' imprisonment. It was widely reported as the first provision of its kind. Why it exists: the concern is forum-shopping and pressure on individual judges — if litigants can rank judges by outcome, they will steer cases and judicial independence erodes. And note the live problem on your own site: LawReformer's *Legal Reality Engine* includes analysis of judge behaviour. That design would be unlawful in France. You already have this recorded in the interview brief; the reason it belongs here too is that volunteering it is the strongest possible demonstration of the competency being hired for — you built something whose legality varies by jurisdiction and you found that out yourself.

And the methodological point underneath the legal one

The judges'-surnames result from section 3 is the uncomfortable empirical core here. If who hears a case predicts its outcome, then either judges differ systematically, or panel assignment correlates with case type, or both. All three are worth knowing and all three are explosive to publish. The defensible research posture, and it is a real position you can state: *study variation at the level of the institution rather than the individual. Whether outcomes vary across chambers, periods or procedures is a question about institutional design and is publishable. Whether Judge X is more claimant-friendly than Judge Y is a question about a person, and the French prohibition is a considered judgment that the second is not worth the cost even though the data supports it.* That is a nuanced answer on an ethics question, which is exactly what a panel with a privacy scholar on it wants to hear.

7. Legal analytics in competition law specifically

This is the part that connects the chapter to the actual job, and the chair of your panel is one of the people building it.

What exists	What it is
Stanford Computational Antitrust	Founded by Thibault Schrepel. The project that named the field, and it runs an annual cross-agency report — the fourth, with Teodora Groza, drew on 25 agencies
A knowledge graph of European Commission decisions, 1977 to 2025	His own project, and it is legal analytics in the precise sense of this chapter — the decisional record as a structured, queryable graph rather than a pile of PDFs
*The digital brain* proposal, April 2026**	Offered to agencies: consistency-check a draft decision against the full decisional record. This is the single most interesting artefact to have an opinion about, because it is citation-and-consistency analysis applied to an agency's own output rather than prediction of a third party
His Kluwer post, July 2026	Titled *Competition Agencies Compute. The Law Still Assumes They Read.* States that all 27 EU national competition authorities and DG Competition rely on computational tools. It is a title, so you can attribute it precisely

The thing to have a view on, and here is a defensible one

**The *digital brain* consistency-check idea is methodologically much safer than outcome prediction, and you should be able to say why.** *It is not forecasting anything — it is retrieval and comparison against a corpus the agency itself authored. So there is no leakage problem, no selection problem on the dependent variable, and no question of predicting a third party's conduct. The failure mode is different and more tractable: it is a retrieval-quality problem. If the index misses a relevant prior decision, the tool silently reports consistency that is not there — and a false reassurance is worse than no tool, because it will be relied on.* So the evaluation it needs is recall-oriented rather than accuracy-oriented, and you would want to know, for a set of decisions where the relevant precedents are known, what fraction the system actually surfaces. *That is a concrete methodological contribution I could start on in the first year.* This is probably the best single answer you have for the chair.

Four questions legal analytics could answer for an agency

Framed as research, not as products. One — consistency: does the agency treat like cases alike over time, measurable as variation in outcome conditional on coded case features? Two — doctrinal drift: which concepts have changed meaning in the decisional record, measurable through text reuse and shifting citation neighbourhoods? Three — duration: what predicts how long an investigation takes, using docket features only, which is the one genuine forecasting task in the list and the one least likely to raise objections? Four — the effect of the instruments: did the DMA change conduct, which is the discontinuity design from the data-science chapter rather than a text problem at all. **Notice that none of these is *predict who is guilty* — and that is deliberate. Each measures the institution rather than the firm**, which keeps you inside what the method can carry and outside what a court would have to decide.

8. The probes, and what to say

Panellist	Likely probe	Your answer
Thibault Schrepel	What would you do with the decisions corpus?	Section 7's consistency-check answer: retrieval recall, not prediction accuracy. Plus section 2's identification-versus-forecasting distinction, which is the framing contribution
Catalina Goanta	How would you validate a coding scheme over legal text?	The NLP chapter's validation plan: a defined codebook, two coders, an agreement statistic fixed in advance, and a reported threshold. *Not I would fine-tune a model***
Tijmen Wisman	What about the people being analysed?	Section 6. The French prohibition, the institution-versus-individual line, and the fact that your own site sits on the wrong side of it — volunteered
Georgiana Mirza	Doctrinal change and regulatory design	Section 4's community detection and citation decay as a way of watching a doctrinal area split — and the caveat that decay cannot distinguish overruled from irrelevant

9. If you remember seven things

Three tasks, one word. Outcome *identification* reads a decided judgment; outcome-based *categorisation* removes the explicit result; only outcome *forecasting* uses pre-decision information. Most headline accuracy figures are the first.
Aletras and colleagues, 2016, PeerJ Computer Science: about 79 percent on roughly 600 ECtHR cases, with the facts section most predictive — but the facts section is written after and by the deciding court.
Medvedeva, Vols and Wieling, 2020: about 75 percent, falling to roughly 58 to 68 percent when forecasting future cases from past ones — and high accuracy from judges' surnames alone. That pair of numbers is the most useful thing you can quote in this area.
**Fowler and Jeon, 2008, *Social Networks*: 30,288 US Supreme Court majority opinions, 1754 to 2002, with centrality used to quantify precedent authority. Caveat: citation is partly rich-get-richer, so centrality measures salience rather than quality. And a citation can be a rejection**.
Priest and Klein, 1984: litigated cases are neither random nor representative, because parties settle what they can both predict — so litigation selects for close cases. The tendency toward 50 percent is real but assumption-sensitive.
The enforcement funnel has the same shape as the dark figure in your own paper. Detected cartels are selected on concealment competence. Say this — it is the bridge between your work and theirs.
France, Article 33 of Law 2019-222 of 23 March 2019: analysing or predicting individual judges' practices is prohibited, with penalties up to five years. Study the institution, not the individual — and your own LawReformer judge-behaviour feature is on the wrong side of that line.