Retrieval-Augmented Generation for Case Law: A Practical Primer

Retrieval-augmented generation is the reason a legal research tool can cite. Understanding roughly how it works explains most of what such tools can and cannot do.

The basic mechanism

Rather than answering from what a model absorbed during training, the system first searches a corpus for relevant passages, then asks the model to answer using only those passages. The answer is constrained by what was retrieved. If retrieval fails, a well-built system says so; a badly-built one lets the model fill the gap from memory, which is where fabricated citations come from.

Why chunking decides what you can find

Documents are split into passages before indexing, and the split determines what is findable. Split a judgment into fixed 500-word blocks and a ratio that spans a paragraph boundary becomes unfindable, because no single block contains it. Legal texts need structure-aware splitting — by paragraph number in a judgment, by section in a statute, by recital and article in a regulation — with enough overlap that a proposition is never cut in half.

Hybrid search beats either method alone

Semantic search finds passages that mean the same thing in different words. Keyword search finds the exact citation, the party name and the section number. Legal research needs both: “the case about the tanker and the lost charter” is a semantic query; “section 33(2)(b)” is not. Systems that use only embeddings are frustrating for precisely the queries lawyers issue most.

Freshness and provenance

A retrieval corpus is only as current as its last update. Ask how often each source is refreshed, from where, and whether each document records its retrieval date and official URL. Without provenance metadata you cannot verify a citation without leaving the tool, which defeats the point.

What it still cannot do

Retrieval finds what is textually similar to your query. It does not reason about whether an authority remains good law, does not weigh competing lines of authority, and does not know that a first-instance decision from 2011 was quietly abandoned in practice. Negative-treatment signals help, but the judgement remains yours — and any tool that implies otherwise is selling something it does not have.