Benchmarking Legal AI: The Metrics That Matter Beyond Accuracy

Every legal AI vendor quotes an accuracy figure. Almost none of them define it, and the ones that do are usually measuring something that does not predict whether the tool will help you.

Precision and recall, separately

A single accuracy number hides the trade-off that matters. Precision tells you how many flagged findings were real; recall tells you how many real issues were flagged. For a first-pass review tool, recall is the safety property — a missed indemnity is worse than a false alarm — while precision determines whether reviewers keep using it. Ask for both, per clause type, on a document set that resembles yours.

Calibration

Of the findings a system reports at 90% confidence, roughly 90% should be correct. Systems that report uniformly high confidence are not calibrated, and uncalibrated confidence is worse than no confidence score at all: it teaches reviewers to trust the wrong things. Ask to see a reliability plot. Vendors who have one will show it happily.

Citation fidelity

For any system that produces citations, the question is not whether a citation is present but whether it supports the proposition. We measure three failure types: the cited passage does not exist, it exists but says something else, and it says something related but does not support the specific claim. The third is the most common and the hardest to spot in review.

Reviewer time saved, measured properly

The metric that decides whether a deployment survives contact with a practice group is how long the work takes with the tool versus without it, measured on the same documents by the same people. Run it as a controlled comparison over a fortnight. Most tools that fail this test fail because of interface friction rather than model quality — the findings were right, but checking them took longer than reading the clause.

Override rate and its clustering

Track how often reviewers reject a finding, and whether rejections cluster by clause type, document type or drafting style. A flat 8% override rate is healthy. An override rate of 40% concentrated in one clause type is a configuration problem you can fix this week.

A practical evaluation protocol

Take fifty documents you have already reviewed manually, with the findings recorded. Run them through the candidate system without telling it what you found. Compare. This takes about two days, costs nothing, and tells you more than any vendor benchmark — because it measures the system on your documents, against your standard of what counts as a finding.