When the right case is missing, will legal AI admit it?
On 1,670 altered US case-law questions, six generative models still chose a holding 38% to 86% of the time. A cheaper decision model was more cautious, but still guessed in 30%—useful as a check, not a substitute for lawyers.
The short version
- What happened
- The study used all 5,314 CaseHOLD test items and replaced the correct holding with a similar wrong one in 1,670, adding a none-of-these option.
- Who it affects
- Lawyers and court staff could use a calibrated second model to prioritise citation checks, but responsibility for verifying the authority and proposition remains with the professional submitting the work.
- What remains unproven
- The preprint studies a constructed multiple-choice benchmark using US opinions, not live legal research or filed documents.
Keep in mind
The preprint studies a constructed multiple-choice benchmark using US opinions, not live legal research or filed documents.
Closed-model training data are unknown; anonymisation tests reduce but do not eliminate contamination concerns.
The relative cost of a wrong citation versus an unnecessary review was illustrated rather than empirically estimated.
The paper reports no separate funding statement and includes proprietary model interfaces whose behaviour may change.
Source analysis · 1 linked source. Primary analysed source date: 21 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The benchmark made absence visible
The researcher started with CaseHOLD, a benchmark built from US court opinions in which a model selects the holding attached to a cited case. All 5,314 test items were used. In a random 1,670, the true holding was replaced with a similarly worded holding from another case and a sixth answer—none of these—became correct. Four probability-returning “decision models” and six generative models then saw the same passages and options.
When the authority was missing, ordinary generative models still selected a holding in 38% to 86% of items. The most cautious decision model guessed in 30%, but abstained unnecessarily on 26.2% of items where the correct holding remained. Another decision model reached 88.1% on standard CaseHOLD, essentially matching the best generative model's 88.2%, at 1/55 of the reported inference cost.[1]
A gate can reduce wrong citations, not remove legal responsibility
The study paired models in three ways. The most practical accepted a generative answer only when a decision model agreed; disagreements went to review. This roughly halved wrong citations for several pairings, but it also increased the number of questions a person had to inspect. Thresholds were tuned on 30% of items and reported on the remaining 3,720, with 2,000 bootstrap resamples and exact paired tests.
The absence was constructed, not a live legal-research failure. CaseHOLD opinions predate 2021, model training data are undisclosed and contamination cannot be ruled out. Real authority may be absent because it was overruled, belongs to another jurisdiction or never existed. The evidence supports an abstention test and a review gate—not autonomous legal research. Prospective trials on current dockets and independent replication would change the assessment.[1]
The distinction that matters
How the legal check works
The study separates choosing a plausible holding from knowing when none is supported.
Remove
Replace the true holding in 1,670 items
Creates a controlled missing-authority condition
Compare
Test four decision models and six generative models
Measures wrong picks and false abstentions
Gate
Send model disagreements to review
Trades fewer wrong citations for more human checks