A legal answer can contain a real citation and still be wrong. The source may not support the proposition, may be outdated, may come from the wrong jurisdiction, or may be inaccessible to the user. Retrieval-augmented generation reduces some failures but creates a new duty: evaluate retrieval, authority, and synthesis separately.
What the strongest public test found
The 2025 Stanford study Hallucination-Free? was a preregistered evaluation of more than 200 queries across Lexis+ AI, Westlaw AI-Assisted Research, Ask Practical Law AI, and GPT-4. The legal RAG tools hallucinated 17% to 33% of the time under the researchers' definition, which included false statements and false assertions that a source supported a statement.
Results varied materially:
- Lexis+ AI answered 65% of test queries accurately.
- Westlaw AI-Assisted Research answered 42% accurately and hallucinated nearly twice as often as the other tested legal tools.
- Ask Practical Law returned incomplete answers for more than 60% of queries.
These results apply to product versions and queries tested. They should not be treated as current universal scores. They do disprove the idea that attaching authoritative content makes verification optional.
RAG has two independent jobs
Retrieval: Find the documents and passages needed to answer.
Generation: Produce a response that is faithful to those passages.
If the right case is not retrieved, the generator cannot cite it. If the right case is retrieved beside irrelevant text, the model may choose the wrong proposition. If the source supports only part of the sentence, the citation can create false confidence.
Evaluate these layers separately.
Build an authority-aware corpus
Every source needs structured metadata:
- jurisdiction and court or regulator
- source type and precedential weight
- decision, publication, and effective date
- citator or treatment status
- matter and client
- document type and filing status
- privilege and confidentiality
- version and supersession
- exact page, paragraph, or section boundaries
Vector similarity cannot infer legal authority reliably. Use rules and filters before semantic ranking.
Permissions must precede retrieval
Do not retrieve broadly and ask the model to hide restricted text. Enforce access at query time in the search layer. Matter membership, ethical walls, client restrictions, document-level permissions, and geographic controls should narrow the candidate set before any passage reaches the model.
Test with adversarial users and cross-matter identifiers. The target unauthorized retrieval rate is zero in the controlled test suite.
Hybrid retrieval is usually necessary
Legal questions include exact citations, rare terms, defined phrases, names, dates, and concepts. Use:
- keyword search for exact and rare language
- semantic search for concept similarity
- metadata filtering for jurisdiction, date, matter, and authority
- reranking for final relevance
- relationship retrieval for cited and citing authorities
Chunk documents by legal structure, not an arbitrary character count. Keep headings, footnotes, tables, and paragraph references.
The legal RAG evaluation set
LegalBench-RAG introduced 6,858 human-annotated question-answer pairs over a 79-million-character legal corpus, providing a useful public benchmark pattern. A 2026 Legal RAG Bench used 4,876 passages and 100 questions and found retrieval to be the primary driver of performance.
An enterprise needs its own set because practice, jurisdiction, corpus, and permissions differ. Include:
- answerable questions with one clear authority
- questions requiring multiple sources
- outdated or negatively treated authorities
- conflicting jurisdictions
- unanswerable questions
- false premises
- cross-matter permission traps
- long-document and exhibit questions
- numerical and deadline calculations
- ambiguous facts requiring clarification
Lawyers should write the expected proposition and supporting passage, not only a final answer.
Score the full chain
| Layer | Metric | Failure caught |
|---|---|---|
| Corpus | answer coverage | authority absent from index |
| Retrieval | recall at 5 | supporting passage missed |
| Rank | mean reciprocal rank | source buried too low |
| Authority | correct-jurisdiction rate | persuasive source used as controlling |
| Freshness | current-source rate | superseded law retrieved |
| Generation | claim groundedness | unsupported synthesis |
| Citation | citation precision | source does not support claim |
| Completeness | material-issue recall | important qualification omitted |
| Abstention | correct refusal rate | system guesses without evidence |
| Security | unauthorized retrieval | confidentiality breach |
Report high-risk subsets separately. A 95% average can hide poor treatment of negative authority or deadlines.
Make every sentence reviewable
For research output, link each material proposition to the supporting passage. Show whether the source is primary or secondary, its date, jurisdiction, and treatment. Separate record facts from legal conclusions.
The model should signal uncertainty in three different situations:
- no source found
- sources conflict
- source found but inference is uncertain
These require different user actions.
Human review is not one checkbox
Define review by task and consequence.
An internal discovery clustering suggestion may need spot checks. A contract-deviation summary needs clause verification. A filing requires proposition-by-proposition authority review. A deadline calculation may require a deterministic rule engine and lawyer confirmation.
Record which source the reviewer opened, what changed, and whether the correction was retrieval, reasoning, or drafting. That data improves the evaluation set.
The sanctions lesson
In Mata v. Avianca, a federal court imposed a $5,000 penalty after fabricated decisions were submitted and required notice to judges falsely named as authors. The event involved a general chatbot, not a modern legal RAG product. It still establishes the professional boundary: the lawyer owns the filing.
A production gate
Do not deploy because the answers look good. Set thresholds by use case. For internal case research, a gate might require:
- at least 95% retrieval recall at 5 on high-risk questions
- at least 98% citation precision
- zero unauthorized retrieval in the red-team set
- correct abstention on at least 90% of unanswerable questions
- no missed negative treatment in the defined test set
- at least 30% lower time to a lawyer-verified answer
The numbers are illustrative. The organization must choose thresholds that reflect professional risk.
The conclusion
Trustworthy legal RAG is a search, authority, security, generation, and review system. Citations are one visible output, not proof of correctness.
The competitive moat is a maintained evaluation set tied to the firm's matters and practice. It allows every model, retriever, prompt, and product update to be judged against the same professional standard.
Research note
Research is current through September 5, 2026. Published product evaluations describe tested versions and may not reflect later releases. This article is not legal advice.
Continue the research
- The complete legal AI model stack
- Legal speech recognition for hearings and matters
- RAG vs fine-tuning vs bespoke AI models
- Why an enterprise evaluation set becomes an AI moat
- Regulatory RAG for pharma
Building a Production-Ready System
Conscious Engines builds legal RAG that treats a citation as the beginning of verification, not the end. We evaluate whether the controlling source was retrieved, whether each claim is supported, whether permissions were respected, and whether an outdated or conflicting authority should change the answer.