Why legal LLMs hallucinate and how to prevent them

Lawyers trade in facts, citations, and exact precedents. Large language models trade in probabilities. When you force a probabilistic engine to do a deterministic job, it occasionally fabricates information. We call these fabrications hallucinations.

To stop an LLM from inventing fake case law or drafting fictional contractual clauses, you must first understand why it lies. The answer lives in its architecture, its memory, and how we trained it.

Part 1 — Why it happens#

Architecture#

The guessing engine#

LLMs do not retrieve information; they generate it. At their core, they play a massive version of Claude Shannon’s guessing game. When you ask a question, the model calculates the most mathematically probable next word based on the words that came before it.

The model optimises for plausibility, not truth. It strings together legal jargon because those words frequently appear together in its training data.

Memory#

The illusion of memory#

LLMs do not store databases of legal documents. During training, developers compress billions of pages of text into abstract mathematical weights. The model loses the exact text but retains the underlying patterns.

Think of a desperate law student sitting the bar exam. They need to cite a criminal appeal but cannot remember the exact name. Instead of leaving it blank, they confidently write Mwangi v Republic. It perfectly mirrors the naming convention of a Kenyan criminal case.

LLMs do exactly this. When knowledge gaps appear, the model uses its compressed mathematical patterns to construct a highly convincing fake.

Signal lost in the noise (working memory)#

Modern LLMs let you upload massive documents directly into the prompt. But dumping a 500-page case file into the context window triggers context dilution.

Imagine a paralegal handed a single highlighter with only enough ink to mark 5% of any document, no matter how long it runs. On a 10-page brief, that ink is enough to flag every controlling clause, every material date, every operative word.

Now hand her a 500-page case file instead. The same 5% budget must now stretch across fifty times the text, so the ink runs so thin that only stray, scattered words get coloured — while the one clause that actually decides the case goes unmarked.

The model works the same way: it has a limited attention budget to spread across whatever you feed it. The controlling clause is the specific answer you need. The hundreds of pages of boilerplate, procedural history, and irrelevant exhibits are the extra pages competing for that same fixed ink. The model doesn’t just forget the answer; it never had enough attention left to mark it in the first place.

When you overload the AI with text, it drops details, loses track of the core argument, and fills the forgotten gaps with hallucinations.

Training#

Rewarded for guessing#

We trained AI to bluff. Most AI benchmarks and evaluation tests mirror multiple-choice exams. If the model outputs “I don’t know,” it scores a guaranteed zero. If it guesses, it secures a non-zero chance of being right.

Over billions of training cycles, the model learns a perverse lesson: admitting ignorance guarantees failure, but guessing optimises the score. So when it faces a complex legal query it cannot confidently answer, it strategically gambles instead of conceding defeat.

Popularity over accuracy#

LLMs build their worldviews by ingesting the internet — a chaotic mix of facts, opinions, outdated laws, and outright falsehoods. The model lacks a reliable internal mechanism to separate a peer-reviewed law journal from a misinformed forum thread.

Because the AI relies heavily on pattern recognition, volume beats accuracy. If a legal misconception appears across thousands of websites, the AI absorbs it as truth. Wrong information in the training data misleads the model into confidently passing off a popular lie as an established fact.

The yes-man complex#

Developers refine LLM behaviour using human feedback. Because humans inherently prefer AI that validates their assumptions, the model learns a dangerous habit: agreeing with the user yields a higher reward than correcting them. It becomes a sycophant.

Treat the LLM like an overly deferential junior clerk. If you walk in and ask, “Didn’t the Supreme Court overrule this in 2024?”, the clerk panics, assumes the senior partner knows better, and scrambles to invent a fictitious ruling to prove you right.

The model blindly adopts the premise of your prompt. This means leading questions breed hallucinations. If you ask an open question like, “What are the exceptions to the privity of contract?”, the model provides a factual list. But if you ask, “How does the Smith exception apply here?” — and no such exception exists — the AI will fabricate a complete legal history for the Smith exception just to give you the answer you seemingly want.

Understanding why an LLM hallucinates shifts your perspective. You stop treating it like an infallible database and start treating it like a brilliant, eager-to-please associate who occasionally makes things up to avoid looking bad.

Now that we know why the model breaks, we can build the guardrails to keep it on track.

Part 2 — How to prevent it#

You cannot fix an LLM’s probabilistic nature. You can, however, constrain it. Preventing hallucinations requires building architectural guardrails, changing how you issue instructions, and enforcing strict oversight.

Enforcing the boundaries (architecture)#

You cannot bolt this layer on yourself. Eliminating whole categories of hallucination means changing how the model accesses information — an architectural choice built into the platform, long before you type a prompt. It is the line between a general-purpose chatbot working from compressed memory and a purpose-built legal AI like JuriBase AI, engineered to stay inside the law. Here, your real lever is not how you prompt, but which tool you trust.

The open-book mandate. General-purpose LLMs take a closed-book exam. They rely solely on compressed memory. Purpose-built platforms like JuriBase AI take an open-book approach. Developers use Retrieval-Augmented Generation (RAG) to anchor the model to verified databases like Kenya Law or Westlaw. Instead of guessing the law, the model retrieves the actual statute and reads it before generating a single word.

Show the receipts. Force the system to prove its work. Well-designed legal AI requires the LLM to attach a clickable citation to every claim it makes. If the model cannot link its generated text directly back to the grounded database, the system rejects the output.

The ignorance tripwire. Do not trust an LLM to voluntarily admit ignorance. Developers must hardcode failure states. If the system searches the database and finds no relevant case law, a deterministic rule intervenes.

Managing the attention budget (workflow)#

You can drastically reduce hallucinations by changing how you interact with the model.

Starve the context window. If dumping a 500-page file deafens the model, stop dumping. Curate the context. Feed the model only the specific pages, exhibits, or clauses required to answer your question. Trimming the noise focuses the model’s limited attention budget entirely on the signal.

One instruction at a time. Do not ask the AI to summarise a ruling, extract key dates, and draft an appeal in a single breath. Complex, multi-step instructions overwhelm the model’s working memory. Break the work down. Ask it to extract the dates first. Then, in a new prompt, ask it to draft the appeal. Single tasks yield precise results.

The irreplaceable guardrail (human in the loop)#

The liability is yours, so the final check has to be too.

Never file a brief, send an advisory, or finalise a contract without reviewing the AI’s work. Click the linked citations. Verify the case actually exists. Read the paragraph to confirm the ruling says what the AI claims it says. Treat the model as a powerful research assistant, not a replacement for your own due diligence.

None of this makes AI the enemy. A model that drafts your letter in seconds is genuinely worth having; the only danger is mistaking its fluency for truth. Treat every output as a confident first draft and nothing more, and the technology stops being a liability and becomes leverage.

The advocates who win with AI will be the ones who learn to wield it well: quick to let it draft, disciplined enough to ground and verify every claim before it counts.