Courts across the country are increasingly sanctioning attorneys whose filings cite cases that do not exist, quote language never written, or misconstrue holdings. Filevine’s Legal Operating Intelligence System (“LOIS”) catches those errors at the draft stage.
Filevine’s internal legal research team tested LOIS against briefs and pleadings that had previously been sanctioned or admonished, running the same prompt across each document. The starting pool contained 1,314 potential cases. After filtering to attorney-filed, federal matters with accessible records, 68 remained. The team compared LOIS’s output on each offending filing against the court’s admonishment or sanctions order. Across the reviewed sample, LOIS audited 2,073 individual citations. It verified 61.8% of those citations, flagged 38.2% for a substantive issue requiring attorney review, and identified 8.4% as “severe” (generally meaning mischaracterized holdings or citations to the wrong, or non-existent, case). Every document LOIS reviewed produced at least one issue for attorney review. It identified fabricated citations and altered quotations in multiple cases and flagged mischaracterized authority the court later rejected.
LOIS Review: Methodology and Dataset Selection
The evaluation began with a database of 1,314 potential cases in which a judge had identified potentially hallucinated or other problems as a result of admitted or suspected use of generative AI tools. Because state court filings proved less accessible, the review was limited to federal filings, and then focused on matters in which an attorney, not a pro se litigant, submitted a filing containing problematic citations. That produced an operative sample of 103 cases, 68 of which had records available for full analysis. The other 35 were excluded because 12 involved sealed, redacted, or inaccessible documents; nine involved rulings finding no AI hallucination, usually misquotation or accidental misattribution; and 14 could not be located after a reasonable search, often because they arose from bankruptcy filings. For each of the 68 in-scope matters, Filevine retrieved the court’s order and the related offending brief, memorandum, or pleading.
After collection, each offending brief or pleading was run through LOIS with the same prompt, and its output was cross-referenced against the court’s order. LOIS drew upon its case corpus database only, and was not permitted to do a generalized search of the web to prevent pulling articles or blog posts about any reported sanctions that may have identified the issues with the filings analyzed. Where the order enumerated specific citations that the court found to be hallucinated, the review analyzed whether LOIS identified those court-identified citations. Where the order flagged only a subset of defects or described them broadly, LOIS’s output lacked a specific benchmark for comparison. For this exercise, LOIS’s output could be measured against the citations the court named, but not against a hypothetical complete list of every defect the filing might have contained. Comprehensive detection and false-negative rates require a fully annotated dataset and are left for future work. This scope condition is revisited in the qualifications and limitations section below.
As part of the review, LOIS was instructed to look for three categories of hallucinated materials:
- Fabricated case law
Citations to cases that do not exist in any federal or state reporter. These are typically generated wholesale, with a plausible caption, reporter cite, and court designation.
- False or misattributed quotations
Typically, the filing cites a real case, but the quoted language either does not appear in the cited opinion or has been materially altered.
- Misrepresented or mischaracterized authority
The filing cites a real case and quotes it accurately but relies on it for a proposition it does not support or describes its holding in a manner inconsistent with the opinion. A related subset, fabricated legal norms, captures invented or altered statutory text. LOIS was instructed to flag this subset as well.
LOIS Review: Findings
Fabricated case law
Across the sample, LOIS identified fabricated citations in multiple matters and flagged additional sources, citations, and quotations that should have been verified before a filing went out. What constituted a “fabrication” was either an impossible citation (e.g., LOIS flagged Renschler v. Yosh, 909 F. Supp. 2d 1197, where no case by that name surfaced anywhere in the reporter database) or that cited to a real reporter address occupied by an entirely different case.
False or misattributed quotations
LOIS also identified altered quotations from cited authority in many of the filings. The pattern that most often escapes conventional cite-checking is when a cited sentence is a misquotation. For example, in one reviewed filing, a Rule 37 sanctions argument quoted Shcherbakovskiy v. Da Capo Al Fine, Ltd. for the proposition that dismissal requires “willfulness, bad faith, or any fault of the sanctioned party” (emphasis added), when the Second Circuit’s opinion actually says “of the deponent” (emphasis added), a small and potentially inconsequential substitution in this instance, but in other instances may misconstrue the rule far beyond what the court wrote. When instructed, LOIS surfaces that class of error because the citation itself is real; only the quotation is either fully or partially incorrect. LOIS also flagged “unverified” citations that appeared to undermine the argument it was offered to support, signaling a case was likely fabricated.
Misrepresented or mischaracterized authority
Then, in what Filevine perceives to be the exercise’s most critical observation, across multiple reviews LOIS confirmed that a case citation was potentially legitimate but flagged that the case either (i) did not stand for the proposition asserted, (ii) did not contain the argument attributed to it, or (iii) contained citations that LOIS could not access in order to confirm the authority. The same thing happens at the statutory level. In one reviewed filing, a brief cited Watson v. State of California, 21 Cal.App.4th 836, for the proposition that California Government Code § 845.6 imposes an affirmative duty on the public entity to summon medical care. But the opinion holds the opposite: § 845.6 confers broad general immunity on the public entity, with liability attaching only for intentional or unjustifiable failure to furnish immediate care in serious, obvious cases—and the plaintiff in Watson itself lost. So, while the citation was accurate and the statutory text appears in the opinion, the direction of the holding was inverted, resulting in a critical error. LOIS’s review can surface these types of inversions for the practitioner’s more careful review.
The clearest adjudicated example of the same failure surfaced in a federal case in the Western District of Michigan, Nai v. National Asset Mortgage, LLC. The offending party’s motion for summary judgment quoted the Real Estate Settlement Procedures Act’s (“RESPA”) statute of limitations, 12 U.S.C. § 2614. The party presented the quote as running “from the date of the first occurrence of the violation.” Defendant’s MSJ at p. 11 (emphasis added). However, the statute reads “from the date of the occurrence of the violation”—no “first.” One inserted word changed an occurrence-based limitations trigger into a first-occurrence trigger, which counsel used to argue that two full tax years of the plaintiff’s claims were time-barred. The court found the quotation fabricated, and accepted counsel’s on-the-record admission that the misquotation “materially altered the statute’s plain meaning.” Identifying a single, silently added word that changes a cited rule is precisely the class of error LOIS is built to flag at the draft stage.
Not every problematic citation is a hallucination, and LOIS is designed to reflect that distinction. In Sherwood v. County of Botetourt, Virginia, et al., the offending party quoted two cases accurately with correct citations, but the underlying facts of those cases were distinguishable from the proposition the plaintiff sought to draw. LOIS correctly declined to label those citations as hallucinations, because the quotations and citations themselves were accurate. Whether a distinguishable authority may support a party’s argument is a judgment call that belongs to the attorney. LOIS is designed to support that judgment, not to make it.
Notable qualifications and limitations
In Roger Patel v. Chandresh Patel, LOIS characterized the cases as “unverified or ambiguous” rather than “not found,” consistent with the parties’ declarations and the court’s findings.
LOIS also did not flag passages a party attributed generically to “the court” without a case citation, because those passages offered no cited authority to check. In limited situations, the court flagged that the offending brief also misconstrued the court’s own ruling. These mischaracterizations were not included in the review.
Lastly, prompting is critical, and Filevine has best practice guides in this regard. For example, if prompted incorrectly, LOIS may label a citation as “verified,” indicating it is a legitimate case, while qualifying its conclusion with further instructions for the reviewer. For example, where a citation appears in a string cite with a parenthetical, LOIS may report that it cannot determine whether the parenthetical is accurate and instruct the user to review or confirm that language. With the correct prompt, LOIS takes the further step of reviewing and confirming the parenthetical’s language itself.
Scope conditions bear on these findings. The courts’ orders did not always enumerate every hallucinated citation, quotation, or characterization in the offending filing. The review therefore reports how LOIS performed against specific, court-identified defects rather than a comprehensive detection, false-negative, or false-positive rate across every possible defect. Even without direct access to those databases, LOIS still caught quotations that misrepresented the authorities cited to support them.
These are limits on what the legal research team could measure LOIS against, not on what it caught.
Overall Assessment
Across the reviewed sample of 2,073 citations across 68 cases, LOIS verified 61.8%, flagged 38.2% as needing attorney review, and identified 8.4% as severe (either mischaracterized holdings or citations that resolved to the wrong case entirely). Among the highest-citation documents in the corpus, none was fully clean; the cleanest still had roughly 9% of its citations flagged, and the most egregious had 76% flagged. LOIS identified fabricated case law and false quotations in multiple filings and, in every document reviewed, flagged at least one issue for attorney review. In matters where the offending citation had already been flagged by a court, LOIS caught the same defect that drew judicial rebuke, and it identified authorities cited for propositions that contradicted the underlying argument. Some surfaced items required independent verification and were not confirmed hallucinations, which is by design: LOIS surfaces uncertain citations for attorney review instead of guessing. Used that way, LOIS can help a practitioner avoid public embarrassment and, in some cases, sanctions or fines. Every document filed with a court or received from opposing counsel should be run through an anti-hallucination tool like LOIS. LOIS also gives the attorney context on the cited cases and authorities themselves to support the practitioner’s own analysis. These results measure LOIS against what courts flagged in the sample, not a comprehensive detection rate. Even so, in every reviewed document LOIS surfaced something worth a second look before filing.