Ask a legal AI system a question about a case, and before it can say anything useful it has to make a decision nobody asked it to make: which of the case's documents to read. On a small matter that decision is invisible, because the answer is all of them. On a case of 40,000 pages, or across a portfolio of cases, it quietly becomes the decision that determines whether the answer is worth anything.
We have argued before that retrieval is the cornerstone of legal AI, and that surfacing the right context is a prerequisite for a correct, actionable result. Vector and keyword search are how a system finds documents that resemble a question. Resemblance, though, does not necessarily mean relevance. A demand letter arguing about a plaintiff’s spinal injury resembles a question about that injury very closely. It is also close to the last thing you want the system to answer from, because it was written to persuade, rather than to record.
What separates demand letters from medical reports is not what they say; it is what they are. A system with no representation of what a document is will keep making that same category of mistake, regardless of how good its search becomes.

A demand letter and an MRI report can both match the topic of an injury. For this question, the relevant scope is only primary records about the injury topic. A legal AI system without classification will not be able to distinguish between those.
This article is about the other half of retrieval: the layer that describes every document in a case before any question is asked of it. We cover why the description needs two dimensions rather than one, why legal documents resist classification more stubbornly than most people expect, and how we ended up compressing the judgment of several frontier models into a small model we run ourselves. How this feeds scoped retrieval inside LOIS, Filevine’s agentic system, is the subject of a follow-up article.
Two questions about every document

A predefined list of document categories can never capture the full variability of real documents. Sooner or later some document that doesn’t fit any category will arrive, and either the taxonomy needs to be extended or this document will end up unclassified. An unfamiliar document type, however, still has a recognizable subject and form. The examples on the right receive combinations of existing labels without requiring a dedicated category for each new type.
A case file is heterogeneous. It can hold the maintenance history of the car involved in the accident, an ER intake form, six months of physical therapy notes, an employer wage record, a deposition transcript, a scheduling order, and a demand letter. Ask what distinguishes those documents, and you get two answers, not one.
The first is subject: what the document is about. Medicine, money, insurance, employment, real property, and so on. The second is form: what role the document plays in the case. Is it a primary factual record, a court filing, expert analysis, testimony, advocacy, or internal work product?
These two questions are independent, and both of them matter for retrieval. When a user asks about the plaintiff's medical condition, we want the system reading medical records rather than the brief that argues about those records. When a user asks what actually happened, we want primary records rather than anyone's characterization of them. A taxonomy built on subject alone cannot express the second request, because the treatment notes and the demand letter discussing them are both medical. A taxonomy built on form alone cannot express the first.
Thus, the taxonomy we developed has two axes, each with about a dozen classes. Both were designed in collaboration between machine learning specialists and legal domain experts; we did not derive them from whatever happened to be in the data. A document can carry more than one value on each axis, which is simply a concession to reality. A medical care bill is both a financial and medical item at the same time. A treating neurologist’s deposition is testimony, discovery material, and about medicine all at once. Most documents carry between one and three values per axis.
Allowing several values is what makes the scheme useful, because it turns a question into a combination instead of a category. Consider something a litigator wants constantly: show me what happened, not what anybody said about what happened. That resolves to primary factual records, together with discovery materials, but excludes advocacy documents and internal work product, with no constraint on subject whatsoever. Internal work product, strategy, and case management is a single value on the form axis, so those documents can be excluded wholesale, irrespective of what they are about.
The more conventional approach gives every document type a class of its own: Deposition Transcript, Physician Notes, Demand Letter, Will, Deed, etc. That list is never finished. A document that was never anticipated has nowhere to go, and an AI system asked to choose among three hundred categories would be significantly less reliable. A couple of dozen of combinable values describe hundreds of real-world document types without anyone having to enumerate them, and a genuinely novel document still lands somewhere sensible.
Legal documents resist tidy categories
Having settled on what to ask about each document, we ran into the part we had underestimated. Legal classification is not just fuzzy at the edges; it is highly ambiguous and open to interpretation.
We can quantify this ambiguity. Several frontier models read the same pages under the same instructions, working independently of one another. They disagreed on roughly a quarter of the corpus.
That disagreement reflects, at least in part, the ambiguity of legal documents rather than model performance alone. Consider a defense-ordered independent medical examination. Its subject is plainly medical, but its form is contested: it is expert analysis, and it is discovery material, and there is a real argument that it is advocacy, since a party procured it to make a point. Consider a hospital lien letter. Its subject is medical, financial, and insurance-related, and its form is both a communication and an instrument creating a security interest. Consider a Bates-stamped photograph of an intersection attached as an exhibit to an expert report. It is a photograph, it is discovery material, and it is part of an expert’s analysis, all at once, and its subject depends entirely on what the case happens to be about. None of this is sloppiness. It reflects the practical nature of law. Law solves problems, and it does not care very much about formal categories. Documents get created ad hoc, tailored to the case sitting in front of the lawyer, and no taxonomy imposed from outside will ever fit them perfectly. Our two-axis scheme absorbs a good deal of this, precisely by refusing to force a choice the world does not offer. It does not absorb all of it, however: some degree of ambiguity will be naturally present in any legal classification system.
A panel of experts, compressed
Hand-labeling tens or hundreds of thousands of legal documents would be prohibitively expensive and time-consuming. Moreover, it would not be sufficient due to subjective interpretation of some document classes, and labeling the same set by multiple experts would explode the costs even more. A realistically small, manually-annotated document set would have been too thin to train on. So AI models produced the labels, which relocates the engineering problem from annotation to label quality control.
Our answer to the ambiguity above was consensus. A panel of frontier models votes on every document. Every one of these experts carries its own biases. Combining their judgments can reduce the effect of any single model’s errors, although it does not eliminate shared bias or establish a definitive ground truth. That consensus judgment became our training target.
Calling frontier model for every document creates recurring costs that increase with document volume. We instead used frontier models during dataset creation and trained a specialized model that we operate within our own infrastructure. This approach allows the classifier to run efficiently at scale.
The fact that this approach works at all deserves a word of explanation, because it is not obvious. Frontier models could do this task on day one, so capability was never the question. Using one is simply inefficient in a particular way: classifying legal documents uses a tiny portion of what a frontier model knows, and you pay for the whole of it on every document, in money and in time. A small model can be specialized for what matters for this specific task. Our own results make the point rather bluntly. On our evaluation set, fine-tuning significantly improved the small model’s classification quality, matching the frontier models we tested at a tiny fraction of cost and latency. The model that comes out the other end is not much good for anything else. That is what specialization means, and the model is one component within LOIS rather than LOIS itself.
Every label arrives with its evidence


Our model starts with evidence before classification. This makes results interpretable, and provides additional useful information to LOIS. For this hospital lien letter, the model extracts identifying phrases before assigning labels.
One design choice is worth describing, because it changes what the output is for. The model first identifies concise evidence from the document that supports its classification. The labels follow, allowing developers and reviewers to evaluate the document-based support for the model’s output.
This gives us four things: (1) The classification is anchored to text on the page rather than floating free of it; (2) This is a form of "constrained reasoning", which does not require burning hundreds of tokens before answering; (3) when a label turns out to be wrong, we can usually see at a glance whether the model misread the page or misapplied the taxonomy–hose are different problems with different fixes, and most classifiers offer no way to tell them apart; and (4) those pieces of evidence can be directly used by LOIS to understand the case structure without reading every document directly.
The final classifier: results
In our internal evaluation, the fine-tuned model achieved results comparable to the frontier models we tested while operating at substantially lower estimated cost. Our measurements show that it is four- to twenty-times less expensive than the small hosted models we evaluated as alternatives, while achieving better quality metrics. Compared with calling several frontier models for every document, the estimated cost reduction is almost 1,000-fold.
We deliberately biased the model toward recall rather than precision, reflecting its role as an initial filtering stage for the LOIS information retrieval. In other words, we encouraged the model to produce more labels per document, even when it was less certain about them. This matters significantly when we compare the costs of different types of errors: a document assigned an uncertain label may be read unnecessarily, but this is unlikely to affect the quality of the response. In contrast, a document missing a relevant label will not even enter the LOIS reading list, and its information may be lost.
After training, we evaluated the model on selected document types that could be assigned to specific labels with a high degree of certainty, i.e., cases outside the region of legal ambiguity. For example, the custom model correctly classified medical documents as medical in 99.8% of the tested cases, suggesting that the risk of missing important evidence is low.
The bundle is not a document
Long concatenated PDFs are a known challenge. A bundle might be hundreds of individual documents of quite different kinds, glued together into one file because that is how they were produced, served, or scanned. Our model reads the bounded window of the most informative pages. For the overwhelming majority of documents that is the right economy, since the letterhead, the caption, the form number, and the signature block are almost always where a document announces what it is.That economy fails for a thousand-page bundle whose substance begins on page 40.
To handle bundled files, the system identifies likely document boundaries and separates the file into sub-documents before classification. Each part can then be described on its own terms, improving the usefulness of retrieval; for example, a search for primary medical records can return the relevant portion of a bundle rather than the bundle as a whole.
What comes next

Real legal documents span multiple subjects and forms. At ingestion, our classifier captures that complexity by producing combinations of labels describing the document. When a user asks a question, LOIS can select a combination of filters which best reflect the user’s intent. In this example, the query tells the system to focus on primary records and discovery materials, while completely excluding advocacy and internal work product. The hospital lien letter and deposition therefore remain in scope, while the demand letter and strategy memo are set aside. Documents are selected because their forms match the question, rather than simply because they discuss the case. In turn, it helps LOIS to read only what’s necessary, reducing distractions and improving response time.
The classifier is now deployed, describing documents as they are ingested. What LOIS does with those descriptions is the next thing we ship: retrieval that scopes itself to the question it was asked, so that a question about medical causation reads medical records, and a question about what happened reads primary records rather than arguments about them.
For a lawyer working a 40,000-page case, the goal is concrete: when they ask what the medical records show, the system should find those records and ground its answer in them. Document classification gives retrieval the information it needs to make that distinction. That is the difference between a system that searches everything and a system that knows where to look.
