At ML Prague, I gave a talk on a problem that doesn't get enough airtime: how do you actually know whether a legal AI system is any good? Much of the energy in our field right now goes into the generation side - bigger models, smarter agents, and more capable tool use. That's fair, and it matters. But the longer my team has been shipping AI into real legal work, the more I've come to believe that the hardest part isn't producing an answer. The hardest part is knowing whether the answer is good in a way a lawyer can stake their work on.
That turns out to be a very different question than the one most public AI benchmarks were designed to answer.
What we're actually evaluating
A quick framing, because it matters for everything that follows. We're not evaluating a generic assistant over a single document. The systems we build operate over full case PDFs, scanned records, contracts, emails, handwritten forms, calendar events, deadline chains, audio, video, and structured fields. Increasingly, they also operate across entire client portfolios, where a single question can sweep thousands of documents at once. And every output has to be traceable to a source a lawyer can independently review and verify.
LOIS is the agentic layer most users see. Underneath it sits what we call the Knowledge Engine: the ingestion, enrichment, and retrieval system that has to make sense of this corpus at scale. That's the piece I'll focus on here, because it's where the evaluation problem gets genuinely interesting.
Throughout the rest of this piece, I'll use one query as a running example: ‘List all treatments in this case.’ The same query exposes very different problems at every level of the system. There is rarely a single, clean ground truth: experts may disagree about what counts, source records may conflict, and human labels may be incomplete. Various failures occur at different levels, and specific feedback loops are useful for particular problems.
Three feedback loops, three levels of evaluation
Internally we call this approach evaluation-driven development. The idea is simple: instead of treating evaluation as a checkpoint at the end of a build cycle, you let it shape what you build, in what order, and how you know it's working. That only works if your evaluation is honest about the failure modes that actually matter.
We think about it along two axes.
The first axis is where the signal comes from. We rely on three feedback loops:
- Domain experts - our in-house attorneys and paralegals, who give us judgment we cannot get anywhere else.
- An annotation team - a dedicated resource that produces labeled, structured data at scale.
- Production feedback - how real users actually interact with the system day to day.
The second axis is what level of the system the signal tells us about:
- Answer-level - properties you can judge by reviewing an individual response and its supporting sources.
- Case-level - properties that require understanding the whole case the answer was drawn from.
- Population-level - properties that only show up when you look across many cases, queries, and users.
Drawing it as a 3×3 matrix is a useful exercise, because it forces you to be honest about which combinations you actually have covered and which you've been hand-waving past. The thesis underneath the whole framework is simple: different feedback loops are useful for different failure modes. The job isn't to pick one. The job is to match the eval to the failure mode.
What an expert can judge at the answer level
Start with the answer itself. Take our running query: ‘List all treatments in this case.’. The system returns a tidy list with citations attached. Five things matter to a legal user looking at it: style, relevance, faithfulness to the source, citation correctness, and citation completeness.
These are dimensions an expert can judge directly, usually in a side-by-side comparison where two candidate answers are scored against each other. That gives us two things at once. It gives us an immediate signal we can feed back into the system. And it gives us a growing dataset we can use to train automated judges that can run at the scale humans cannot, while still tracking expert taste.
The calibration step is the one I'd encourage anyone to take seriously. An automated judge that hasn't been validated against expert judgment and periodically recalibrated is just a confident opinion. The point of the expert loop isn't only to score today's answers; it's to earn the right to trust the judge that scores tomorrow's.
A polished answer can still be incomplete
This is a failure mode that answer-level review often misses, and it can be especially damaging in legal work.
Back to our query. The system returns eight treatments. They're correctly named, well cited, and written in a perfectly appropriate tone. By every answer-level metric, the response is excellent. The problem is that the case actually contains ten treatments, and the two that were missed are the ones the attorney needed.
To measure that, someone has to understand the full case. That's where our annotation team comes in. They work inside a framework we built specifically for this purpose, enumerating relevant passages across all the documents in a case so we can compute recall against a human-labeled reference set for list, search, and grouping queries such as ‘everything before this date,’ ‘every event at this location,’ and ‘every treatment after this surgery.’
Gold labels at that fidelity are expensive, so we complement them in two ways.
The first is synthetic cases. We've built a case synthesizer that generates complete synthetic cases with full control over their facts and document structures. We know exactly how many events occurred before a given date because we placed them there. That gives us a fast, controllable, lower-fidelity loop for things like buried-fact retrieval and completeness checks.
The second approach uses structured historical data created through routine legal work. Across the Filevine platform, customers have managed millions of closed cases. For years, attorneys have manually extracted key information from those cases into structured custom fields, and many of those fields link back to the source documents from which the information was drawn. Knowing the name of a field tells us what query it answers. The extracted value gives us a target. The linked source lets us test whether the system retrieved the document or passage associated with the extracted value. That turns years of routine legal work into a large corpus of silver-quality labels - useful but imperfect signals that may be incomplete, stale, or inconsistent which we can use to evaluate retrieval and completeness at a scale no annotation team could reach on its own.
The average score is not enough
The last level is the one I think gets the least attention in our field and matters the most in production.
Suppose I tell you a system performs at 82% overall. What have I actually told you? Nothing about which practice areas work and which don't. Nothing about whether search queries are great and chronology queries are broken. Nothing about whether we perform strongly on plaintiff-side fact patterns and poorly on defense-side matters.
So we slice. We break the system down by practice area - personal injury, criminal law, family law, and patent litigation because the document distributions and the questions attorneys actually ask are wildly different across them. We slice by query category, from simple lookups to full case summarization. We slice by the data modality the query requires, because multimodal questions over scanned records and audio behave nothing like questions over plain text. We slice by the attorney's posture in the matter (plaintiff or defense) and by who the question is actually about: their own client, a deponent, opposing counsel. Both axes change what "a good answer" even means.
Take the treatments query one more time. It's useful on personal-injury and medical-malpractice cases, where the medical record is the spine of the matter. It is usually irrelevant in a patent dispute and highly context-dependent in a criminal case. An overall accuracy number represents an average across all of these and can conceal substantial differences among them.
The same slicing logic also lets us configure the system for different kinds of work. Once you can see that personal-injury chronology questions behave differently from patent-law search questions, you stop trying to ship one generic pipeline and start tuning retrieval, ranking, and synthesis for each slice. Production feedback helps show which slices users engage with most and where they struggle. We weigh that alongside expert judgment because low-frequency tasks can still carry significant legal risk.
Evaluation helps us decide what to build next
If I had to leave you with one idea from the talk, it would be this: reliable legal AI doesn't come from one benchmark. It comes from combining the right feedback loops at the right level of evaluation.
Experts are strongest where the question is about answer quality. Annotators are strongest where the question is completeness. Production data is strongest where you're trying to see which segments matter most and where the biggest gaps live. No single feedback loop is sufficient on its own; together, they provide a more reliable basis for shipping a system lawyers can rely on.
The other thing I've come to believe is that evaluation, done this way, stops being a measurement function and starts being a planning function. For example, if cross-case medical retrieval is the slice where demand is highest and quality is weakest, you have a clear signal about what to investigate next. The data narrows the decision and gives the team evidence for prioritization. Evaluation is not just measurement; it is one of the ways we decide what to build next.
That's the bet we're making with the Knowledge Engine and LOIS. The teams that pull ahead in legal AI over the next few years won't be the ones with the biggest model on the marquee. They'll be the ones who can answer a harder question with evidence: where does our system actually work, where does it fail, and how would we know if that changed?
That is the standard legal AI should be held to. Not whether it can produce a convincing answer, but whether we can demonstrate where that answer is reliable, where it falls short, and whether the system is getting better. In legal AI, trust is not something you claim. It is something you measure.