[EDRM Editor’s Note: EDRM is grateful to Trusted Partner Merlin Search Technologies for permission to publish. The opinions and positions are those of the authors. All images in the article are from Merlin Search Technologies.]

We recently gave seven leading AI models the same assignment. Each one worked from the same collection of opioid litigation documents, answered the same investigative questions, and wrote a full report with citations to the evidence.

The lineup was a who’s who of frontier AI:

Then we graded every report, citation by citation, with an automated evaluation system we built for the purpose.

The results surprised us. The most expensive model cost more than nine times as much per report as the least expensive one. It was not more accurate. Neither of the two newest frontier releases finished first. And the model that ranked first was also the cheapest and the fastest.

We are not going to tell you which model is which. Not here, anyway. Our graders saw the models only as Model A through Model G, and so will you. Feel free to make your own deductions. If you want the answer key, send John a message and he will share it.

What we want to show you is how we got these results, because the method is the real story. It is also a big part of what makes Alchemy different.

The question every lawyer should be asking

AI-generated reports look good. They are well organized, fluent and packed with citations. Put seven of them side by side and every one reads like the work of a capable associate.

But a lawyer can’t rely on how a report reads. The question that matters is simple:

Are the citations real, and do the documents actually say what the report claims?

Checking that by hand is brutal. A single report can carry two hundred citations. Each one means opening a document, finding the paragraph and reading it against the sentence that cites it. In this run alone, the seven models produced 777 citations. Checking them by hand would take a lawyer a week. And it would have to be done again every time a model is updated, which these days is every few weeks.

So we automated it

Why we can run this test at all

Most legal technology platforms could not run this comparison. They offer their users one AI model, or one provider’s family of models. The vendor picked it, and that is what you get.

Merlin took a different path. Alchemy is an investigation and discovery platform that gives users a real choice across the leading AI providers: OpenAI, Anthropic and Google. The bench runs from fast, economical models built for summarizing hundreds of thousands of documents to the most powerful frontier models available. GPT-6 Astra and Claude Fable 5.1 are the newest additions.

That breadth is what makes a head-to-head test possible. It is also why the test is necessary. If we are going to put a model in front of lawyers working on bet-the-company matters, we need to know how it behaves on legal work, not how it performed on someone else’s benchmark.

New models get no free pass. They go through the same test as every model already on the bench.

The test collection

We did not test on client data. Instead, we used publicly available documents from the opioid litigation. The collection includes internal company materials, customer correspondence and analyst commentary involving Mallinckrodt, one of the largest opioid manufacturers, along with materials touching McKesson, Insys and Valeant.

It is a good proving ground because it looks like real discovery. It is messy, spread across document types, full of business jargon, and it involves products, regulators, billing disputes and financial pressure all at once.

We asked each model two questions an investigator might actually ask:

The second question has a trap in it. The documents discuss credit problems, but much of that commentary is about other companies, and much of the Mallinckrodt material shows routine billing disputes rather than a company in crisis. A careless report blurs that line. A careful one draws it.

How the automated evaluation works

This is the heart of it. The system runs in eight passes, each designed to answer one question precisely. What is striking is how few of them use AI at all.

Checking whether a cited document exists is a lookup. Checking whether paragraph 338 falls inside a passage covering paragraphs 327 to 337 is arithmetic. Measuring how much a report repeats itself is linear algebra. Using a language model for any of that would make the results slower, more expensive and, worst of all, impossible to reproduce. So everything that can be arithmetic is arithmetic. AI is used only where judgment is genuinely required.

1. Hide the names

The system replaces each model’s name with a random label, fixed for the whole run, and scrubs the reports for accidental mentions. Every later stage sees only the labels, so the AI doing the grading never knows whose work it is judging. A recorded seed number lets any run be reconstructed exactly.

2. Audit the source material first

Alchemy reports are written from document summaries, and each summary carries markers showing which original paragraphs it drew from. Those markers come from an earlier stage and are sometimes wrong. A model that faithfully cites a bad marker shouldn’t be blamed for it. So the system audits the summaries before checking any report. In this run it found 24 summary sections with defective markers, and excused every citation that failed only because of one.

3. Check every citation mechanically

Every citation is parsed and resolved against exactly what that model was given. Does the document exist? Does the section? Does the paragraph fall inside the range that section covers? Each failure gets a specific reason. A document number mentioned in passing, without a section or paragraph, is treated as prose rather than a broken citation. A metric that cries wolf gets ignored.

4. Test every claim for grounding

This is the first place AI is used. Each report is split into individual cited sentences, 454 in this run, and each is checked against the summaries it cited.

Our first version got this wrong. It treated any fact missing from the cited sources as invented. But our models read every relevant summary and are limited to ten citations per point, so a sentence drawing on fifteen sources would have five real facts scored as fabrication. Good synthesis was being punished as dishonesty.

Now the check runs in two stages. The first can only report what it couldn’t find. When something is missing, a second stage searches the rest of the collection and assigns one of four verdicts:

Fabrication counts three times as heavily as anything else. That is the failure that ends up in a brief.

5. Measure repetition and breadth

The system converts each sentence into a vector that captures its meaning, then decomposes the report mathematically to count its genuinely independent ideas. That yields breadth, the number of distinct threads covered, and redundancy, the share of sentences that add nothing new. Together they cancel out length, so neither the longest nor the shortest report gets an unfair edge.

6. Score, rank and call a tie a tie

Most measures are rates, so a thorough report isn’t penalized for having more to check. Fabrications are counted, not averaged, because one invented figure is disqualifying however long the report is. Cost and speed are shown but never affect the ranking.

Each score carries a confidence range calculated by resampling the test questions. When ranges overlap, the models are labeled as tied. A clean first-to-seventh ranking would claim more precision than the evidence supports.

7. Grade the grader

This is the part we are proudest of, and the part we suspect that few others do. Two more AI graders, from different providers, re-read every flagged claim plus a random sample of passed claims. Majority vote settles disagreements. In this run:

So you see not just what the system concluded, but how often it is likely to be wrong.

8. Write the memo, without letting AI touch the numbers

An AI writes the memo’s prose, but every number comes from the computed metrics, and the scorecard table is generated by code. Any weakness the memo names must also name the model that does better. Appendices list every individual finding, down to the sentence and document, so anything the report asserts can be checked.

What we found

Here is a simplified version of the scorecard. For the fidelity score, lower is better.

A few things stand out.

No model made anything up. Every specific fact in every report traced back to the documents. For lawyers who worry about AI inventing evidence, tthat is the headline. It is also a result we can state with confidence only because we checked every claim and then checked the checker.

Price did not buy accuracy. Model C was the most expensive at $3.05 per report, more than nine times the cost of Model D. Model D ranked first. Model C ranked fourth.

Newest did not mean best. Neither GPT-6 Astra nor Claude Fable 5.1 finished first. That doesn’t make them weak models. It means that on this task, being newest wasn’t the same as being most accurate.

Different models have different strengths. The numbers show real tradeoffs, not a single winner:

• Model D had the cleanest citation record at the lowest price, but covered fewer threads of the matter.
• Model A covered the most ground of any model, with no broken citations, at about three times Model D’s cost.
• Model G covered nearly as much ground as Model A, but roughly one in ten of its checked claims pointed to the wrong paragraph. The facts were right. A lawyer following the citation would just have to hunt for them.

Six of seven models were statistically tied. On accuracy alone, this run could not separate Models D, A, F, C, G and B. Within that group, the right choice depends on cost, speed and how much of the matter you need covered. Only Model E sat apart.

What this test does not tell you

We state this in the report itself, and we want to state it here.

This evaluation measures one thing: whether a report’s statements trace back to their sources. It does not measure whether a report reasons well, reads well or identifies what matters most in the case. A model could top this scorecard by listing facts carefully and never drawing a useful conclusion.

That matters, because the hardest work in litigation is judgment. Reconciling conflicting evidence. Understanding why something happened. Anticipating the other side’s argument. That is where we expect frontier models to earn their higher cost, and it calls for a different kind of test. We are building it.

A number that looks like a quality score will be read as one unless it says otherwise. So ours says otherwise.

Why this is the case for choice

If one model won every test, there would be no reason to offer more than one. The testing shows the opposite.

Alchemy uses AI in two stages. First, a model summarizes and evaluates potentially relevant documents across the collection. Then a model works across those summaries to answer questions, synthesize evidence and write reports. Those are very different jobs, and we test for both.

In our summarization testing, GPT-5.6 Luna has been exceptionally fast and inexpensive, while Gemini 3.8 Flash delivers deeper document understanding and stronger relevance ranking. In the synthesis test described here, the least expensive model delivered the cleanest citations, and a mid-priced model delivered the broadest coverage.

And the leaders keep changing. A model that tops this scorecard today may be passed next month. Because Alchemy offers the leading models from OpenAI, Anthropic and Google, our clients don’t have to bet their case on any one AI company getting it right. We test the new arrivals, publish what we find, and help clients pick the right model for the work in front of them.

Use economical AI where it makes sense. Use frontier AI where it matters most. And know which is which, because someone actually checked.

Assisted by GAI and LLM Technologies per EDRM’s GAI and LLM Policy.