Waterloo and Elevate researchers say Redgrave’s aiR study compared two different tasks

Waterloo and Elevate researchers say Redgrave’s aiR study compared two different tasks
Image: ComplexDiscovery OÜ.

[EDRM Editor’s Note: This article was first published here on October 15, 2026, and EDRM is grateful to Rob Robinson, founder and managing director of EDRM Trusted Partner ComplexDiscovery OÜ, for permission to republish. All images in the article are from ComplexDiscovery OÜ.]

ComplexDiscovery Editor’s Note: Generative AI review got a headline benchmark in June, when a Redgrave LLP working paper reported that Relativity aiR for Review found 88 percent of responsive documents against 64 percent for a managed review run with Relativity Active Learning. A research paper posted Oct. 12 by Gordon V. Cormack, Maura R. Grossman and Jeremy Pickens reanalyzes the same data and argues the two figures measured different tasks: in one, contract reviewers had the final say, and in the other, they did not. Scored against the same human-coding gate, the authors report, aiR is comparable to, but not superior to, the managed review.

For eDiscovery counsel, information governance leaders and the compliance teams that pay for review, the exchange is a practical guide to reading any head-to-head study. It turns on who defines ground truth, which arm a human reviewer filters and what a recall figure costs. Watch for the analysis Redgrave says it has reserved for future work on a manual review of the full corpus, and for whether either paper’s framing reaches ESI protocol negotiations.

Disclosure: Rob Robinson, founder and managing editor of ComplexDiscovery OÜ, also serves as chief marketing officer of HaystackID, a provider of eDiscovery services, including document review. HaystackID had no role in the preparation of this article, which is based solely on publicly available information.

Eighty-eight percent against 64 percent was the comparison that traveled. A new research paper built on the same 45,004 documents argues the two figures never measured the same task. Measured on a common task, its authors say, the generative AI tool’s lead over the managed review it was tested against largely vanishes.

Gordon V. Cormack and Maura R. Grossman of the University of Waterloo and Jeremy Pickens of Elevate posted the paper, “Comparing GenAI to Machine Learning and Manual Review for eDiscovery,” to the arXiv preprint server on Oct. 12. It reanalyzes a June 2026 Redgrave LLP working paper, which reported that Relativity aiR for Review reached 88 percent recall against 64 percent for a managed review run with Relativity Active Learning (RAL). Redgrave shared its case-study data with the researchers, according to the paper’s acknowledgment. The Redgrave study describes itself as a working paper that has not been peer-reviewed, and the new paper is posted as a preprint.

For review teams, eDiscovery counsel and the information governance leaders who approve review budgets, the exchange is less a vendor fight than a lesson in reading a head-to-head result. The same 73 responsive documents now anchor two competing headlines, and the argument between them is mostly about study design rather than software.

Same 73 documents, two tasks, two headlines. A reanalysis finds aiR comparable to, but not superior to, the managed review.

What the Redgrave study measured

That design starts with the Redgrave paper, Working Paper 2026-01, by Robert D. Keeling, Ray Mangum, F. Eli Nelson and Kevin A. Reiss. The team ran two workflows against a corpus drawn from the Mallinckrodt collection in the UCSF Opioid Industry Documents Archive. Its standard was demanding by design. A document counted as responsive only if it contained evidence of compliance with, violation of, or reckless disregard of federal rules on how drug companies market their products, including controlled-substance requirements.

On the generative AI side, Mangum iterated an aiR prompt 43 times over about 18 hours and then ran it across the corpus in one pass. On the other side, a 24-person managed review team from Cimplifi, the review vendor, worked through 34,409 documents in the order RAL prioritized them and logged about 1,123 hours in that active learning phase. The remaining 10,595 documents, which fell below RAL’s relevance cutoff, were reviewed separately over two days, and the paper reports its hour count for the active learning phase only.

Reiss, a Redgrave counsel and one of the co-authors, then reviewed a 1,000-document random sample without seeing either workflow’s predictions and coded 73 documents responsive. Measured against his calls, aiR found 64 of the 73, while the managed review found 47.

Redgrave’s own write-ups carried that result from there. Keeling and Mangum summarized the findings in a June guest post on Relativity’s blog titled “Results Are In: 5 Lessons from an Independent Study of aiR for Review.” A Redgrave announcement posted on the Corporate Counsel Business Journal’s news and announcements page ran under the headline “Redgrave LLP Study Finds Generative AI Outperforms TAR.” That claim of superiority is the conclusion the new paper challenges.

Where the comparison splits

The research paper’s first objection turns on who made the final call. In the managed review, contract reviewers coded every document in the corpus, both those RAL prioritized and those below its cutoff. A document counted toward the managed review’s results only if those first-level reviewers, which the authors call the L1 team, coded it responsive. In the aiR workflow, the model’s label stood without that human check. The authors call these “filtered” and “unfiltered” methods and argue that setting one against the other mixes two different tasks.

The Redgrave authors describe their own number the same way. Their paper says the 64 percent figure captures the contract reviewers, the vendor’s quality checks and the calibration process working together, “not a property of the algorithm in isolation.” So the two papers agree on what the number measures, and they part company over what it allows a reader to conclude about generative AI against active learning.

A second objection concerns the yardstick itself, because recall and precision are scored against one assessor’s judgments. Of the documents the subject-matter expert called responsive, the L1 reviewers marked 34 percent non-responsive, the research paper reports. Of the documents the L1 reviewers called responsive, the expert marked 61 percent non-responsive. Scores keyed to the expert may therefore track how closely each team’s idea of responsiveness matched his, the authors wrote, and not a difference in technology.

The third objection is proportionality. The two workflows sit at different points on the trade-off between finding more and reviewing less. No production is automatically better, the authors argue, until cost, burden and risk are weighed for the parties on each side of the request.

Like against like

To test the first objection, the authors rescored Redgrave’s existing aiR and RAL results against the same human-coding gate rather than running either tool again, and the ranking changed. In the filtered task, where L1 coding has the final say for both, RAL reached 64 percent recall and aiR reached 59.5 percent, by the authors’ estimate. A continuous active learning model built on the open-source baseline code used at the TREC Total Recall tracks beat both at every point on the gain curve, they reported.

The unfiltered task leaves RAL ahead as well, at a price. When every document a system identified counts as responsive regardless of L1 coding, aiR stayed at 88 percent and RAL rose to 97.2 percent, according to the paper. Running RAL that way, the authors judged, would probably be seen as disproportionate given what the extra review would cost. Their alternative, Scalable Continuous Active Learning (S-CAL), needed L1 coding on only a weighted sample of 2,000 documents. It substantially matched both of aiR’s operating points, they wrote, and its estimated recall eventually reached 100 percent, measured against the expert’s 1,000-document sample.

Their bottom line is narrower than a reversal. Within each task, they wrote, the gaps between aiR’s results and the other methods’ curves are no larger than sampling error, and what remains carries little practical weight. For this task, the Redgrave data shows aiR “to be comparable to, but not superior to RAL,” whether the tool or the L1 reviewer makes the final determination of responsiveness, the authors concluded.

They also mark the edges of that claim. The Redgrave data reflects one matter, one team of L1 reviewers and one round of prompt engineering, the authors wrote, drawn from a small slice of the 1.4 million-document Mallinckrodt collection that was not randomly selected. Redgrave’s prompts and the method used to develop them also remain confidential. Those limits matter most for the question underneath every recall figure in both papers, which is whose judgment counts as correct.

How much reviewers disagree

The research paper’s overlap figures show why one expert’s calls serve as a reference point rather than an infallible answer. Measured by the Jaccard index, the expert and the L1 team overlapped at 0.322, the expert and S-CAL at 0.361, and the expert and aiR at 0.275 or 0.421, depending on how aiR’s score categories are read. Earlier studies the authors cite put overlap between professional reviewers at 0.158 to 0.350, and between one expert’s own judgments at different times at 0.602 to 0.806.

Those two aiR readings trace to a choice the Redgrave team made. It counted aiR’s error documents as responsive predictions, calling that a conservative treatment because they were routed to human review. It counted Borderline scores the same way, because aiR saw “at least a plausible basis for responsiveness.” The research paper calls that the liberal reading and says it is the one needed to reach nearly 88 percent recall. A strict reading, limited to aiR’s two highest score categories, is the only other usable cut point, according to the authors.

Two earlier critiques pressed on a part of the Redgrave design the new paper does not discuss. After the primary measurement, Reiss re-examined 151 documents he had coded non-responsive that aiR had flagged, this time with aiR’s rationale visible, and changed 10 to responsive. Greg Buckles of eDiscovery Journal wrote in July that the step has “a circularity problem,” comparing it to a grader shown the reasoning behind the answer key. Neil Cameron, lead analyst at Legal IT Insider, wrote in June that the study cannot show whether the AI’s reasoning corrected the expert or merely persuaded him.

Cost, speed and the price the researchers could not get

Uncertainty about who is right also clouds the question practitioners usually ask first, which is what each approach costs, and the two papers report different parts of that ledger. The Redgrave paper puts aiR’s prompt work at about 18 hours and its processing run across all 45,004 documents at about one hour, against the managed review’s 1,123 hours in its active learning phase. The research paper reports that S-CAL trains and runs in 8 seconds of elapsed time on commodity hardware once documents are vectorized, a step it says takes less than one hour. Those figures mix attorney labor, processing time and computer run time, and none of them is a price.

Price is where the researchers’ analysis stops. Relativity, when asked, would not put numbers on the aiR unit allowance that comes with a subscription or on the charges for exceeding it, the authors wrote, and they said that gap precludes evaluating cost. Relativity’s public help documentation says a relevance analysis in aiR for Review consumes one aiR Unit per document and counts a separate analysis each time a document is run and returns a result. It refers threshold details to users with Relativity Community credentials. Survey data offers only a rough frame. In the Winter 2026 eDiscovery Pricing Survey that ComplexDiscovery OÜ conducted with EDRM, 19 of 53 respondents placed per-document generative AI review pricing between $0.11 and $0.50, a survey result not specific to aiR.

The papers split on richness, too. Redgrave attributed its depressed precision to a 7 percent richness, well below the 20 to 25 percent design target it cited for TAR. The research paper answers that it found neither guidance nor empirical evidence behind either point. If higher richness helps, it adds, the methods would do worse on the full Mallinckrodt collection, which the authors say has lower richness. It also notes that TREC 2009 Legal Track topics, with richness of 0.3 to 1.5 percent, produced precision between 67.3 and 88.4 percent.

Who is speaking, and from where

Each side reads these numbers from a particular vantage point, and both papers come from people with professional positions on the question. The Redgrave study’s prompt engineer and its ground-truth expert are both among its authors, and the two June write-ups that carried its headline result were written or issued by Redgrave. Cormack and Grossman wrote the 2014 evaluation of continuous active learning protocols that the Redgrave paper cites. Google Patents lists both as inventors on U.S. Patent No. 10,671,675, covering a scalable continuous active learning approach, which was assigned in February 2025 to Adaptive Classification Technologies LLC. The research paper does not mention the patent, and the patent record does not show whether either inventor holds a current financial interest in it. Pickens, a managing director at Elevate, co-authored the Sedona Conference TAR 1 Reference Model that the Redgrave paper invokes for its defensibility framework.

Keeling defended the study’s scope in an earlier exchange, before the research paper appeared. “Our study focused on first-pass responsiveness accuracy, where the data showed generative AI can find more responsive documents and miss fewer than active learning,” he said in a statement published by eDiscovery Journal in July. The Redgrave paper notes that the full corpus also went through manual review as part of the broader experiment, with that analysis reserved for future work.

What review teams can take from it

That forthcoming analysis may add another reference point, but practitioners do not need to wait for it. Before relying on any head-to-head figure in a meet-and-confer, confirm that both arms performed the same task, including whether a human reviewer’s coding gated one arm and not the other. Ask whose judgments define ground truth and how closely other assessors agree with that person.

Both papers also point at validation design. The Redgrave paper reports that RAL’s own elusion sample showed 100 percent recall while the blind population sample showed an estimated 64 percent. The gap opened because a reviewer’s wrong non-responsive call drops a document from the production set, and an elusion sample drawn only from the algorithm’s below-cutoff discards cannot see it. The two figures answer different questions, the Redgrave paper says. A protocol that needs the second answer has to reach the documents reviewers coded out, which a random sample of the whole population does.

Cut points deserve the same scrutiny. The research paper argues that aiR offers two usable cut points and “no principled way to choose between them,” while the slope of a gain curve shows where returns diminish. Teams negotiating ESI protocols can ask any provider how a recall target will be chosen and documented, and what each added point of recall costs in review hours and AI usage.

Eighty-eight against 64 traveled fast because it was simple. The same 73 documents now support both a headline that generative AI outperformed TAR and a research finding that it is comparable to the managed review it was measured against. What should a validation protocol require before either number travels again?

Paper Posted: Monday, October 12. arXiv preprint server. 2610.16925. It reanalyzes Redgrave LLP Working Paper 2026-01 (June 2026), which reported 88 percent recall for aiR against 64 percent.

Read the original article here.

News sources

About ComplexDiscovery OÜ

ComplexDiscovery OÜ is an independent digital publication and research organization based in Tallinn, Estonia. ComplexDiscovery covers cybersecurity, data privacy, regulatory compliance, and eDiscovery, with reporting that connects legal and business technology developments—including high-growth startup trends—to international business, policy, and global security dynamics. Focusing on technology and risk issues shaped by cross-border regulation and geopolitical complexity, ComplexDiscovery delivers editorial coverage, original analysis, and curated briefings for a global audience of legal, compliance, security, and technology professionals. Learn more at ComplexDiscovery.com.

Source: ComplexDiscovery OÜ
Assisted by GAI and LLM Technologies per EDRM’s GAI and LLM Policy.

Author

  • Rob Robinson is a technology marketer who has held senior leadership positions with multiple top-tier data and legal technology providers. He writes frequently on technology and marketing topics and publish regularly on ComplexDiscovery.com of which he is the Founder and Managing Director.

    View all posts Founder and Managing Director, ComplexDiscovery OÜ