[EDRM Editor’s Note: The opinions and positions are those of Michael D. Berman.]

Every search must be validated. It is “entirely self-evident” that “ipse dixit pronouncements from lawyers unsupported by an affidavit or other showing that the search methodology was effective for its intended purpose are of little value to a trial judge.” Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251, 261 (D. Md. 2008).

“Three-strata validation” was requested and rejected in In re Class Action Settlement Administration Litigation, 2026 WL 2444936 (D.D.C. Aug. 19, 2026). The parties in that case had narrowed their disputes and their “only remaining disagreement concerns how a party producing documents in discovery must validate that its productions are substantially complete.”

Plaintiffs asked for “three-strata validation.” Defendants objected to the second and third strata.

The court explained the “three-strata” approach as: “validation of documents coded as non-responsive, validation of documents coded as responsive, and validation of documents excluded from the universe of ESI material subject to discovery review.”

THE PARTIES AGREED TO NULL SET TESTING

The parties agreed that there is an obligation “to confirm that responsive documents are not erroneously coded as non-responsive and omitted from their productions.” This involves review of a “statistically valid random sample” of documents coded as non-responsive. Id. at *1. 

This standard practice is commonly called “elusion testing,” review of the “discard pile,” or, review of the “null set.” See Deal Genius, LLC v. O2COOL, LLC, 682 F. Supp. 3d 727, 734–35 (N.D. Ill. 2023) (“explaining that elusion testing, where the producing party reviews a random sample from the ‘null set’ of documents that did not hit on search terms and documents that were deemed non-responsive, is a ‘standard quality assurance practice’ that can ‘confirm the reasonableness and proportionality’ of the production”).

VALIDATION OF DOCUMENTS CODED AS RESPONSIVE – REJECTED

Defendants argued that validation of documents coded as responsive, to find false positives, was superfluous. They asserted that Fed.R.Civ.P. 26(g) required that they certify that their response is complete and correct. The court wrote: “Rephrased in statistical terms, producing parties must confirm that their false-negative rate falls within acceptable limits.” In my less elegant words, a production may include nonresponsive documents; but, it cannot be a “dump truck.”

The court explained:

Review of documents marked responsive falls outside the bounds of litigants’ Rule 26(g) obligations because such review cannot identify whether documents are missing from a production, rendering it incomplete. Confirming that documents designated responsive are truly responsive reveals only the production’s false-positive rate, a proxy measure for whether the litigant has overproduced discovery material. But the Federal Rules do not prohibit parties from overproducing documents. Accordingly, this Court will not require defendants to incur additional costs re-reviewing documents that they have previously coded as responsive and intend to produce, barring claims of privilege. [emphasis in original].

VALIDATION OF DOCUMENTS EXCLUDED FROM THE UNIVERSE – REJECTED

The second strata consists of “documents excluded from the universe of ESI material subject to discovery review.” In other words, Plaintiffs asked for a review of documents that Defendants had not collected and that, presumably, were not in Defendants’ litigation review platform.

The court wrote that examination of this set of documents, the “Excluded Set,” “could reveal false-negatives: responsive materials that should be produced but were erroneously omitted from the universe of materials collected for discovery review.”

However, the court responded that Plaintiffs’ request was duplicative and made too late, long after collection. It wrote: “Yet the Court is puzzled why plaintiffs request review of the Excluded Set after discovery review is largely complete, given that the parties also intend to review the Excluded Set for responsive material prior to beginning that review.” [emphasis in original].

The court explained that “ESI discovery generally proceeds in two phases: document collection and merits review.” The parties had earlier “set forth agreed upon parameters for collecting the universe of documents that will be subject to merits review.” That earlier agreement had required statistical sampling.

Insofar as the parties have agreed to statistically validate the completeness of the set of documents collected for merits review before even beginning that review, the Court finds that further validation of the Excluded Set after merits review is largely complete would be duplicative and unnecessary.

However, Plaintiffs made a different argument—so-called “substantial completeness evaluation.” They argued that:

[T]heir proposed validation efforts “answer different questions at different points in time,” because search term validation “tests whether agreed terms are performing as intended before review begins,” while substantial completeness validation of the Excluded Set “tests whether the entire process — including all downstream coding decisions, review workflow, and production choices — resulted in a complete production at its conclusion.”

This argument was unsuccessful because the court held that “the Excluded Set is a collection of documents excluded before merits review begins and not subject to downstream coding decisions, review workflows, and production choices.”

Plaintiffs also contended that “the prevailing practice is for parties producing discovery to validate the Excluded Set both upon its creation and after merits review….” The court noted that the authorities are split; however, it held that “the Court will not require producing parties to incur the cost and burden to re-validate the Excluded Set at the substantial completeness stage, provided that they statistically validate that documents excluded from merits review do not contain responsive material before beginning that review.”

I would add that “substantial completeness validation” of the Excluded Set could de-rail a case and mandate a “do over.” If a sufficient number of false negatives was found in that set—and sufficiency is always debatable–resumably Defendants would have to engage in a new collection, review, and production process.

Assisted by GAI and LLM Technologies per EDRM’s GAI and LLM Policy.