[Editor’s Note: EDRM is grateful to the project team and Duke Bolch Judicial Institute for producing this point in time guidance in 2019 focused on TAR 1.0. The EDRM Analytics and Machine Learning Project is currently working on updated guidance for more advanced protocols and research, including CAL. To join a subteam on this project, email us at info@EDRM.NET. 1 Sep 2021 update]
Guidelines aim to define and demystify technology-assisted review and to encourage widespread adoption of TAR processes
EDRM has released a comprehensive set of guidelines that aim to objectively define and explain technology-assisted review for members of the judiciary and the legal profession.
The goal, editors say, is to provide a wide audience with an authoritative and clear understanding of the reasons to use technology-assisted review, known as TAR, in litigation, and the processes by which TAR works.
A PDF of the full document is available here.
“We wanted to address the growing confusion about TAR, particularly marketing claims and counterclaims that undercut the benefits of various versions of TAR software,” said John Rabiej, deputy director of the Bolch Judicial Institute of Duke Law School, which oversees EDRM. “These guidelines provide guidance to all users of TAR and apply across the different variations of TAR. We avoided taking a position on which variation of TAR is more effective, because that very much depends on facts specific to each case. Instead, our goal was to create a definitive document that could explain what TAR is and how it is used, to help demystify it and to help encourage more widespread adoption.” (Read a TAR Q&A with Rabiej here.)
More than 50 volunteer judges, practitioners, and e-discovery experts contributed to the drafting process over a two-year period. Three drafting teams worked on various iterations of the document, led by Matt Poplawski of Winston & Strawn, Mike Quartararo of eDPM Advisory Services, and Adam Strayer of Paul, Weiss, Rifkind, Wharton & Garrison. Tim Opsitnick of TCDI and U.S. Magistrate Judge James Francis IV (Southern District of New York, Ret.), assisted in editing the document and incorporating comments from a public review period.
“This is a consensus document and by no means represents the views of everyone who participated in its development,” notes Jim Waldron, director of EDRM. “We made many compromises in drafting the guidelines, and many who participated in the drafting effort held passionate and opposing views. But we worked throughout to be thorough, objective, and clear, so that the final product would be useful to as many people as possible. We are tremendously grateful to all who assisted in the drafting, who submitted comments, and who provided new and differing perspectives. This was an incredible effort on the part of a lot of people.”
The guidelines define TAR as “a review process in which humans work with software to train it to identify relevant documents.” The process consists of several steps, including collection and analysis of documents, training the computer using software, quality control and testing, and validation. The guidelines say TAR is an alternative to the manual review of all documents in a discovery process that is at least as accurate and far more time- and cost-efficient than manual review.
The guidelines note that most prominent law firms and government agencies use TAR, that judges have deemed TAR to be a reasonable and acceptable method of review, and that no reported case decision has found TAR to be invalid. “But in order for TAR to be more widely used and accepted in discovery,” the document says, “the bench and bar must become more familiar with it, and certain standards of validity and reliability should be considered to ensure its accuracy. These guidelines will not only demonstrate the validity and reliability of TAR but will also demystify the process.”
The 50-page document contains four chapters: The first chapter defines technology assisted review and the TAR process. The second chapter lays out a standard workflow for the TAR process. The third chapter examines alternative tasks for applying TAR, including prioritization, categorization, privilege review, and quality and quantity control. Chapter four discusses factors to consider when deciding whether to use TAR, such as document set, cost, timing, and jurisdiction.
“Judges generally lack the technical expertise to feel comfortable adjudicating disputes involving sophisticated search methodologies. I know I did,” said Magistrate Judge Francis, who assisted in editing the document. “These guidelines are intended, in part, to provide judges with sufficient information to ask the right questions about TAR. When judges are equipped with at least this fundamental knowledge, counsel and their clients will be more willing to use newer, more efficient technologies, recognizing that they run less risk of being caught up in a discovery quagmire because a judge just doesn’t understand TAR. This, in turn, will further the goals of Rule 1 of the Federal Rules of Civil Procedure: to secure the just, speedy, and inexpensive determination of litigation.”
Rabiej said the TAR Guidelines represent the first step in a multifaceted effort to develop a broad understanding of TAR and to encourage its adoption. Under the auspices of the Bolch Judicial Institute and EDRM, a second document, a protocol for when and under what circumstances TAR should be used, is currently being developed by a drafting team of 40 judges, lawyers, and e-discovery experts who attended a 2017 conference focused on TAR best practices, hosted by Duke and EDRM.
In addition, the Bolch Judicial Institute at Duke will host an invitation-only conference June 20–21, 2019, in Washington, D.C. to evaluate the 2015 amendments to the Federal Rules of Civil Procedure that emphasized proportionality in discovery. The conference aims to explore how successful the bench and bar have been in applying the Rule 26 proportionality factors and the role that technology has or can play; TAR proponents say the technology is an effective means of reducing discovery costs and ensuring proportionality.
And finally, EDRM will host a conference later this year focused on developing a consensus document to advocate the use of machine-learning methods over traditional discovery means. The goal is to create a document that can be endorsed by data scientists, corporate officials, plaintiff and defendant practitioners, and discovery service and software providers to encourage broad adoption of TAR or equally effective technological methods.
“We want to explore whether it makes sense to expect that in cases involving large volume and text-laden document discovery, machine learning or equally effective technology will be presumptively used for discovery unless an articulable justification is presented,” Rabiej said. “The traditional deference to the producing party’s choice of discovery means, no matter the cost, should be reexamined in light of the court’s responsibility to independently assess proportionality and society’s call to lower litigation costs. It is an important access-to-justice issue.”
