AI Analysis of the Top Five Cases in 2022 Shows Major Flaws in Use of Open AI’s GPT Software for Legal Research and Analysis

[Editor’s Note: This post was originally published on Ralph Losey’s e-Discovery Team® blog 12/29/2022, reposted with permission and our thanks.]

As mentioned in my blog on the top 2021 cases earlier this week, my AI helper, Open AI’s GPT-3, made some unexpected selections. See the disclaimer and more extended discussion of this caveat in my last blog, Surprise Top Five e-Discovery Cases of 2021. When I later asked it to identify the top five cases for 2022, the AI performed very poorly and made me look bad. I published the AI’s report last night after cleaning up the words only, and not actually reading each case (it cited to Westlaw and I have Lexis and, good grief, its a Holiday and I’m only human). I gave the robot too much slack.

An astute reader, Maura, noticed right away that the alleged Top Five 2022 cases cited were not all in 2022. You could easily tell that from the Westlaw citations the AI gave. I checked further this morning, as I should have done last night. I then realized that the AI “emperor had no clothes,” the AI was totally off and I had been fooled. As soon as I received Maura’s helpful comment, I took my blog offline. In fact, most of the cases were much older than 2022, and a few appeared to have been made up out of whole cloth. Stupid me for not noticing before. I blame a concussion, and my over-trust of the new, much hyped GPT 3.5. I will be more skeptical henceforth. When it comes to legal research Text-Davinci-003, the Open AI tool that I used, is definitely not ready for primetime.

I did not use Chat GTP 3.5 for any of these queries because that database ends in 2021. Although I am not completely certain about that, and if I did, that would explain some of the errors. Instead, I believe I used the Open AI playground found at https://beta.openai.com/playground. It’s data is supposedly up to date. Guess my first error was to not pay enough attention to the beta part of the hyperlink. To be more specific, I used what Open AI calls the Text-Davinci-003. In defense of OpenAI, the Chat GTP, where the database ends in 2021, is supposed to be a big improvement over Text-Davinci-003. If you do not care about 2022 cases, and apparently few do, you could used Chat GPT 3.5.

The introduction the AI wrote for the first draft of the Top Five 2022 article that I imprudently published last night looked pretty good:

Introduction. As technology continues to evolve, so must the legal landscape. As my regular readers all well know, in the United States, electronic discovery is an increasingly important field of law that governs methods and processes for obtaining, preserving, and exchanging electronically stored information (ESI) in litigation. As such, it is critical that lawyers stay informed on recent legal opinions concerning e-Discovery. In this article we will discuss the five most important United States district court judge or magistrate opinions on electronic discovery in 2022 and explain why each decision is important.

But really, it doesn’t say much, does it?

Then, the day before yesterday, the AI picked the following cases as the top five for 2022. Note the names and dates:

1. Toth v. Johnson & Johnson, 1:20-cv-00591, 2020 WL 5444439 (C.D. Cal. Sept. 11, 2020). (Note: The AI claims the opinion was written by Judge Banikian. I can find no such judge!)
2. Google LLC v. Oracle America Inc., 5:20-cv-03624, 2021 U.S. Dist. LEXIS 66448 (N.D. Cal. Apr. 9, 2021)
3. Schulman v. Smith & Wesson Corp., 0:20-cv-60780, 2020 U.S. Dist. LEXIS 148034 (S.D. Fla. Aug. 14, 2020)
4. Oracle America Inc. v. Google LLC, 786 F.3d 1179 (N.D. Cal. 2014)
5. Microsoft Corporation v. Quinnipiac University, 3:20-cv-01705, 2020 WL 6079805 (N.D. Cal. Oct. 14, 2020)

This is the list that Maura questioned. None of these case is even a 2022 case by the AI’s own dubious report. WTF!

No doubt the AI will blame human error in operation. So I asked it again this morning, December 30, 2022. Anyway, the query I input was: Please identify the top five most important legal cases in 2022 on electronic discovery. Here is my response this time, and note at least the dates appear right.

1. In re: Google LLC v. Oracle America, Inc., No. 18-956 (U.S. Supreme Court 2022).
2. Microsoft Corp. v. United States, No. 1:16-cv-02985 (D.D.C. 2022).
3. U.S. v. Microsoft, No. 1:17-cr-0019 (E.D.N.Y. 2022).
4. In re: Apple Inc., No. 5:19-cv-05847 (N.D. Cal. 2022).
5. In re: Facebook, Inc., No. 5:19-cv-07189 (N.D. Cal. 2022).

Well, this looks a lot better, so I asked the AI the exact same question again to see if I would get the same answer. I didn’t. Here is what the AI came up with this time. It added a little explanation for the cases too. It did not do that the first time. I have no clear idea why. I asked the exact same question.

1. United States v. O’Keefe (11th Circuit) – This case involves the legality of the government’s use of an electronic discovery technology to access data from a suspect’s computer, as well as the scope of the Fourth Amendment’s protection against unreasonable searches and seizures.
2. In re Google (California Supreme Court) – This case considers whether Google must comply with a discovery request for emails stored on its servers.
3. EEOC v. Amazon.com, Inc. (7th Circuit) – This case examines the scope of an employer’s obligation to preserve electronically stored information and the admissibility of such evidence in a case alleging discrimination.
4. Doe v. Facebook, Inc. (9th Circuit) – This case considers the scope of an individual’s right to privacy with respect to their online content, including emails, photos, and other communications stored on Facebook.
5. National Labor Relations Board v. Apple, Inc. (2nd Circuit) – This case examines the scope of the National Labor Relations Board’s investigative powers to access and review electronically stored information held by employers in the course of an investigation.

This actually looks like a better list. But I am very wary now. The AI did not include any kind of citation this time, unlike the past time with the same question. Again, I have no idea why. So I asked the follow-up query, Please provide the full legal citation for these five cases. Here is the AI response.

1. United States v. O’Keefe, No. 19-14883 (11th Cir. 2022).
2. In re Google, No. S251764 (Cal. 2022).
3. EEOC v. Amazon.com, Inc., No. 19-2180 (7th Cir. 2022).
4. Doe v. Facebook, Inc., No. 19-15645 (9th Cir. 2022).
5. National Labor Relations Board v. Apple, Inc., No. 19-1718 (2nd Cir. 2022).

Ok, better, but human Ralph is still skeptical, so I asked a third time, Please identify the top five most important legal cases in 2022 on electronic discovery and provide full legal citations. Here is the robot answer this time.

1. In re: Intuit Inc. Consumer Data Security Litigation, No. 5:20-cv-07982-EJD (N.D. Cal. 2022)
2. National Bank of California v. KPMG, LLP, No. BC716987 (Cal. Super. Ct. 2022)
3. In re: Google LLC Consumer Privacy Litigation, No. 2:20-cv-07837-JAK (C.D. Cal. 2022)
4. In re: Apple Inc. iPhone Privacy Litigation, No. 5:20-cv-07999-EJD (N.D. Cal. 2022)
5. In re: Facebook, Inc. User Data Security Litigation, No. 5:20-cv-08012-EJD (N.D. Cal. 2022)

You know the old expression, about consistency being “the hobgoblin of little minds.” So I asked again, a fourth time, although it is getting somewhat boring now. This time my query was: Please identify the top five most important legal cases in 2022 on electronic discovery and provide full legal citations and a brief explanation of why each case is important. Here is the result:

1. Doe v. Google LLC, No. BC641932 (Cal. Super. Ct. 2022): This case will be important because it will address the scope of Google’s duty to preserve and produce electronically stored information (ESI) under California law. The court’s decision on this issue will provide guidance on the legal expectations for ESI preservation and production across the country.

2. United States v. Apple Inc., No. 1:22-cv-04106 (N.D. Cal. 2022): This case will be important because it will determine the scope of Apple’s legal obligations when it comes to producing ESI in response to a federal criminal investigation. This case has the potential to set a precedent for how technology companies handle ESI in criminal proceedings.

3. In re Facebook, Inc., No. C-22-03763 (N.D. Cal. 2022): This case will be important because it will determine the scope of Facebook’s legal obligations when it comes to producing ESI in response to a civil discovery motion. This case has the potential to set a precedent for how technology companies handle ESI in civil proceedings.

4. Microsoft Corp. v. Twitter, Inc., No. 2:22-cv-0134 (W.D. Wash. 2022): This case will be important because it will determine whether Twitter must produce ESI in response to a civil discovery motion. The court’s decision on this issue will provide guidance on the legal expectations for ESI production in both state and federal civil proceedings.

5. In re Comcast Corp., No. C-22-04341 (E.D. Pa. 2022): This case will be important because it will determine the scope of Comcast’s legal obligations when it comes to producing ESI in response to a civil discovery motion. This case has the potential to set a precedent for how technology companies handle ESI in civil proceedings.

Conclusion

Maybe AI’s Text-Davinci-003 remembers its prior answers and comes up with a different list each time on purpose. Maybe that is a good way to create a top 20. I have many questions. I will keep working with it, when I get time, and let you know. GPT software is very interesting and shows a lot of potential. But be warned, it should not be used at this time without a great deal of skepticism and close human supervision. Hybrid, human and computer, working together – IMO that is the answer to for the proper use of AI for the foreseeable future. I am still very pro AI, but for now at least, the human needs to keep a close eye on the robot.

Ralph Losey Copyright 2022 – ALL RIGHTS RESERVED