[EDRM Editor’s Note: This article was first published November 19, 2024, and EDRM is grateful to Dr. Tristan Jenkinson for permission to republish. The opinions and positions are those of the author.]
Introduction
In part one of this series, I discussed a section within Practice Direction 57AD which could potentially be used to allow for the use of Generative AI for the conduct of disclosure, assuming that the relevant workflow can be demonstrated to be reliable, efficient and cost-effective. I also discussed each of those areas in detail, raising some of the concerns that could potentially be used as arguments against the use of Generative AI.
In this second article, I want to look at other areas in eDiscovery where Generative AI could potentially be used, and also look at the current state of Generative AI in the Courts of England and Wales.
Other Use Cases
At the start of part one, we discussed the “obvious” use case where Generative AI would be used as a method to determine likely relevancy for disclosure on discovery matters. That said there are plenty of other ways in which Generative AI could be used on eDiscovery cases, which do not directly tie to the disclosure of documents. Therefore these other uses may not be covered by the practice directions.
It is also important to appreciate that as an industry, we are still learning about what could potentially be accomplished with Generative AI. It would not be surprising if new use cases are discovered in the future that have not yet been considered.
As an industry, we are still learning about what could potentially be accomplished with Generative AI. It would not be surprising if new use cases are discovered in the future that have not yet been considered.
Dr. Tristan Jenkinson.
A point that I have made regularly is that (as we have seen historically with the use of predictive coding) using Generative AI to assist directly with disclosure would need statistical verification to support the findings. For this reason, I think that we can expect to see the usage of Generative AI elsewhere in the disclosure process before we see it regularly used to assist directly in decisions on disclosure.
Areas that I have previously said were likely to see Generative AI usage in the short term would be ECA and investigation type cases. Not to make decisions on disclosure, but to try and get a fast-track view on what relevant information the dataset may contain, to decide quickly if there is likely to be evidence to support a specific case, or even to find out if allegations against you or your client may be true. We are now starting to see real world implementations where Generative AI is being used for this purpose. Care still needs to be taken as the success of such approaches will be dependent on their cost-effectiveness, which may not always be easy to assess ahead of time.
Similarly, Generative AI could be used to analyse incoming disclosure from opposing parties. Some eDiscovery teams have previously used predictive coding models based on their clients’ review exercise to identify documents of particular importance within incoming disclosure. This seems another natural extension – using Generative AI to look at incoming disclosure as an alternative to predictive coding methods, or as part of an augmented approach.
There are also other uses of Generative AI that could be applied to incoming disclosure. For example, analysis of the produced data for potential issues. Seeking to automatically identify issues such as emails where attachments have not been provided, potential gaps in the loadfile, missing metadata, or flagging other inconsistencies within the data set.
On the flip side, Generative AI could also be used to assist in QC steps on outgoing disclosure, looking for similar problems to flag potential issues so that they can be resolved.
There are also many ways that Generative AI could be used parallel to disclosure cases. For example in deposition preparation, or even using the technology on case correspondence to assist in finding information, summarizing issues or creating a chronology. It is even possible that such technology could assist with preparing disclosure statements, or summarising information that can be used in the Disclosure Review Document or Electronic Documents Questionnaire – though obviously care must be taken to ensure that information is correct.
A separate potential use of Generative AI would be to run mock cases, or use information to try and predict how a case may be decided. I recall attending a Legal Futures conference back in 2019 at which Solomonic were talking about exactly this sort of analysis. It seems that Generative AI would be a good fit for this type of endeavour. I have seen recent commentary from Rob Johnson talking about this point, looking at how cases will likely play out to assist in working out which cases a client may want to settle early. Other approaches could be to analyse your case documents and initial disclosure and see if Generative AI can identify potential approaches that the other side may take to try and respond to your claims.
To see a basic (and quite fun) example of how a mock trial approach could work, it is definitely worth taking a look at an experiment run by Douwe Groenvelt, Deputy Legal Counsel at ASML. Douwe built a simple setup using three phones running ChatGPT (GPT 4o), looking at the expulsion of Joost Klein from Eurovision. The instances of GPT were given official statements and each was prompted to act in a certain capacity. You can find more details and a video of the outcome in this LinkedIn post.
Generative AI in the Courts
As a final note I thought that it would be interesting to consider some views of Generative AI within the courts.
Haber v HMRC
The first example highlights the potential dangers of ChatGPT and hallucinations regarding case law. Reminiscent of the Mata v Avianca case in the US which I have discussed many times in the past, the UK has also had cases where Generative AI has been used. One such case is Haber v HRMC (https://www.bailii.org/uk/cases/UKFTT/TC/2023/TC09010.html). This is a tax case in which Haber acted as a litigant in person (i.e. representing themselves in court). It appears that a friend assisted Haber with putting their case together using ChatGPT. Five examples of case law were documented to support Haber’s case… unfortunately all five were fictitious.
Simmons and Simmons provided a good summary of that case here https://www.simmons-simmons.com/en/publications/clq2gkar900fcu2ewb904uhrj/inappropriate-use-of-chatgpt-exposed-in-tax-case.
Oakley v ICO (ChatGPT is not an expert)
This matter involved a freedom of information request made by Trevor Oakley to the Department for Work and Pensions. The relevant decision can be found here (https://www.bailii.org/cgi-bin/format.cgi?doc=/uk/cases/UKFTT/GRC/2024/315.html).
The freedom of information request asserts that staff employed by DWP engage in criminal misconduct. It asks for information about DWP policies on visitors filming “for the purpose of evidential collection” as well as policies on the use of force and other actions which could be used to prevent visitors from collecting evidence. The request also asks what policies DWP has regarding perverting the course of justice and witness harassment.
Oakley claimed that the response from DWP was incomplete, in part because the electronic searches performed were inappropriate “as they focussed on filming only and not evidence collection”. Oakley sought to back up this claim with information from ChatGPT.
Oakley stated that he gave the wording of the request to ChatGPT and asked it to generate the top ten keywords that should be used to search a database to meet the requirements. Oakley then used differences between the terms used by DWP and those generated by ChatGPT to suggest that the DWP search was inappropriate.
The decision compares the information generated by ChatGPT to expert evidence:
“If comparisons are drawn to expert evidence, an expert would be required to explain their expertise, the sources that they rely upon and the methodology that they applied before weight was given to such expert evidence”
Oakley v ICO [2024] UKFTT 315 (GRC).
The conclusion was that, perhaps unsurprisingly, ChatGPT is not an expert.
“We place little weight upon that evidence because there is no evidence before us as to the sources the AI tool considers when finalising its response nor is the methodology used by the AI tool explained”
Oakley v ICO [2024] UKFTT 315 (GRC).
Oakley’s appeal was dismissed. This note highlighting the black box nature of Generative AI systems such as ChatGPT, could be something that we see frequently referred to by the courts in future, where Generative AI has been used.
Lord Justice Birss
That is not to say that all usage of Generative AI in the courts has been viewed negatively.
It is notable that Lord Justice Birss states that he has used ChatGPT, and in particular has included content from ChatGPT in one of his judgments, based on reporting from the Law Society Gazette (https://www.lawgazette.co.uk/news/solicitor-condemns-judges-for-staying-silent-on-woeful-reforms/5117228.article). Lord Justice Birss is quoted as saying:
“I thought I would try it. I asked ChatGPT can you give me a summary of this area of law, and it gave me a paragraph. I know what the answer is because I was about to write a paragraph that said that, but it did it for me and I put it in my judgment. It’s there and it’s jolly useful.”
Lord Justice Birss, quoted in The Law Society Gazette, Solicitor condemns judges for staying silent on woeful reforms.
Before concluding:
“I’m taking full personal responsibility for what I put in my judgment, I am not trying to give the responsibility to somebody else. All it did was a task which I was about to do and which I knew the answer and could recognise an answer as being acceptable.”
Lord Justice Birss, quoted in The Law Society Gazette, Solicitor condemns judges for staying silent on woeful reforms.
In a speech on Future Visions of Justice Lord Justice Birss talks about the potential for justice to be “dispensed by algorithm”. He refers to the Online Civil Money Claims system where transparency was added to the legal process as part of automating parts of the claims process. On this topic he concludes by linking the use of automation of a claims process by applying an algorithm to the potential use of Large Language Models and machine learning, stating:
“Now this is about a simple mathematical algorithm and in that sense is much less complex than the algorithms used in Large Language Models and machine learning. Nevertheless it is unquestionably an algorithm applied by a machine. As far as I know this is caused no difficulty of any sort and attracted very little comment.”
Lord Justice Birss, Future Visions of Justice (Speech, King’s College London Law School, 18 March 2024).
Lord Justice Birss goes on to talk about potential uses of ChatGPT, and in particular how it could be useful in making legal advice and assistance more available:
“One of the ways in which Large Language Models have the potential to transform access to justice is in their ability to give legal advice and assistance either for free, or at very little cost in circumstances which represent a significant improvement on the current state of affairs.”
Lord Justice Birss, Future Visions of Justice (Speech, King’s College London Law School, 18 March 2024).
This approach certainly has its merits, and the focus here is on litigants in person:
“The problem for unrepresented litigants is that they don’t know what the right answer is and so they need a different kind of help, such as may be provided by a tool of this kind… it does seem to me that they indicate that Large Language Models are worth investigating as they may have a role to play in the provision of advice and assistance for those with limited means. Simply improving the quality of what is available for free, even if it does not make it perfect, is a significant step forward.”
Lord Justice Birss, Future Visions of Justice (Speech, King’s College London Law School, 18 March 2024).
I would agree that it is certainly worth exploring if Generative AI has a reliable future in this sphere. However, the case of Haber, discussed above, demonstrates some of the difficulties that remain with this approach. Ultimately the issue is exactly as Lord Justice Birss identifies – litigants in person are not (typically) able to tell if the information that Generative AI is giving to them is correct or accurate. With regard to case law, litigants in person would also likely not have access to the relevant systems and journals that lawyers would access to check that case references etc. are correct.
This means that Generative AI could be something of a double edged sword for litigants in person. It can assist and provide information to allow litigants in person to pursue justice. However, the litigant in person is not in a position to know if the information being provided is correct. This could lead to cases such as Haber, where case references and arguments given have bee hallucinated, or worse, a litigant in person could be persuaded to take a case to court based on incorrect information, when they stood no realistic hope of winning the case.
Master of the Rolls – Sir Geoffrey Vos
Sir Geoffrey Vos is no stranger to the use of technology, and AI in particular. He recently addressed potential issues with AI and GDPR in a speech to the Irish Law Society, in which he also addressed the potential copyright issues with AI systems trained on large data sets from the internet (a point which I have also discussed in the past).
In his address to the Law Society of Scotland, the Master of the Rolls discusses the Mata v Avianca case in the US, and also raises the potential risks of usage in particular by litigants in person, raising a different case to Haber (covered above) which happened in Manchester (covered by the Law Society Gazette here).
Vos also see’s the potential benefits that could come from such technology, stating:
“In my view, we are going to have to develop mechanisms to deal with the use of generative AI within the legal system. We may even hopefully turn it to the advantage of access to justice and effective and economical legal advice and dispute resolution.”
Sir Geoffrey Vos, AI and the GDPR (Speech, Irish Law Society Industry Event, 9 October 2024).
Vos goes on to then discuss the potential of “Robo-judges and Robo-lawyers”.
In an article by Enyo Law (written with the assistance of ChatGPT – https://enyolaw.com/news/chatgpt-and-the-future-of-dispute-resolution-the-benefits-and-risks-of-using-ai/), the use of ChatGPT in the legal field is discussed. Of particular relevance is some of the advice of ChatGPT itself:
“… while ChatGPT has the potential to be a valuable tool for predicting case outcomes, it should be used in conjunction with human judgment and expertise. Ultimately, legal decision-making involves a range of factors beyond just predicting the outcome of a case, including strategic and ethical considerations and client goals”
Dr Irina Durnova, ChatGPT and the Future of Dispute Resolution: The Benefits and Risks of Using AI (Enyo Law, 4 May 2024).
Anthropic and Disclosure to the Court
I saw a recent post from Judge Scott Schlegel, who made a really interesting point regarding whether the use of AI needs to be disclosed to the court if it is used. While Judge Schlegel is based in the US, the point is a valid one regardless of jurisdiction.
“While I’ve argued against courts mandating special orders for AI disclosure in legal work, it turns out disclosure might be required anyway – not by courts, but by the AI providers themselves.”
Judge Scott Schlegel, LinkedIn post.
Judge Schlegel links to Anthropic’s terms of use (which can be found at https://www.anthropic.com/legal/aup), highlighting that those terms identify legal use as “high-risk”, and require disclosure to end users, human review and oversight from a qualified professional.
While the requirements are at current to disclose the use of Anthropic’s AI models to end users, it is possible that internal requirements from the providers of Generative AI systems could indeed lead to the requirement of disclosure of their use in legal cases.
It is interesting to note that Anthropic explicitly state that users must not use their products and services for criminal justice or law enforcement.
Summary
Under the wording of the Practice Directions, if an approach to disclosure is reliable, efficient and cost effective, then the parties should be promoting it. On this basis, if a workflow utilising Generative AI can be shown to have these properties, then it should be a use that the court would approve.
The properties of Reliability, Efficiency and Cost-Effectiveness may not be easy to demonstrate, and workflows could face challenges based on issues that have already been seen and demonstrated with Generative AI systems. That said, there are workflows that are being developed and used which seek to mitigate these challenges. As with predictive coding, it is likely that some parties will agree certain usage in civil matters long before we have case law rulings on permissible uses of Generative AI in the conduct of disclosure.
Senior members of the judiciary are advocating for the use of technology going forward, and have used Generative AI to assist them. The key will be for some formal agreements and documentation on how the technology can (or should) be used.
Read the original article here.
Assisted by GAI and LLM Technologies per EDRM GAI and LLM Policy.
