We have approximately 290,000 emails from and to Jeb Bush during his two terms as Florida’s governor in our discovery collection. These are the emails used during the annual TREC conferences put on by NIST. Program organizers reviewed the collection and came up with about 35 topics to test machine learning algorithms (none of which included LLMs back in those days).
Our assignment today is to find information about this topic:
Summer Olympics — All documents concerning a bid to host the Summer Olympic Games in Florida.
The question is this: How quickly can we get a handle on issues relating to this topic including who is involved, what positions did they take and what were the arguments on both sides.
This is not about finding relevant documents for humans to review. Rather, it is about putting large language models like GPT and Claude to work analyzing documents and answering questions. And, they can do in seconds what would take humans hours or days to accomplish. That is the point of our testing.
For these exercises we will be using B2, our internal system created in the Merlin lab to explore how we might best integrate the analytical power of LLMs like GPT and now Claude, in a traditional discovery platform. Other eDiscovery platforms may or may not be able to integrate one or multiple LLMs into the workflow. Our goal is to identify relevant documents in response to a natural language prompt, submit them to GPT for analysis and then review its answers to our questions.
In this case, B2’s job is to:
- Comprehend the initial prompt and identify relevant documents from the database that could aid in responding to it.
- Distill these documents into a summary that aligns with the informational requirements of the initial prompt.
- Condense this information into a form that can be fed into GPT for response. Depending on the complexity and volume of the information, this may necessitate breaking our efforts into a series of prompts and responses.
- Receive GPT’s response, which should be based on both the reviewed information and the content of the original prompt.
- Record this information, and transmit it back to the user.
B2 allows us to choose which LLM we want to use for which purpose. In this case we are using GPT 3.5 Turbo to summarize documents and Claude 2 for document analysis and synthesis.
This is an ongoing project where we will use B2 to investigate different TREC topics. Our purpose is to illustrate the new capabilities an LLM like GPT can offer for investigations and discovery.
Search engines find relevant candidates for review but stop there. Expensive legal professionals have to take over from there to review, analyze and determine whether the found documents are relevant and then to uncover the story they might tell about the topic. Suffice it to say that the process is tedious, time consuming and expensive.
Can B2 do better? Let’s see. Keep an eye on how well B2 performs its tasks through these exercises. We believe you’ll be impressed with its analytical prowess and capabilities.
We start with a simple natural language question to get our investigation started:
Hello B2. I am looking for information on this topic:
Summer Olympics — All documents concerning a bid to host the Summer Olympic Games in Florida.
Summarize the main topics of discussion regarding a bid to host the Summer Olympic Games in Florida and identify the people involved in the discussions by name, role and email address.
Link to the documents which support your answers.
Here is a summary of the key topics and people involved in discussions regarding a bid to host the Summer Olympic Games in Florida, based on the document summaries provided:
The main topics of discussion regarding a bid to host the Summer Olympics in Florida centered around transportation infrastructure, venue planning, hurricane preparedness, legacy benefits, and competing against other US and international cities.
Specific venues discussed include the Olympic Village, Olympic Stadium, and Aquatic Center in Tampa (Document 086195, Document 085327). High-speed rail between Tampa and Orlando was also a key topic, with involvement from Amtrak and FDOT (Document 075542, Document 083085).
Hurricane preparedness and holding the Olympics during hurricane season was a major concern, given the risks and costs involved (Document 066406, Document 076262). Developing a detailed hurricane evacuation plan was recommended (Document 064277).
The potential legacy benefits of hosting the Olympics, including economic impact, community redevelopment, and statewide/regional recognition and pride were cited as reasons to support the bid (Document 083085, Document 086195).
The competitive nature of the bidding process, both domestically against other US cities and internationally, was discussed. Florida was noted as the strongest US bid at one point, ahead of Texas, Washington, New York, and others (Document 081228).
The main people involved in these discussions were:
–Ed Turanchik, leader of Florida 2012 Olympic bid committee (Document 075542, Document 086195)-
–Governor Jeb Bush (Document 075542, Document 085327)
–Cornelius Cosentino, Chairman of FL2012/16 Citizens Advisory Committee (Document 076262, Document 066406)
–John Sykes, member of Florida 2012 Olympic Committee (Document 085327)
–Tom Barry, transportation planning (Document 084880)
–Russell Williams, involved in initial Olympic bid efforts (Document 050307)
–Tara Wainwright, Governor’s office (Document 083085)
–Sandy MacKinnon (Document 086195)
–Carlos Alfonso (Document 086195)
No direct links to bid documents were provided in the summaries.
Before preparing the above response, B2 used a vector-based search engine to find 30 of the most relevant documents, summarizing each as it related to the topic being investigated. Here are some sample summaries:
|Docid: 066406 |
Email from Cornelius Cosentino to Joe Myers requesting recommendations on a comprehensive Olympic hurricane evacuation plan for a bid to host the Summer Olympic Games in Florida
The document is an email sent on December 3, 2000, from Cornelius Cosentino, Chairman of the Citizens & Business Advisory Committee to the Florida 2012/16 bid, to Joe Myers, regarding the bid to host the Summer Olympic Games in Florida. The email requests recommendations from Myers on a comprehensive Olympic hurricane evacuation plan, as the proposed dates for the event fall within the Florida hurricane season. The email highlights the need for a plan due to the potential risks and costs involved in the event of a hurricane or tropical storm. The email also mentions the state of Florida’s expenditure on a consultant firm to calculate the number of people and vehicles that will be in the Tampa Bay area during the proposed Olympics. The email includes the email address of the sender, firstname.lastname@example.org, and the phone number, 813-251-4669. Other people involved in the discussions mentioned in the email include Dantram, Ed Turanchik, Francis E Cape, and Polly Bryan, whose email addresses are not provided. The email does not provide a link to any supporting documents.
Email from Russell Williams to Governor Jeb Bush in support of bid to host Summer Olympics in Florida
The document is an email sent on May 12, 2000, by Russell Williams to Governor Jeb Bush in support of a bid to host the Summer Olympics in Florida. Williams encourages the governor to ignore negative people and highlights the potential profits for worthwhile projects. He mentions Ed Turanchik and his Olympic Committee as doing a great job of promoting the bid. Williams also mentions his own involvement in making first contact with the National Olympic Committee and finding the non-refundable bid money. He provides his contact information as 115 N. Hubert Avenue Apt. 6 – Tampa, Florida – phone (813) 288-0864 – Email email@example.com.
Email from John Sykes to Governor Jeb Bush expressing concerns about transportation and aquatic center for a bid to host the Summer Olympic Games in Florida
This document is an email from John Sykes to Governor Jeb Bush, sent on July 25, 2001. Sykes expresses concerns about the pending site visit of the USOC and discusses two key issues: transportation and an aquatic center. He mentions a meeting between the Governor and the Florida 2012 Olympic Committee, which he cannot attend due to a conference call. Sykes discusses the constitutional issue of rail and his support for a balanced transportation system. He also mentions legacy issues and the need to address them with the USOC. The email does not provide any documents related to a bid to host the Summer Olympic Games in Florida.
– John Sykes, sender (John.Sykes@sykes.com)
– Governor Jeb Bush, recipient (firstname.lastname@example.org)
Email from Ed Turanchik to Governor Jeb Bush outlining factors important
The document is an email from Ed Turanchik to Governor Jeb Bush, discussing a bid to host the Summer Olympic Games in Florida in 2012. Turanchik outlines the factors that will be important in winning the bid, including Florida’s popularity as a tourist destination, its hemispheric position, the potential for profits, and the quality of the Olympic Games plan. He also discusses the challenges that need to be addressed, such as the need for high-speed rail and the development of the Olympic Village. The email includes the names and email addresses of several other individuals involved in the bid, including Sandy MacKinnon, John Sykes, and Carlos Alfonso. The document was sent on July 14, 2001.
Email from Betsy Couch of VISIT FLORIDA to Scott Openshaw and Katherine Morrison regarding a Summer Study conducted in May 2006 to gauge the current summer travel market
The document is an email from Betsy Couch of VISIT FLORIDA to Scott Openshaw and Katherine Morrison regarding a Summer Study conducted in May 2006 to gauge the current summer travel market. The study was co-sponsored by the Convention and Visitors Bureaus for Miami, Orlando, and Kissimmee. The results of the study will be shared at VISIT FLORIDA’s June Board Meeting on June 15. The email includes a Q&A and message points regarding the study for media calls. The document does not relate to a bid to host the Summer Olympic Games in Florida, and there is no mention of any people involved in discussions regarding such a bid. The email was sent on June 7, 2006.
Letter from Neil Cosentino to Mr. Myers of DCA discussing the concern of holding the bid during the hurricane and tropical storm season
The document is a letter from Neil Cosentino, President of FL2012/16 C-BAC, to Mr. Myers of DCA regarding a bid to host the Summer Olympic Games in Florida. The letter discusses the concern of holding the bid during the hurricane and tropical storm season and questions whether it is necessary to do so. The letter also mentions a “detailed response plan” that DCA requested and asks if it has been reviewed. The main people involved in the discussion are Neil Cosentino and Mr. Myers of DCA. There is no email address provided. The document was written on an unspecified date in 2012. The document does not provide any information on a bid to host the Summer Olympic Games in Florida.
Email exchange discussing bid to host Summer Olympics in Florida, involving Ed Turanchik, Kathleen Shanahan, and Tara Wainwright
Date: May 2, 2001
Main people involved: Kathleen Shanahan, Jeb Bush, Ed Turanchik, Tara Wainwright, email@example.com
Summary: The email exchange discusses a bid to host the Summer Olympic Games in Florida. Ed Turanchik updates Governor Jeb Bush on the bid developments and requests his attendance at a formal reception during the USOC site visit. Turanchik also expresses concerns about transportation between Tampa and Orlando and requests help from FDOT and Fla Highway Patrol. Kathleen Shanahan suggests that being part of the pitch to host the Olympics is worth the time and effort, as it would bring statewide and regional pride and recognition if Florida wins. No link to documents was provided.
B2 provides a link from the summary to the document itself:
You might be wondering how long it took–from the time we entered our topic–for B2 to retrieve the search results, summarize the documents, and produce the above analysis of key issues. We received the first document summary in around 20 seconds, and the full analysis was complete in under 2 minutes.
How much did this cost us in LLM processing fees? About 30 cents. Compare these times and costs with doing it yourself—or paying someone to do it for you.
Now that doesn’t cover the costs of building our platform or developing our expertise. But, well, you get the point here. This is game changing stuff.
Next we will ask B2 for more information about the pros and cons of the issues being discussed.
What were the concerns about a Summer Olympics bid. Describe the concerns, who did they affect and who was concerned.
Based on the document summaries provided, the main concerns expressed regarding a bid to host the Summer Olympics in Florida were:
Hurricane/tropical storm season:
– Holding the Olympics during hurricane season was seen as a major risk due to potential costs, liability, and need for evacuation planning (Documents 066406, 076262, 064277)
– This concern was raised by Cornelius Cosentino of the Citizens Advisory Committee, who recommended changing the bid dates or stipulating no Olympic hurricane bids (Document 076262)
– The safety of visitors, residents, and taxpayers was seen as an issue during storm season (Document 080529)
– Inadequate transportation between venues was a concern, especially rail between Tampa and Orlando (Documents 075542, 083085)
– This was raised as an issue by Ed Turanchik of the bid committee, who sought help from state agencies (Document 083085)
– Meeting requirements for venues like the Olympic Village, Stadium and Aquatic Center was discussed as a key issue (Documents 085327, 086195)
– John Sykes from the bid committee expressed concerns about venue plans to the Governor (Document 085327)
Costs/return on investment:
– The return on taxpayer investments and overall costs were seen as potential issues (Document 080529)
– Citizens Advisory Committee expressed concerns about prudent use of public funds (Document 080529)
The main groups expressing concerns were the Citizens Advisory Committee, led by Cornelius Cosentino, and some members of the bid committee, like John Sykes. The concerns impacted Florida residents, visitors to the state, and taxpayers in general.
We will end this exercise by asking B2 to prepare an investigation report on our topic:
Prepare an investigation report regarding the information you have found. Present it in the following sections:
Section 1: A bullet point listing of the people involved including their positions (if you have them) and their roles in this discussion.
Section 2: A bullet point timeline of the emails you have reviewed by date ascending. Give me a two line summary of each including From and To.
Section 3: Summarize the issues involved and the pros and cons discussed for each.
Section 4: List any rules, regulations or statutes being discussed with a summary of how each relates to this discussion.
Make the report as comprehensive as you can.
Here is the investigation report with the requested sections:
Section 1: Key People Involved
– Ed Turanchik – Leader of Florida 2012 Olympic bid committee
– Governor Jeb Bush
– Cornelius Cosentino – Chairman, FL2012/16 Citizens Advisory Committee
– John Sykes – Member, Florida 2012 Olympic Committee
– Tom Barry – Transportation planning
– Russell Williams – Involved in initial Olympic bid efforts
– Tara Wainwright – Deputy Chief of Staff for Governor Bush
– Sandy MacKinnon – Consultant assisting with Olympics bid
– Carlos Alfonso – Involved with Florida 2012 bid committee
Section 2: Timeline of Discussions
– November 19, 1999: Email from Ed2jen@aol.com to Governor Bush encouraging him to visit Sydney Olympics and support Florida’s bid, stating Florida has superior plans.
– May 12, 2000: Email from Russell Williams to Governor Bush urging support for Olympics bid and praising work of Ed Turanchik’s committee. Williams helped make initial contact with National Olympic Committee.
– May 30, 2000: Email from Busy Shires to Governor Bush suggesting hosting Olympics would benefit Florida horse industry. Proposed equestrian venue at Florida Horse Park.
– March 21, 2001: Email from Ed Turanchik to Governor Bush requesting availability for first USOC site visit between June 1 – September 30. Also invited Governor to triathlon event related to bid.
– May 2, 2001: Email exchange between Kathleen Shanahan, Governor Bush, Ed Turanchik, and Tara Wainwright regarding bid updates. Turanchik asked for Governor’s attendance at USOC reception. Expressed concerns over Tampa-Orlando transportation.
– May 21, 2001: Email from Ed Turanchik to Governor Bush giving update that Florida has strongest US Olympics bid ahead of Texas and Washington. Discussed transportation needs.
– May 28, 2001: Email from Cornelius Cosentino raising concerns about hurricane season and suggesting Spring bid instead.
– July 14, 2001: Email from Ed Turanchik to Governor Bush outlining Olympics plan components including venues, transportation, and housing. Emphasized showcasing Florida’s strengths.
– July 25, 2001: Email from John Sykes to Governor Bush regarding USOC site visit. Discussed transportation and venue concerns.
– July 31, 2001: Email from Tom Barry outlining $3 billion in transportation projects related to Olympics bid across central Florida.
– August 4, 2001: Email from Cornelius Cosentino detailing hurricane concerns including costs, liability, and evacuation needs. Suggested changing dates.
Section 3: Key Issues and Perspectives
– Hurricane season: Significant concern over safety, costs, liability if Olympics held during storm season. Some suggested changing bid dates.
– Transportation: Need for expanded transit, especially between Tampa and Orlando venues seen as issue. State transportation agencies lobbied for help.
– Venues: Meeting specifications for facilities like Olympic Village and Aquatic Center discussed as challenge. Legacy value of new venues noted.
– Costs: Projected large public costs and concerns over return on investment raised as potential downside. But also seen as economic opportunity.
– Legacy benefits: Supporters touted redevelopment potential, state pride, tourist destination promotion as positives of hosting Olympics.
Section 4: Rules and Regulations
No specific statutes, rules or regulations related to the Olympics bid discussed in the provided information.
That’s the end of today’s exercise.
Welcome to a new era of Discovery–one where we can quickly move from search hits to discovery answers with the extraordinary power of artificial intelligence. By seamlessly integrating an LLM with an algorithmic search engine like Sherlock, legal professionals can harness the immense capabilities of large language models to streamline discovery processes and quickly dive deeper into the key documents that are most relevant for the case.
Up until now, keywords were the primary means to find relevant documents. The search engine’s job was to locate potential candidates that might be responsive to your information needs. Once this step was complete, the search engine’s job ended. The onus then fell on you and your team to read, analyze, and interpret the results, a process that was often tedious, time-consuming, and expensive.
Generative AI systems like GPT and Claude can take discovery beyond simple search. They can analyze result sets and use them to answer questions or otherwise provide meaningful information in response to your prompt. For the first time in history, we have at our fingertips a generative AI system that can assist with the second half of the discovery process–ESI review and analysis.
The applications we have explored in this article merely scratch the surface of an LLM’s potential to assist in investigations and discovery efforts. Using an integrated system like B2 to help in synthesizing information, summarizing documents, answering questions and creating investigation reports, will make investigations and discovery more efficient, improving outcomes and saving time and money.