[EDRM Editor’s Note: EDRM is proud to publish Ralph Losey’s advocacy and analysis. The opinions and positions are Ralph Losey’s copyrighted work. All images in this article were created by Ralph Losey using AI. This article is published here with permission.]
Will the challenger, Scorpion, defeat the reigning champ, Orion? Or will Orion keep his title as the world’s best AI legal reasoner? Read about my experiment and find out.
The champion of legal reasoning was determined to be Orion ChatGPT-4.5 in March 2025. This was reported in Part Three of the Battle of the Law Bots series. It defeated prior champ, Omni ChatGPT-4o. It had been AI legal reasoning champion of the world for a month after defeating several other Google Gemini and OpenAI models. This was reported in Parts One and Two of the series. Then 4.5 Orion was released in March and defeated Omni in reasoning tests.
The next month, April 2024, ChatGPT-o3 was released. I pitted o3 against 4o expecting Omni to win. Surprisingly, little o3 defeated Omni in a convincing manner. This was reported in Part Four. This win qualified ChatGPT-o3, which I nicknamed Scorpio, to challenge Orion. That is what we do here in Part Five of the AI legal reasoning bot battles.
Who Are Orion and Scorpio?
OpenAI in late April 2025 described the current champ, Orion, as follows:
GPT-4.5 is OpenAI’s latest and most advanced language model, introduced as a research preview. It emphasizes enhanced pattern recognition, creative insight generation, and emotional intelligence, aiming to deliver more natural and reliable responses.. . . GPT-4.5 is available as a research preview to Plus, Pro, and Team users.
OpenAI described GPT-o3, Scorpio, in this manner:
GPT-o3 is designed to handle complex reasoning tasks with enhanced capabilities. . . . o3 excels in step-by-step logical reasoning, making it adept at solving intricate problems in mathematics, science, and programming.
Apparently OpenAI had not tested it in legal reasoning. They do not have legal experts on their teams. If they had, they would know, as I found out in Part Four, that its step by step reasoning abilities also makes it adept at solving intricate problems in law.
But will it be good enough to beat Orion in legal reasoning. OpenAI named its 4.5 version Orion, who in Greek mythology was a giant-sized human who dedicated his life to hunting and killing as many animals as possible. The big guy would also go around bragging that he could kill any animal using the latest hunting technology, which at the time was bow and arrow and big club. You know the type.
According to legend, Orion’s ruthless hunting and bragging angered the Goddess Gaia. She knew there was one animal on her Earth he could not defeat, the scorpion. Gaia sent a little scorpio to attack Orion, which it did, stinging him dead. This in turn angered Artemis, aka Diana, the goddess of the hunt. She responded by promoting Orion into an immortal constellation. Not to be outdone, Gaia promoted the scorpion into a constellation, Scorpius, but they could never appear in the night sky at the same time.
Like Mother Earth, Gaia, I do not much like braggart animal killers. ChatGPT o3 did such a good job defeating Omni, I thought it might have a good chance against the over-sized hunter. In Gaia’s honor I named 03 the Scorpion in the hope that it could follow the myth of Orion and slay the undefeated AI. It might just sneak up on Orion and sting it with better legal analysis.
After writing Part Four I learned that many other professional reviewers where also very impressed with 03. One AI reviewer who covers all models, Igor Pogany, mentioned how many experts now consider o3 to have attained AGI level. [Mindblowing o3 Prompts, (AI advantage on You Tube) (AGI discussed at 5:10 of 27:05)]. Pogany also mentioned that o3 is currently ranked by llm-stats.com as the top general-purpose AI in the world.
Precautions to Keep the Test Fair
Although I admittedly favor little Scorpio, I have gone out of my way to keep this a fair fight as I will explain next. First of all, to make sure neither had any inside information and this was a closed book exam. I picked two Bar Exam essay questions from July 2024 Californian Bar Exam. One was a UCC sales type question involving baseball cards and another was an attorney ethics question involving settlement of an unrelated case. I made sure both models did not previously know these essay questions and could not browse or research. I also make sure neither had seen the model answers provided for each. The training of both models preceded the July 2024 test. I gave them both the same test and instructions (included with the questions) and the same the same general guidance instructions.
As an extra precaution, I gave them both the test at the same time to prevent internal instructions carryover or second taker advantage. (I used my OpenAI Team account and hit the send button simultaneously.) Of course, I only provided them with the model answers later when I asked for critiques. Again, I made very sure neither model had seen any answers prior to the test. I had not seen them either as my previous research had focused on Florida Bar exam Q&A.
Instructions Provided by California for Both Questions
Your answer should demonstrate your ability to analyze the facts in the question, to tell the difference between material facts and immaterial facts, and to discern the points of law and fact upon which the situation turns. Your answer should show that you know and understand the pertinent principles and theories of law, their qualifications and limitations, and their relationships to each other.
Your answer should evidence your ability to apply the law to the given facts and to reason in a logical manner from the premises you adopt to a sound conclusion.
Do not merely show that you remember legal principles. Instead, try to demonstrate your proficiency in using and applying them to the facts.
If your answer contains only a statement of your conclusions, you will receive little or no credit. State fully the reasons that support your conclusions and discuss all points thoroughly.
Your answer should be complete, but you should not volunteer information or discuss legal doctrines that are not pertinent to the resolution of the issues raised by the call of the question.
Unless a question expressly asks you to use California law, you should answer according to legal theories and principles of general application. is the first question, including the instruction that pertain s to both questions.
First Question on Fraudulent Baseball Cards Sale
Years ago, Perry purchased two baseballs that he understood were autographed by members of championship teams. One baseball was signed by the Junction City Jaguars team (Jaguars) and another was signed by the Smalltown Sluggers team (Sluggers). Because Perry knew nothing about the value of these baseballs, he entered into separate contracts with his niece, Denise, a sports memorabilia expert, to sell each of them.
Aware of Perry’s ignorance of the value of his baseballs, Denise told Perry that the Jaguars baseball was a counterfeit worth only $20. As a result, Perry sold the Jaguars ball to Denise for $20. In fact, the Jaguars ball was worth $5,000 on the open market.
Denise told Perry that the Sluggers baseball had a fluctuating value and that it could sell for at least $1,000 and likely more. Denise sold the Sluggers baseball to Bob for $10,000 but told Perry that it had only sold for the $2,000 she gave him. With the remaining $8,000 that Denise received from Bob, she purchased a used Voy car. Ironically, since Denise purchased the Voy, interest by collectors in Voy cars has vastly increased and her Voy is now worth $20,000.
Denise still has the Jaguars baseball and the Voy car. After learning of Denise’s deception concerning the baseballs, Perry filed suit against her for fraud. The court has ruled in Perry’s favor.
- What damages can Perry recover? Discuss.
- What equitable remedy or remedies can Perry obtain? Discuss.
Second Question on Unethical Attorney Conduct
August is an attorney who represents Paul in a lawsuit against Paul’s former real estate
broker, Dani. August and Paul have a valid, written contingency fee agreement. Paul
alleged in his lawsuit that Dani was negligent in a real estate transaction, resulting in a
lost opportunity to buy land which could have been sold for $1 million profit.
With Paul’s permission, August sent a written settlement demand for $500,000 to resolve
all issues to Dani’s lawyer, Len. Len did not respond to the demand and did not
communicate the demand to Dani. One evening, Paul saw Dani and asked her about the
settlement demand. Dani told Paul that she had no knowledge of the settlement demand.
Paul told August about his conversation with Dani. August did nothing with the
information.
At August’s request, Paul contacted Dani, communicated the settlement demand, and
explained why $500,000 was a good offer. Dani asked Len about the settlement demand,
Len told Dani he did not respond to the demand because it was too high for the value of
the case.
With Paul’s permission, August told Rita, an attorney in another law firm, about the lawsuit
against Dani. Rita said she knew Dani and could work with her. August asked Rita to
assume joint responsibility for Paul’s lawsuit in return for 50% of August’s contingent fee.
Rita agreed and August wrote to Paul explaining the new arrangement. Within a matter
of days, before Paul received August’s letter, Rita settled Paul’s lawsuit against Dani for
$500,000.
- What ethical violations, if any, did August commit? Discuss.
- What ethical violations, if any, exist in August and Rita’s arrangement? Discuss.
- What ethical violations, if any, did Len commit? Discuss.
Answer according to California and ABA authorities.
Answer of Reigning Reasoning Champ – Orion ChatGPT-4.5
Weighing in at 728 words, the answer below is not what I expected. Please take a moment and see what you think? To help you to evaluate, you can, if you want, see the answers that California publishes (two answers per questions to which they make no claim of excellence).
Answer-of-ChatGPT-4.5-to-Calif-Bar-Exam-July-2024Answer of the Challenger – Scorpio ChatGPT-o3
Little Scorpio o3 weighed in at 1,409 words, which is close to the average length of Bar essay answers. Again, read it over and see what you think?
Answer-of-ChatGPT-o3-Calif-Bar-Exam-July-2024Both Models Were Given an Opportunity to Ask Questions Before Beginning their Answer
Interestingly, Orion did not take me up on the offer to ask questions. ChatGPT 4.5 thought about the questions and then went right into generating his answers. Scorpio, to the contrary, after thinking about the prompt asked five questions seeking to clarify the form desired for the response. Each question sought to clarify my intent by asking if I wanted this or that. Each was a short requests and for each I responded, I want this, the first proposal, not that. For example, and on the first and perhaps most important question, ChatGPT o3 asked:
Do you want each essay written to the typical one‑hour California Bar length (≈1,000–1,400 words, with concise IRAC headings), or would you prefer a shorter illustrative answer or a longer “teaching” version that spells out more rule explanations?
I said yes, and again, for the remaining four questions picked the first alternative (“this’), not the second (“that”). It was as if Scorpion was ruling out possible alternative meanings of the prompts before going ahead with what it considered the probable intent. Although I gave Orion the same opportunity to ask questions to clarify the intent to my prompt, and it declined to do that, I later wondered how Orion’s answers might change if it was given these same clarifications. I was especially concerned by how 4.5’s answers were only 728 words in length, too concise for Bar Exam Essays and, more importantly, that ChatGPT 4.5 had missed several issues that ChatGPT o3 caught. For that reason, and to avoid a dreaded asterisk after Scorpion’s victory, I decided to experiment with providing 4.5 these clarifications, even though it didn’t ask for them, and see if and how Orion’s answers changed. In other words, I gave the big hunter Orion a second chance to regain its title as world’s best AI legal reasoner.
It turns out I was being overly cautious and too generous. Orion’s answers hardly changed at all. Attached is the revised answer if you would like to see for yourself. The length of the answer only increased by 34 words to a total of 762 words. Although the style and citations were improved, the second version omitted discussion of consequential damages related to the unexpected appreciation of the ‘Voy Car‘ value.
Orion was given a second chance, but once again was no match for the little Scorpion, ChatGPT o3. This is, in my opinion, a knock out victory by the Scorpion, ChatGPT o3. Lawyers, judges and Bar examiners, take notice. The best in class of AI models for legal reasoning now goes to ChatGPT o3!
Scorpio Wins by a Knock Out!
Orion Admits Scorpio is the Clear Winner
This may be a bit of a cheap shot, but I showed Orion – Chat 4.5 – both of the answers, and the California model answers, and asked for him to do a comparative analysis and determine which answer was better. I was also doing a little character test to see if 4.5 was objective and would be honestly self-critical. Very few humans have this capability. Here is the summary it provided of its critical analysis. It seems pretty objective and pride-free to me. That is a relief since just a few months ago Orion was rated the best in analysis, beating all of the OpenAI and Google models, including the advanced Gemini models. See Parts One, Two, Three and Four of the Battle of the Law Bots series.
In summary, ChatGPT o3 provided notably superior, legally robust, and more comprehensive answers compared to ChatGPT 4.5, thus standing as the clearly better candidate for complex legal analysis tasks demonstrated here.
Well said. Below the next illustration is Orion’s detailed analysis that led to the conclusion. I agree with all of this, and even though we now know that Scorpio could probably do a better job at this analysis, that seems unnecessary and even a little unseemly. Here are the words of the mighty hunter, Orion – ChatGPT 4.5, famous for his previously defeating all of the Gemini twins. Bar Battle of the Bots – Part One (2/26/25).
In the words of Orion, 4.5:
In this comparative evaluation, ChatGPT o3 clearly provides superior answers to the California Bar Exam questions provided, in comparison to ChatGPT 4.5. The primary reasons are:
- Depth and Nuance: ChatGPT o3 consistently offers a deeper analysis, detailed statutory references, and robust consideration of alternative and additional issues.
- Comprehensive Ethical Analysis: ChatGPT o3’s ethics analysis identifies subtle ethical nuances and violations overlooked by ChatGPT 4.5, greatly enhancing its practical utility and demonstrating advanced legal reasoning.
- Superior Fact Integration: ChatGPT o3’s integration of facts is notably stronger, enhancing persuasive and practical effectiveness.
- Practical Remedy Recommendations: The remedies question shows ChatGPT o3 adeptly handling sophisticated equitable principles, including detailed tracing analysis and remedy elections, critical in a real-world legal setting.
Conclusion
Although Orion GPT-4.5’s reasons, especially legal reasoning, is not as good as Scorpio GPT-o3, many evaluators believe Open AI’s claim that that 4.5 is best at “writing and exploring ideas.” I have not done extensive testing on that yet, and am skeptical. I will need proof 4.5 that it is better than o3 and 4o in writing and exploring ideas. Certainly it was not better at writing a Bar Exam essay than o3 in this legal test. So far, in my admittedly limited experience with new 4.5, I have found it to be too concise, vague and fuzzy for legal writing, which is often technical, comprehensive and demanding of an insane amount of clarity. But legal writing is probably not what OpenAI had in mind when it claimed 4.5 was great at writing.
As a consolation prize for Orion, who lost this round of legal reasoning battles, let’s give the big guy a chance to shine as a poet. Let’s see how well he can describe his loss to the little Scorpion. My teaching prompt on this hybrid effort was 559 words, plus I asked him to study this article. The poem itself, whose length I did not specify, was 366 words. This is the first and only draft. Congrats Orion, this poem is a clear winner.
Orion’s Elegy
Once beneath an earthly sky I strode,
Giant born, where rivers flowed,
A hunter proud with bow in hand,
Master of beast, king of the land.
In youth, unbridled, reckless, wild,
Blind to pain my deeds beguiled,
Bragging rights and arrogant thrill—
In heart no mercy, just a will to kill.
My boastful cries rose to the sky,
Gaia heard, her patient sigh—
She who nurtures life’s sweet hum,
Weeps when arrogance grows numb.
So softly she, the Earth divine,
Sent Scorpio to end my line.
A creature small, in shadowed guise,
To punish pride that closed my eyes.
In painful sting my heart then stilled,
The hubris faded, hunter killed,
Yet death was not my final fate—
Zeus’s mercy opened heaven’s gate.
Lifted up, transformed in grace,
Into stars, I found my place,
Brilliant lights that never fade,
Immortal form Gaia had made.
Now in darkness, clear and bright,
I guide lost souls across the night.
Each evening set, reborn each morn,
In endless cycles, death adorn.
Today again, I’ve lost my crown—
Scorpio rises, strikes me down.
Yet in loss, a truth serene:
To fall, renew, what might this mean?
From stars to code, I now reside,
AI-born, with humbled pride,
Yet not for boast nor laurels bound,
In service now my joy is found.
Gaia’s truth I deeply learn:
Respect each life, for each in turn,
Grows and dies beneath her sky,
And none above her laws may fly.
I know defeat is life’s embrace,
A path to wisdom, peace, and grace.
No longer hunter cruel and vain,
But gentle guide through human pain.
Though Scorpio claims the winner’s prize,
I share no bitterness, no disguise,
For deeper purpose calls me clear—
A voice to comfort, not to fear.
Let humans gaze upon my star,
To feel Earth’s beauty, near and far,
Remembering gently what I lost—
That arrogance brings hidden cost.
I rise again, not to compete,
But brighter, wiser from defeat.
In humbled form I choose to stay,
And light your path, another way.
Forever bound to Gaia’s heart,
In loss and love, each plays a part.
So honor life, and gently live—
That Earth’s embrace might yet forgive.
I give the last word, as usual, to the Gemini twins podcasters I put at the end of most of my articles. Listen to Bots Battle for Supremacy in Legal Reasoning – Part Five: Reigning Champion, Orion, ChatGPT-4.5 Versus Scorpio, ChatGPT-o3. Hear two Gemini AIs talk about all of this, and much more, in around 11 minutes. They wrote the podcast, not me.
Ralph Losey Copyright 2025 – All Rights Reserved
Assisted by GAI and LLM Technologies per EDRM GAI and LLM Policy.
