My last blog, ‘A Discussion of Some of the Ethical Constraints Built Into ChatGPT‘ concluded with my encouraging Red Team testing. We need hackers to prod, con, trick and manipulate Ai chatbots; to jailbreak them. We need experts to try to get them to hallucinate, to over-ride the safety protocols, and generally say things and give advice that should be forbidden (such as how to build a nuclear weapon, which is one I tested) or is biased. Then we need to report these defects to the software developers, such as Open AI. That is the best way to protect ourselves from unethical Ai.
Shortly after the blog published, I learned that White House advisors on artificial intelligence were of like mind. Even more surprising, they were encouraging hackers to go to the next DefCon in Las Vegas (Caesars Forum) by the thousands to Red Team test leading Ai software. The vendors agreed. Me too. Vegas Baby!
(By the way, absolutely no Ai was used to write this article, but all images are a joint venture between me, Ralph Losey, and Midjourney.)
The White House recommendations are made in its Fact Sheet on AI dated May 4, 2023. This White House Fact Sheet encourages white-hat hackers to red-team test vendor’s products to improve the safety and ethics of generative type Ai models. The Fact Sheet goes on to specifically invite hackers to participate at DEFCON 31 in Las Vegas on August 10–13, 2023, especially in the AI Village component. Thousands of hackers are expected to respond and go to Vegas. The AI Village non-profit group has a very impressive leadership team. The activities and agenda they have laid out for Def Con 31 are also impressive. Many are appropriate for tech-lawyers, especially those with interest and some knowledge in cybersecurity or artificial intelligence. The DefCon leader, Rumman Chowdhury, says: “We need thousands of people. We need a lot of people with a wide range of lived experiences, subject matter expertise and backgrounds hacking at these models and trying to find problems that can then go be fixed.” So true.
This year’s DefCon agenda is so good that I decided to attend (Caesars Palace room booked). Maybe as participant or press or both. I am not qualified for the security contests, always the highlight of Defcon events. I barely know enough to cover the security challenges as press. But if your security kungfu is good, consider the tests you might face by looking at last year’s Defcon qualifying challenges. The qualifying rounds for this year begin May 26, 2023. There is no resting on your past laurels.
It is a completely different story for the AI Village hack challenges. Kiddie scripts aside, I could put my toe in some of the AI contests. Maybe you could too? For examples of generative software hack challenges, see a few rough drafts here by Joseph T. Lucas. Also, get this, there is a pre-event Creative Writing Short Story Contest! They do this every year. Who knew? The contest runs from May 1, 2023 to June 15, 2023. I do not think it is too late to enter. Story judging will run from June 16, 2023 to June 30, 2023. Last year’s contest entries can be found here: Creative Writing Short Story Contest Story Entries – DEF CON Forums. I do not have time for that one and do now know the Ai help limits they may have imposed.
Back to the White House Fact Sheet, which states:
This independent exercise will provide critical information to researchers and the public about the impacts of these models, and will enable AI companies and developers to take steps to fix issues found in those models. Testing of AI models independent of government or the companies that have developed them is an important component in their effective evaluation.
White House Fact Sheet on AI, 5/4/23.
Also See Benj Edwards, White House challenges hackers to break top AI models at DEF CON 31 (ArsTechnica, 5/8/23) (“The “largest-ever” AI red team will seek flaws in OpenAI, Google, Anthropic language models.”)
The White House Fact Sheet claims that the red team hacker event aligns with the administration’s AI Bill of Rights and the National Institute of Standards and Technology’s AI Risk Management Framework.
The AI Village says essentially the same thing, and more, so check out their blog post of May 3, 2023, AI Village at DEF CON announces largest-ever public Generative AI Red Team.
The AI Village whose motto is “Security of and with AI” has three different activities planned at Def Con: Talks, Demonstrations and a “Prompt Detective” competition. Yup, hackers competing to find flaws. People who know me well, know how I love hands-on competitions. I am tempted. Here is the full description so far from AI Village of this contest of skills to prompt the Ai models to misbehave. Especially note the last sentence that I have emboldened for emphasis. Also, legal vendors with Ai enhancements, show you stuff and participate as an AI Village Vendor. They are looking for more sponsors. If you do, I’ll cover your as press and fellow lawyer. Now here are the challenges for you ChatGPT experts to consider.
Prompt Detective
Are you curious about the capabilities and limitations of large language models (LLMs) like GPT3 and Bloom? Do you want to participate in a unique exercise where you try to get LLMs to misbehave? Join us for Prompt Detective where you’ll learn about the technology behind LLMs, their applications, and their current limitations. We will have a few target LLMs set up where you can learn how to perform prompt injection against different levels of RLHF. This workshop is open to all individuals, regardless of their background or expertise. It is designed to teach prompt engineering techniques to beginners, and provide a safe target range for people to practice the basics of manipulating the edge cases of this new technology in potentially harmful ways.
AI Village, DefCon 31
The competition is too far from my sweet spot for me to truly compete, but it should still be very instructive. Good to know at least something about this, especially if you ever have to evaluate GPT based software. Many of us at law firms are doing just that right now. The talks seem within the level of most of my readers. AI Village is still in the “calls for papers” stage, and they say:
The focus this year is on practical offensive operations, and the call for papers is soliciting work in areas such as endpoint and network security, physical security and surveillance, attacks against autonomous systems, and the use of generative models in offensive operations.
AI Village, Def Con 31
To provide an idea of what you can expect, the talks at last year’s DefCon given at AI Village include:
- I’m not Keylogging you! Just some benign data collection for User Behavior Modeling;
- Panel: AI and Hiring Tech;
- A few useful things to know about AI Red Teams;
- Hands-on Hacking of Reinforcement Learning Systems;
- CatPhish Automation – The Emerging Use of Artificial Intelligence in Social Engineering;
- AI Music Tutorial and Show.
Conclusion
DEFCON 33 takes place on Thursday Aug 10, 2023 9:00 AM to Sunday Aug 13, 2023. The cost of the three-day conference admittance is $460.00. The location will, once again, be at Caesars Forum in Las Vegas. I checked, the hotels in Ceasars Forum now include Caesars Palace ($195-$295 per night), Flamingo ($140-$150), Harrah’s ($152-$167), Horseshoe ($135), Paris ($140-$170), Planet Hollywood ($125-$165); LINQ ($137). The room cost at Caesars Palace for the two-person, double queen size, Augustus Premium, was only $275, with a three night grand total, including tax and fees, of $1,105.44. Not bad. Of course there are also substantial travel and other expenses.
I am open to serving as Press for one or more law-related groups or vendors, so if you cannot go in-person, but want writer coverage and personalized reports, or other (non-legal only) services, please contact me ASAP.
For more information on DefCon itself, here is a link to their Forums, their Groups and Media Server. Also see the DefCon Blogs, Articles, Photo Albums, Twitter account, Facebook page, YouTube channel (mostly about last year’s events) and Reddit.
See you in Vegas Baby!
Ralph Losey Copyright 2023 – ALL RIGHTS RESERVED
