Stochastic Parrots: How to tell if something was written by an AI or a human?

[Editor’s Note: EDRM is proud to publish Ralph Losey’s advocacy and analysis. The opinions and positions are Ralph Losey’s copyrighted work.]

There are two types of “tells” as to whether a writing is a fake, just another LLM created parrot, or whether its real, a bonafide human creation. One is to look at the structure and style of the writing and the other is to look at the words used. This blog examines both GPT detectors, starting with the tell words.

All of the words used here are “real,” written by Ralph Losey, whereas all of the images are “fake,” prompted into existence by Ralph using various image generation tools, including his own GPT4: Visual Muse: illustrating concepts with style. This blog is a continuation, a part three, of two other recent blogs, Stochastic Parrots: the hidden bias of large language model AI, and Navigating the High Seas of AI: Ethical Dilemmas in the Age of Stochastic Parrots.

The ‘Tell Words’ favored by Stochastic Parrots

We begin with an introductory video of a Pirate who looks sort of like Ralph talking about AI tell words. The pirate, of course, is fake, created by Ralph Losey, but all of the words were personally written by him.

Just click the image to watch on YouTube Matey.

Transcript of the Pirate Video on AI Tell Words

Ahoy Mates!

Did a real person write my pirate script? Or was it written by an AI? A Stochastic Parrot on me shoulder?  Aye, Matey, it happens all the time these days. Who, is telling who, what to say?  Blimey! How can you tell real writing from fake? Arr, matey, listen up.  I’ll give ye some tips on how to tell the difference!

Here are the top “tell words” favored by our phony feathered friends.  Its not foolproof, as even I have used these cliche words, that AI seems to love so much. Yee be a teacher or supervisor concerned about AI plagiarism?  Then you might save this list. These damn AIs. Stochastic Parrots, they be called, often use these words  incorrectly, or too much. Here they be. In rough order of parrot popularity.

Synergy, that be the worst of all, followed by Blockchain, which is often used inappropriately. Here are some more beauties to look out for: Leverage. Innovative. Disruptive or Disrupt. AI-driven. Pivot. Oh, I hate that one!

Here are more parrot nasties. Scale. Agile. Think outside the box. Paradigm shift. Bandwidth. Deep dive. Shiver me timbers! Are you starting to feel sick from all the vague cliches?

Aye. But There be more, many more. They all need to go to Davey Jones Locker: Ecosystem. Due diligence. Empower. Holistic. Ah, that was once a good word.

Blimey! Here’s  a really bad one. Game-changer. This makes me heave ho! Follow the link below to hear the rest!

List of “Tell Words” Indicative of AI Writing

Thanks again to the “No More Delve” GPT for helping me with this. I recommend this program. These are roughly ranked in order of misuse. This list does not purport to be complete nor based on scientific studies.

1. Synergy – Often used to describe the potential benefits of combining efforts or entities, but frequently seen as vague.
2. Leverage – Intended to convey using something to its maximum advantage, but often considered jargon when overused.
3. Innovative – While it’s meant to describe something novel or original, its overuse has dulled its impact.
4. Disruptive – Used to describe products or services that radically change an industry or technology, but now seen as clichéd.
5. Blockchain – Specific to technology but has been overused to the point of becoming a buzzword even in irrelevant contexts.
6. AI-driven – Meant to highlight the use of artificial intelligence, but often used without clear relevance to actual AI capabilities.
7. Pivot – Originally meant to describe a significant strategy change, now often used for any minor adjustment.
8. Scale – In business, it’s about growth, but its frequent use has made it a buzzword.
9. Agile – A specific project management method that’s now broadly used to describe any flexible approach, diluting its meaning.
10. Think outside the box – Intended to encourage creative thinking, it has become a cliché itself.
11. Paradigm shift – Used to signify a fundamental change in approach or underlying assumptions, but now often seen as pretentious.
12. Bandwidth – Borrowed from technology to describe personal or team capacity, its metaphorical use is now considered clichéd.
13. Deep dive – Meant to indicate a thorough exploration, but often used unnecessarily.
14. Ecosystem – In technology, refers to a complex network or interconnected system, but often used vaguely.
15. Due diligence – Critical in legal contexts but used broadly and sometimes inaccurately in business.
16. Blockchain – Bears repeating due to its pervasive use beyond relevant contexts.
17. Empower – Intended to convey delegation or giving power to others, it’s now seen as an empty buzzword.
18. Holistic – Meant to indicate consideration of the whole instead of just parts, but often used vaguely.
19. Game-changer – Used to describe something that significantly alters the current scenario, but now seen as hyperbolic.
20. Touch base – Intended as a casual way to say “let’s communicate,” but often viewed as unnecessarily jargony.
21. Blockchain – Bears repeating due to its pervasive use beyond relevant contexts.
22. Delve: Often overused to suggest a deep exploration, diminishing its impact.
23. Journey: Used metaphorically to describe processes or experiences, becoming a cliché.
24. Supercharge: Tends to overpromise on the impact of strategies or tools.
25. Embrace: Frequently employed to suggest acceptance or adoption, often without specificity.
26. Burning question: A dramatic way to highlight an issue, but overuse dilutes its urgency.
27. Unlock: Commonly used to imply revealing or unleashing potential, becoming worn out.
28. Roadmap: Overused in business and technology to describe plans or strategies, losing its originality.
29. Uplevel: Buzzword suggesting improvement or upgrade, often vague.
30. Future-proof: Used to describe strategies or technologies, but often without clear methodology.
31. Revolutionize: Promises transformative change, but overuse has made it less meaningful.
32. Navigate: Frequently used to describe maneuvering through challenges, becoming clichéd.
33. Harness: Suggests utilizing resources or forces, but overused to the point of vagueness.
34. Transform: A catch-all term for change, its impact has been diluted through overuse.
35. Drives: Often used to denote motivation or causation, but has become a buzzword.
36. Realm: Used to describe fields or areas of interest, but has grown to be seen as pretentious.
37. Vibrant: A go-to adjective for lively or bright descriptions, now seen as overused.
38. Innovation: Once meaningful, now a generic term for anything new or updated.
39. Foster: Commonly used for encouraging development, but its impact is lessened through overuse.
40. Elevate: Used to suggest improvement or enhancement, often without clear context.
41. In summary: Overused transition that can be seen as unnecessary filler.
42. In conclusion: Another filler transition that may unnecessarily signal the end of a discussion.
43. Testament: Often used to prove or demonstrate, but has become clichéd.
44. Unleash: Implies releasing potential or power, but overuse has weakened its effect.
45. Trenches: Metaphorically used to describe deep involvement, now seen as overdone.
46. Distilled: Suggests purification or simplification, but often used vaguely.
47. Spearhead: Used to denote leadership or initiative, but has become buzzwordy.
48. Revolution: Promises dramatic change, but is overused and often hyperbolic.
49. Landscape: Used to describe the overview of a field or area, but now feels worn out.
50. Imagine this: An attempt to draw the reader in, but can feel contrived.
51. Master: Suggests a high level of skill or understanding, but often used imprecisely.
52. Treasure trove: A clichéd way to describe a rich source or collection.
53. Masterclass: Intended to denote top-tier instruction, but has become a marketing cliché.
54. Optimize: Common in business and tech to describe making things as effective as possible, now overused.
55. Pioneering: Meant to convey innovation or trailblazing, but diluted by frequent use.
56. Groundbreaking: Similar to “pioneering,” its overuse has lessened its impact.
57. Cutting-edge: Used to describe the forefront of technology or ideas, now a cliché.
58. Impactful: Intended to denote significant effect or influence, but overuse has rendered it vague.
59. Thought leader: Aimed to describe influential individuals, but has become a self-applied and diluted term.
60. Value-add: Used to highlight additional benefits, but has become a buzzword with diluted meaning.
61. Big data: Meant to describe vast data sets that can reveal patterns, trends, and associations, but now often used as a buzzword irrespective of the scale or complexity of the data analysis.
62. Thought leadership: Intended to denote influential and innovative ideas, but overuse has made it a nebulous term often devoid of evidence of leading or innovative thinking. (By the way, see thought leader on a Linkedin profile, better run!)

Looking Beyond the AI Favored Words to AI Typical Writing Styles

1. Generalized Statements: AI-generated content often leans on generalized or vague statements rather than specific, detailed examples. When you read an AI written article, notice the details, or lack thereof. Fake LLM writing are often generalized and overly formulaic. They string many words together in an appearance of learned comprehension, but in actuality, they say little. I call it more fluff than substance. In other words, they talk like a typical politician, with lots of words, but little meaning. This gets even worse when the AI does not have access to up-to-date information, or is generating content on a topic on which its data is limited, like lots of “insider baseball” talk. In my field, that includes the kinds of things that lawyers and judges privately say to each other, but are seldom, if ever, written down, much less published. Just go to a bar at a Bar convention. This limited detailed knowledge makes it easy for human experts in a field to detect fake writing in their own area, but outside their field, not so much.
2. Neutral and Diplomatic Language: AI often defaults to very neutral and diplomatic language, sometimes excessively so, in an effort to avoid making controversial or unsubstantiated claims. “WTF” is not a phrase, or even initials they are likely to use. The software manufacturers try to filter out the profanities found in many parts of the internet, which is typically a good thing. Still, the result is often unnaturally squeaky clean language and style that make friggen fake language easy to spot.
3. Excessive Politeness: Especially in responses or interactive content, LLMs use phrases that seem overly polite or formal, such as frequent use of “Thank you for your question,” or “I’m sorry, but I’m not able to.” Miss manners in writing is a dead giveaway. Plus, its so damn annoying to real people, thank you very much!
4. No Real Humor or Wit. The kind of snarky, subtle, almost funny remarks that permeate my writings seem to be beyond the grasp of AI. Much like the AI Android “Data” in Star Trek, LLM’s just don’t grok humor and their jokes usually suck.
5. Lack of Emotion. Kind of obvious, but robot writing often has a tell of being robotic, overly structured, intellectual and emotionless. Personality and emotion seem irrelevant to these writing algorithms, although we are seeing new types of LLMs that specialize in emotion, so be careful, they can be charming and seductive too. See: Code of Ethics for “Empathetic” Generative AI.
6. Too Perfect Spelling. Real humans make typographical errors, and, even with spell checkers, there are often a few mistakes that slip by. My blog is a good example of this. A typo in a text is a good indicator that it was human written. Of course, this again is just a tell. We writers always strive to be perfect and smart computers can help us with that.
7. Lack of Personal Experience: AI-generated texts often lack personal anecdotes, experiences, or strong opinionated statements, unless specifically programmed to simulate such content. This reminds me of lectures I have sat through by self proclaimed e-discovery experts who have never personally done a document review. I could go on and on with examples, if I wanted, because I have a lifetime of experience and my learning is hands-on, just just academic. Remember, no AI has personal experience of anything. It is all second hand book learning, albeit millions of books.
8. Lack of Opinions: AIs are often trained by their makers not to be opinionated. After all, opinions might possibly offend someone. Can’t have that. That is one reason AI writing is often bland, which is another tell. Real humans have opinions, lots of them, many wrong. But that’s one reason we are such a charming species and our writing is so much more enjoyable to read. AI talk is not only general and vague, it is often stogy, over polite and politically correct. Also, try getting an AI to give you its legal opinion. Thank goodness for lawyers like me they refuse and say instead to speak to a real lawyer. There are, of course, many ways to trick them into giving you a legal opinion anyway, but that’s another story, one that you will have to retain me to tell you.
9. AI Avoids Slang and Uncommon Idioms. Since GPTs are trained on public data, they use the most common words, the ones they have literally read a billion times. I may be tilting at windmills, but in my experience GPTs are as blind as a bat to many idioms, phrases and words. We are talking about speech that is too regional, subcultural, seldom used, or still too new, the latest vocab. Unfortunately, I’ve checked, matey, and GPTs do speak fluent pirate. Arr, there be many of us pirates about.
10. Repetitive language: AI written content may repeat words, phrases, or sentences. Yeah man, like over and over. In fake writings you may also see the same sentence structure repeated in different paragraphs. They repeat themselves and don’t seem to care, over and over. That is one reason fake AI talk can be so boring. I mean, they just keep saying things to death. Enough already!
11. Lack of creativity: AI writing is often too predictable. Well, duh or course. LLM intelligence does come from predicting the next word you know. The Top_p or Creativity settings may be too low. Again, the sooo boring speech results. Humans are typically more enjoyable to read. That takes us back to know-it-all Data on Star Trek who does not understand humor. Subtle humor is just not something they grok. Maybe someday. 
12. Error Patterns in Complex Constructs: AI may struggle with complex language constructs or highly nuanced topics, such as law (which is one reason they wisely refrain from giving out legal opinions). This mental challenge when dealing with complex ideas, will, in my opinion, be corrected soon. But in the meantime this limitation in intelligence leads to errors in speech like topic relevance and basic coherence. The LLMs now often may sound like a 1L trying to explain the elements of contract when cold called in Contract One, or like a newbie lawyer trying to argue a subject to a judge where they have only read the Gilbert’s. Note this last run-on sentence could not possibly have been written by an LLM.
13. Overuse of Transitional Phrases: AI tends to overuse transitional phrases such as “Furthermore,” “Moreover,” “In addition,” and “On the other hand,” in an attempt to create cohesion. They seem to lack creativity in that regard, or fear simply doing without transitions. In any event, they often screw up transitions.
14. Hedging Language: AI often uses hedging language like “It might be the case,” “It is often thought,” or “One could argue,” to avoid making definitive statements that could be incorrect. It all depends if this is appropriate or not. Just ask any lawyer. We and our insurers frigging love qualifiers.
15. Synonym Swapping: To avoid repetition, AI might use synonyms excessively or in slightly unusual contexts. This can lead to awkward phrasing or slightly off usage of certain terms. Comes from their being so damned repetitive and too general. The result is often the use of unnecessary “fancy words,” typical of a linguistic show-off.
16. Repetitive Qualifiers: AI might use repetitive qualifiers like “very,” “extremely,” “significantly,” more than a typical human writer would, to emphasize points. I am very guilty of that myself.
17. Standardized Introductions and Conclusions: AI-generated content might start with very formulaic introductions and end with standardized conclusions, often summarizing content in a predictable manner (e.g., “In conclusion,” followed by a restatement of key points). The fondness of opening and closing statements is much like a lawyer in at trail or in legal memorandums, but AI does a poor job at it. I always end my blog with a conclusion and I like to think that most of my blogs do not read anything like fake talk by a LLM. AI writing with catchy, not mere formalistic conclusions, will improve in the future, of that I have no doubt. But in the meantime, this is yet another trait that is a dead giveaway.

Conclusion

These tells words and styles must all be taken with a grain of salt. They are only indicators. None are dispositive, but, taken all together, they can help you to determine if a writing is real or fake. All of these indicators should be used as part of a broader assessment of real or fake, rather than a one and done acid test. As AI technology evolves, the patterns and tells will likely become less noticeable. For this reason, I predict that orals exams will become more popular. Teachers on all levels will be forced to revert to the medieval guild traditions of oral defense of knowledge and skills, as has always been required of PhD candidates.

The obvious should also be stated here. For most people the greatest tell of all of fake writing is how good it is compared to past efforts, education and experience. A sudden improvement in a student’s writing is a strong tell of plagiarism. Indeed, aside from the few humans that professionally, most flesh and blood intelligences are relatively poor writers as compared to LLM writers. Look at the history and qualifications of the writer. It may make it obvious that their writing is too good to be true.

Still, any objective observer, no matter how much they may dislike AI, would have to concede that GPTs can already write better than most people; most, but not all. GPTs are still, at best, mere B-grade writers, often far worse, for the reasons here listed. The can be vacuous blowhards, mere parrots, pushing watered down Gilberts. They can be all style over substance, not to mention inaccurate and sometimes hallucinatory. They are not even close to the best humans writing in their fields of special expertise, legal or otherwise. Yes, I am expressing a strong opinion here that might offend some. Don’t like it? Go back to reading the bland crap of stochastic parrots!

Still, despite these protestations, I love these odd birds for their great potential as hybrid tools. They should be used to help us to speak and think better, and make better decisions. But Stochastic Parrots should not dictate what we say or do, especially those of us engaged in critical fields like law and medicine. It is one thing to use LLMs for writing fiction, quite another to make life and death decisions in courts and hospitals. AI should be skillfully used by professionals as a consulting tool to assist, not replace, good human judgement and empathy.

Published on edrm.net  with permission. Assisted by GAI and LLM Technologies per EDRM GAI and LLM Policy.

Ralph Losey Copyright 2024 — All Rights Reserved. See applicable Disclaimer to the course and all other contents of this blog and related websites. Watch the full avatar disclaimer and privacy warning here.