What a 20-year-old Video Game can Teach Us About LLMs

The AI discourse gets more maddening by the minute. Back in 2022, when ChatGPT was first released to the public, we all were in hopeful anticipation of what was to come. The idea that a piece of software could produce human-like text and was able to converse on a basic level was fascinating. Naturally, we all were there when it happened, and so we all know that the prime time of LLMs was November 2022 and lasted for about two weeks before turning into a man-made horror beyond our collective comprehension.

Shortly before ChatGPT was launched to the public, there was an incident in which a Google Engineer, Blake Lemoine, attempted to “blow the whistle” on the fact that allegedly a Google model has become sentient. This was in June 2022, less than half a year before the ChatGPT launch. I wrote about it, and argued that what Lemoine said back then sounded more like wishful thinking than evidence for sentience. In retrospective, this almost sounds like Lemoine was a prophet in disguise. He believed that an LLM has just become sentient, and was so convinced of it that he got himself fired over it.

Today, more than four years after this incident, LLMs are everywhere, and the basic sentiment that preceded OpenAI’s big launch has remained. LLMs have retained their shroud of mysticism, their veil of magic, while they are at the same time being used in real-world applications right now, and have a tangible impact on the world. It is quite interesting to put into perspective that someone has essentially given away the whole story behind AI before anyone outside of OpenAI even knew that we would soon be able to converse with language models.

I originally wanted to argue how the AI discourse has been shifted from real existing technological applications to esoteric notions of doom and gloom, but after rediscovering this case, I think it is the other way around. AI was always shrouded in mystery; generations of researchers and technology evangelists have imbued the concept with expectations about future potential and unrealized capabilities. All of these notions are direct descendants of post-apocalyptic and futurist ideologies from “Dark Enlightenment” to “Accelerationism” that have taken a hold in Silicon Valley well before the last AI winter came to an end.

With the launch of ChatGPT, however, there was suddenly a contrast to the stories from Silicon Valley. Now, AI was not just the golden calf of a community of techno-progressivists, but a real existing technology that the average Joe could utilize and, crucially, could form his own opinion on. Suddenly, the evangelions from the popes of futurist technology competed with the perception of what AI truly was, now that humanity had the chance to experience the second coming of Christ for themselves.

Now, it’s fun to narrate this story using religious metaphors, but it feels a bit off. Not because it would be wrong — Silicon Valley tech workers have a knack for believing in superstitions —, but because there is so much harm that this overdetermination of the concept has done. And this is what I want to focus on today.

Some observers of the AI industry have already alluded to the fact that the “AI discourse” seems to subsume two very different narratives — one about the actual existing technology that is present right now and which has influence on society right now, and one about vague and nondescript future capabilities and risks that especially the leading figures of the AI industry promote to large audiences.

When a disgruntled office worker talks with a tech CEO about “AI,” they both use the same terms. But what they understand by these terms, what the concepts mean for them, could not be further apart from each other. The CEO would praise the possibilities of AI, what it all could do for us, while the office worker moans about the 572 AI generated emails they received just today, all of which they are contractually obligated to reply. The tech CEOs of the world are talking in the future tense, while the workers of the world are shouting from the rooftops how much they suffer under the various AI mandates of their employers.

The current discourse is poisoned by the creators of ChatGPT and Claude. And this is sad. Because LLMs are great — if they are used in the right settings. For example the abstracts for my articles are increasingly produced with only an LLM and some manual adjustments.¹ But when our tech overlords talk about either the “existential risks” that emanate from LLMs or the “future possibilities,” this real value of LLMs gets drowned out. More so, by focusing on unspecified “risks” that cannot be grasped because they are not real, and by emphasizing vague “possibilities” that no one can judge because LLMs simply cannot (yet) do it, tech CEOs can prevent a discourse from forming which focuses on the current use-cases and environmental risks of AI.

Commercial LLMs threaten the environment, the power grid, the communities in which their data centers are built, and the work force itself. They are tools for depressing wages,² controlling workers, and to lure in investors. At the same time, locally running LLMs are increasingly used for small, but useful tasks. Much less than what Sam Altman has promised us, but more than what we could do with automated software just a few years ago.

In short, we need an entirely new term, a new concept, to differentiate the fever dreams of some lonely tech oligarchs from the real benefits that LLMs can bring (and how to realize them without burning down the rainforest).³ We need a way to completely shield the discourse around what LLMs are doing right now — both their grave risks and negative environmental impact — from what Dario Amodei dreams of at night.

But how?

This is where I would like to introduce you to a video game that has solved this issue more than two decades ago. Just as Blake Lemoine preceded the collective psychosis in Silicon Valley, this video game has pre-empted our current predicament with LLMs during the last financial crisis.

I am talking about Mass Effect (2007), a Sci-Fi game with strong story elements, set in the somewhat near future. The important parts of the story for our purposes are explained quickly. (Spoilers ahead, skip the next paragraph if necessary.)

Mass Effect’s main story arch revolves around an ancient species of sentient machines that have made it their task to destroy all space faring civilizations of the galaxy every 50,000 years. They are referred to as “the reapers.” The story follows the player in the role of Commander Shepard, who, across three games, tries to stop this cycle of mass genocide. While the “reapers” are nothing more than a variant of the paperclip maximizer, the crux of the story is that the entire game is built around the notion that organic beings and sentient machines cannot get along and somehow always end up killing each other. As a result, the game has a strong opinion about AI. The entire galaxy has banned the development of artificial intelligence, and nobody really questions the motives of the “reapers” — everyone just wants to see them gone. But unlike Dune, the Mass Effect writers didn’t want to fully give up on computers, and the citizens of the galaxy need them to steer their spaceships, and compute routes through spaces. By doing so they faced an issue: how do you allow computers while distinguishing them clearly from sentient machines?

The writers of Mass Effect solve the problem of having computers but not AI with a remarkably simple plot device. To build an artificial intelligence, you not only need computer parts, but a special piece of hardware that is never fully explained. The writers call this a “blue box.” This is a very elegant solution for the story, because it allows the writers to have computers and give the player an intrinsic understanding of the difference between computers and AI, while still maintaining AI as some form of machine. But, more importantly, the writers still required somewhat capable computers, something like machine learning algorithms. For that, they invented a new term: “virtual intelligence.” In the universe of Mass Effect, a “virtual intelligence,” or VI, is a tool that runs on essentially what we would call LLMs: A conversational language model that can answer queries in natural language and that has access to a knowledge database. Typically, VIs can perform a limited set of actions, something we would refer to as “tool calling.” But they cannot do more, and the important part is: everyone in the universe understands this.

Across all three games, it is always clear that VIs are never sentient, regardless of what they look like. When characters in the game interact with a VI that refuses to do some task, they sometimes swear — but always in a way that makes clear they know it is just a computer. It is intrinsically clear that these tools, while impressive and useful, are not sentient. Whenever an actual AI enters the stage, however, it is immediately clear that it is an AI, because it acts unprompted, on its own, and has opinions.

This is an incredibly useful linguistic device: By separating the uncanny valley of sentient computers from smart, but soulless VIs, it is much simpler for the player to understand what type of help they can expect from a VI. And I believe that we need something similar.

I don’t think we should adopt the term “virtual intelligence,” both because it is phonetically close to “artificial intelligence,” and because the term is a bit boring. But the basic idea remains. If you think about it, we don’t really have a term that uniquely describes a language model that can support humans without being part of the broader AI discourse. Tim Cook tried this, but “Apple Intelligence” has its very own problems. Whenever we say LLM, we may refer to the model we interact with right now, but there is always a little Sam Altman in the room who confabulates some existential risk in the unspecified future. Whenever we let an “agent” solve a bug in our code, there is this unspecified fear this will cost us our job. And whenever a CEO talks about implementing a new customer support chatbot, we all know how this is going to end.

This is the power of discourse. By ensuring that certain terms are intrinsically part of a specific cluster of meanings, tech CEOs can ensure that any criticism of AI gets invalidated. When we use these terms, we automatically buy into their side of the story; whether we like it or not. When I talk with colleagues about LLMs, I almost always feel tempted to specify “Oh, and I mean locally running AI. Not the big chatbots. And I’m not using them invariantly. Only for this specific task.”

Remember further above, when I mentioned that I’m generating abstracts using an LLM? I felt the need to add a footnote reassuring everyone that my writing is still mine, and that I only delegate tasks to LLMs that aren’t the core of the article. LLMs are great in summarizing articles, so why not let them, to give you a quick TL;DR of my work to decide if it is of interest for you? But there are all these additional negative connotations. AI being a tool against workers; a slop producer; a copytheft machine; causing environmental issues; and the people known for it just generally being unpleasant people.

Imagine a world, we could just talk about … gizmos. For example, I could tell you about the little gizmo that I currently use to quickly summarize research papers to gain a quick overview over what the papers say. What my gizmo can also do is give me some friendly reviews of my papers that can help me identify some weaknesses that need fixing before I submit the paper. I also heard that someone has built a gizmo to offer a friendly natural-language interface towards their research database. The only thing I’m waiting for is that gizmos get smaller that I don’t have to clog my entire system memory with one just to create a draft of an abstract for my article.

Now, this is obviously just an odd example. But imagine how much easier talking about LLMs could become if we could just ignore all these loaded terms and let Dario, Sam, and Elon fantasize about getting mangled by Roko’s Basilisk in private. This would be a real empowerment. And it would probably finally allow me to stop writing about the misuses of AI. Because we won’t stop Silicon Valley tech bros from believing in ghosts. But what we can do is prevent ourselves from getting worked up about how we should feel towards real existing AI.

If a bunch of writers could do that twenty years ago, we can do so, too.