On AI Text Watermarking

Recently, the European Union started requiring generative AI providers to watermark things generated by their models. This applies to text, images, or videos. The decision has been discussed somewhat controversially. Some are happy about the new regulations, as they see the increasing prevalence of generated data online as a threat to the democratic culture, and fear that misinformation is being ballooned by artificially generated misinformation. Others are appalled by the change, since they believe that using generative AI in any way is fundamentally an expression of free speech, and a choice that should not be captured by regulations. In the following I will exclusively talk about text watermarking.

I'm a text analysis scholar, and as such I know that there are more dimensions to this issue, and neither side is really correct. Do not get me wrong: both sides have valid arguments. I, too, am worried about misinformation online. Given that large groups of the population are easily swayed – even before the advent of LLMs – the ability to quickly generate large swathes of misinformation exacerbates the threat to democracy. On the other hand, even with watermarking mandates, there will always be models that generate non-watermarked text and there will be tools to remove such watermarks. So for any actor of any reasonable level of determination, circumventing these requirements is trivial. Additionally, I use generative AI for my research. If the models statistically select certain tokens more often than others with no correspondence in its training data, this can have a real impact on my ability to draw any kind of conclusions.

Or does it? See, here's the first problem: text, and language more generally, are first and foremost social constructs. We don’t use language as an end goal, but as a means to communicate information. So in theory, Anthropic or Google are right when they claim that “changing some tokens with plausible alternatives” will not perceptibly change the semantic meaning of the generated text. But, on the other hand, we have no way of knowing what effect these watermarking techniques will have on downstream applications. Finally, there is a lot of misinformation about language and how watermarking will impact generated text out there, so I believe it’s in order to address this and outline how this will impact text generation.

What is Text Watermarking?

The basic idea behind text watermarking is that the European Union wants anyone to be able to trace back where any portion of text longer than 200 tokens comes from. If the text contains the statistical signs of the Anthropic watermark, we know that it hasn’t been written by a human. Likewise, if the text contains no discernible statistical pattern, we can be reasonably sure that a human has written the text. (Before you complain, I know that this is all very hand wavy, but I want to keep it simple.)

The way this is implemented is essentially the following. When LLMs generate text, they predict the token that is likely to follow the already present text in their context window. So when the model has already generated the text “The weather is nice and…”, it will use this as a context to decide on the next token. To do so, the LLM will assign every token in its entire vocabulary a probability of it being that next token. Afterwards, there will be a bunch of tokens with a high probability, but many more with a low probability. For example, both the words “sunny” and “warm” could complete the sentence sensibly and, more importantly, semantically very similar. Watermarking is a process that will nudge the probability of the model choosing, say, “sunny” over “warm” in this context just enough that it becomes statistically detectable. That’s why the regulation demands “at least 200 tokens” – with too few data points, it is impossible to detect this kind of watermarking, because it isn’t based on specific words (which could be easily filtered out) or actual existing characters (which would be even easier to filter out), but on statistical patterns. The website declaude has a nice visual explainer for this procedure.

This approach works reliably, and it doesn’t just hold for LLM-generated text. Researchers have been using statistical analysis to identify authors of texts since a long time. A 1963 paper, for example, has successfully de-anonymized the Federalist Papers by performing statistical analysis on the writing of the suspected authors.¹ So this is a good method to ensure we can retroactively determine the provenance of some piece of text.

Now, I want to turn to some misconceptions about this process and how it can alter LLM-generated text, and discuss some implications of artificial text generation and watermarking that I think are of utmost importance.

Misconceptions about Watermarking

As an example for the grave misconceptions that exist around watermarking and its influence on LLM-generated text, I want to focus on John Gruber’s recent takes on it. Specifically, in his piece “Anthropic’s ‘Watermark’ Text Adulteration In Claude Is a Perversion of Writing,” Gruber claims that watermarking will perturb the text, and in a bad way. Arguing that “most people are terrible writers,” he is appalled by the thought that AI watermarking would make text generated by Claude (or any other model, really), less good than it would otherwise be.

To prefix this, I enjoy reading Gruber’s blog. Gruber is a man of strong opinions, but, more importantly, of consistent opinions. And I like that. Because when he’s right, I can agree wholeheartedly, but when he’s wrong, I have a lot of material to work through. While he’s generally quite good on general technological things, he is neither a programmer nor someone with intricate understanding of LLMs. But both are topics he recently discussed heavily on his blog, which is great for me, because now I have a lot to work through. So keep that in mind while reading the following. It is not to single him out or say he’s an idiot (although he might think the same of me). It’s just convenient.

“AI Watermarking Reduces the Quality of the Generated Text”

To quote:

I want any LLM I use to choose the very best, most precise words at every single decision point. An obvious constraint that I accept is time and computation. Within the constraint of executing inference quickly, and at a certain cost per token, I want the best words. This constraint matches human writing. […] The idea that anything other than my needs should factor into the generation of text for me is patently offensive.

He is victim to a logical fallacy here. He assumes that there is a “best” combination of words given certain constraints. But that’s just not how language works. He spends paragraphs complaining about the fact that, if you task a model to generate a response to “What’s your favorite fruit?” and it responds with “Mango and banana” that this is less perfect than responding with its actual top-word “pineapple.” But here’s the thing: What actually would be the best word in that context? Sure, banana and pineapple are not interchangeable as things, but as there is no ground truth text that we could reference to see that “banana” was the wrong word in this context, we cannot call it a “wrong word.”

This fundamentally misses the point of what language is supposed to do, and how LLMs work. First, the model has never eaten a fruit, and as such, Anthropic’s and Google’s claims are correct that “pineapple” is exactly a replacement for “banana” in this context. If you tasked the model to summarize an article discussing fruits and then ask it to answer “What is the author’s favorite fruit?” and it would respond with “pineapple” even though the article mentions “banana,” that would obviously be an error. The generated text of LLMs is always probabilistic. As long as the “temperature” setting of the LLM is set to anything but 0, the model will arbitrarily choose a token from a small set of highly likely tokens. Therefore, his argument is wrong and misses the point entirely.

Secondly, this argument ignores what language is supposed to do. Gruber writes:

So even in a private conversation between a user and Claude, which will never be read by anyone other than the user, Claude will begin making word choices in the name of marking its output in statistically predictable ways rather than maximizing clarity and precision.

Again, this phrase demonstrates a decisive misunderstanding of language. “Maximizing clarity and precision” can be done in many ways, and again, as long as the watermarking is done properly, this should have no perceivable impact on the text. He is afraid of a ghost here, because he ignores the central puzzle piece in any text comprehension: the reader.

Text is not a mathematical formula that must be perfect, lest it becomes wrong. It is something to be interpreted by humans. The “best” word in some context is simply the word you choose, which is to some degree arbitrary. What it needs to achieve is communicate something to the reader of the text. If the reader understands what you mean, then your text has fulfilled its goal. If it doesn’t, then it likely isn’t because you picked a slightly different synonym for a word. And, if a language model generates a wrong word that is semantically close to the correct word, then again, watermarking is not to blame, but probably your prompt.

Every text is produced for an audience. It is not a painting where a single wrong line may ruin a feature. Text is open for interpretation, and we humans are great at interpreting text. Just imagine how differently literary works have been interpreted by various people. The same holds true if you prompt an AI model. If it picks slightly different words, you will still be able to understand its generated text without any added difficulty.

To drive this point home, imagine that I mention in a sentence that I develop some “large Open Source project.” If you’re a recurring reader of this site, you will immediately recognize that I am referring to Zettlr. Did this “maximize clarity”? That depends on the goal of the sentence and you, the reader. If you already know about Zettlr, I could have just written “Zettlr,” and thereby increased the information density by many counts. However, for anyone who just stumbles upon this article, this will not make any sense, because they may not know what the term “Zettlr” refers to. But in a context where I write about my hobby, even people who don’t know what it is exactly that I do, “some large Open Source project” is perfectly clear for everybody. In short, the words we choose are not just dependent on what the words themselves are, but the context in which we write them, and what we wish to communicate.

I get the feeling that his real issue is that he feels other people decide what is best for him; he feels “parented.” Because while he complains at length about this new watermarking, he is very explicit in that it probably will not impact him at all. He argues at length that writing takes time and dedication and that writing is deeply personal, and that he would never let an AI write his texts. So even if an LLM doesn’t pick his “best” words, he himself does. He even argues that he will likely not see the difference, but he is still upset. This is a very U.S.-centric world view in which the government is always seen as bad, and rules are always an infringement on personal liberties.

And this leads him to another fallacy.

“The Regulation is Useless Because it Won’t Stop Bad Actors”

To quote:

Complying with this, particularly with regard to text, is only going to create problems for honest users. Dishonest users attempting to pass off AI-generated text as their own writing (students, employees, whoever) will simply circumvent detection through non-compliant AI paraphrasing tools.

This is certainly true, but misses the point once more. One context in which I hear many colleagues and acquaintances worry about generative AI is university teaching. I have many colleagues who have expressed their worry that they cannot know whether a student has written some essay by themselves, or whether they simply let an AI do the work for them. This has led to the proliferation of “AI detectors” that are essentially just random number generators. In turn, this has my (and my colleagues’) students in fear of being falsely accused of having used AI. But in the absence of any useful tool to detect whether a text has a human provenance, or a machine one, AI detectors are something many of my colleagues consider.

What Gruber misses here is that the purpose² of the regulation is not to fully prevent LLM-aided spread of misinformation. It is to nudge most people into just not bothering. It sends a decisively social signal. I see the regulation as incredibly important to skim the most glaring misuses of AI from actors who aren’t determined enough, that is: students. If text generated by AI models becomes easier to be identified correctly, this solves these problems we have at university at once. Teachers will know exactly what to look for, and students can rest assured that they won’t be falsely accused of having used AI. This is because proper watermark detectors will replace those horrible AI detectors that cause so much distress in academia.

In short, if you signal to people via regulation that the use of LLMs to skip work you are supposed to do yourself is not appreciated, this will increase the cost for students to cheat. Once the costs of cheating (having to run LLM-generated essays through another tool, maybe even having to pay for it) outgrow the savings, students will be naturally nudged to just write their essays themselves. At the same time, university teachers will be nudged not to run every piece of text through these terrible AI generators anymore, which reduces the amount of false positives. In turn, this will reduce the fear of honest students of being falsely accused.

Now, it is true that determined actors will have easy ways around this. They can either run their text through non-compliant tools, as Gruber mentioned, or just use a local model. In short, the regulation will do little to counter misinformation online, but it will already provide just enough guardrails that we all can rest a little easier in everyday situations. The watermarking will serve its purpose. Even if a student uses a watermarked tool, but then spends enough time rewriting the generated output so that it becomes indiscernible from human text, I (personally) believe they probably have engaged with the source material enough to get a pass.

“There is a ‘True’ Data Generating Process”

Finally, a misconception that none of the authors explicitly mentions. It is rather implicit in what they write. If you are now afraid of AI watermarking “meddling” with the text, you are probably more naïve than you think. Because if you are afraid that the AI model providers are only now messing with the output of their own LLMs, you likely have never thought about the fact that they may have been doing this all along. Furthermore, logically, this implies that there is a “correct” or “true” data generating process that leads from the training data in the models to a generated text. But that is not how it works.

There is no “true” data generating process that produces text. Neither in machines, nor in humans. How we string together characters to form words to form sentences to form entire texts is inherently probabilistic. You might even think about AI watermarking techniques of “giving the LLM a character.” Because that’s essentially what this is. Let’s return back to the paper I mentioned earlier that de-anonymized the authors of the Federalist Papers. The reason the researchers could do this is that every human being uses function words slightly differently. But of course this isn’t just limited to function words. We all have our quirks when writing, and that’s what makes our texts enjoyable to read. I subscribe to about a dozen personal blogs which I read often. And despite some of them talking about the very same topics, their takes all read differently. Both because they have a different perspective, but also because they use a different language to talk about them.

The same holds true for LLMs. Depending on variations in the training data, the training regime, and whatever companies are doing after the fact to steer their output, LLMs will use different language to talk about the same topics. We can essentially view AI watermarking as just another form of giving the LLM a personality — just one that’s statistically detectable.

Crucially, however, if you are afraid of AI watermarking, this gives me the impression that you took LLM produced output quite literally in the past. And that assumes that companies have essentially left the LLM-generated output alone. But given that any tool on earth allows us to play with the top-k, top-p, min p, temperature, and so on, which change the way the LLMs pick the next token from its probability distribution, means that we already messed with the words they chose.

Think about it: If a company overwrites the probability distribution of the LLM, did it really “prevent” the LLM from picking the “correct” word? Does the LLM even care? And, should we even care? I argue that there is no true data generating process, and as such you cannot distort the models’ probability distribution in a way the word “distort” would imply.

Open Questions and Implications

But these various misconceptions about how LLMs work also point to some questions, or maybe implications, that we ought to address sooner rather than later. Because this whole discussion about “We don’t want your models to change” demonstrates that many simply misunderstand not just LLMs, but language as a whole. In short: If you’re using LLMs correctly, you will never be at risk of being affected by a model provider suddenly changing the way the model generates its text. Conversely, if you feel that you are at risk of the model’s text generation process being altered, you’re using them wrong. Let me elaborate.

First: What even is language? This may sound like a silly question because we all use it every day, but it’s not so silly once you start thinking about it. There are various philosophical arguments for different angles on language, that I don’t want to dwell on here. Rather, I’d like to emphasize the point that language primarily serves the purpose of communicating information.³ And it does so in a lossy way. We can never fully express all the information that is stored in our brains into language. That’s where the phrase “to be lost for words” comes from. The only goal of any language communication is that whoever reads some text, or hears a speech, understands the gist of what is mentioned. Obviously, there are some pieces of text that need to be more precise (hello, academia), but there is a limit to how much you can communicate with language alone.

This leads to a second question: What is the specific type of language that LLMs produce? Again, it is a form of communication. But this time, it is not purpose-driven like human language; it cannot be a signalling device. Instead, LLMs generate text out of its humongous training corpora. Essentially, LLMs are large ZIP-archives which you can query. Media scholar Roland Meyer has once beautifully phrased it: One must imagine any trained neural network as a giant archive of its training corpus. You query information stored in this archive using a prompt, and the LLM will generate a response, mixing together some pieces of its training data to provide you the most probable answer to your query. If the information you seek was legion in the training data, you will get a perfect response. If the information you seek was not part of the training corpus, it will hallucinate an answer. So what an LLM communicates is never intent or purpose, but merely information. So as long as the watermarking is done right and does not replace crucial parts of a sentence, there is little danger in exchanging words that would change the response’s meaning. And if there is, one must rather question their own faculty of reading, rather than a company changing a few words.

Finally, this raises a third question: How should we use LLMs? LLMs are certainly very useful, but as this article was hopefully able to demonstrate, language is fuzzy and as such not as precise as to use their output directly in some downstream analysis. I personally believe that the best use-cases for LLMs are three-fold:⁴ quick questions for which a Google search might take longer; classic “rubber-duck debugging” where they help you think through arguments or hypotheses; and summarization of much longer texts for quick first overviews over some textual material.

We have not yet solved the puzzle of how to make LLMs generate text with specific characteristics; for example to use their generations in our data analysis pipelines predictably and with sufficient statistical certainty that we can draw proper inference on it. The emergence of AI watermarking has just highlighted this issue once more. We still have no way of knowing what the data-generating process of an LLM looks like. And with the sudden realization that LLMs might come with all kinds of post hoc probability distribution meddling, this is even harder.

This is a final thought I cannot emphasize enough: If you are right now worried about watermarked LLM responses, you have not paid attention. You have no way of knowing what kinds of mingling the AI labs have done to their models before you were even allowed to use them. They control the entire pipeline of their model, from text that gets ingested, to how it generates a response. If you are worried about AI watermarking now, this implies that you implicitly assumed that the big AI companies have just sent you the “unadulterated” output of these models until now. But you cannot prove that. They may have done watermarking all along.

We will fundamentally need a different approach to using LLMs in both public discourse and social scientific research going forward, and I hope to be doing this research in the coming years.

After all, I believe that LLMs are immensely helpful tools that will make many tasks much easier, faster, and better over time. But they cannot do so unless we start treating them as what they actually are – tools – rather than magic boxes spewing out text that take all the coding and texting work from us.