Today I want to talk about a recent case that you may or may not have heard of already. A team of researchers from the University of Zürich wondered how dangerous generative AI such as ChatGPT could become if wielded by the wrong people. They were asking themselves: “Can generative AI convince people of arbitrary viewpoints?” In and of itself an interesting research question. But the issue is not the research question in this case. No, the issue is the study design. Because these researchers decided that a lab experiment would not yield proper insights. Instead, they wanted to conduct the study “in the field,” that is, in a setting not defined by experimental conditions.
So here’s what they did: They chose the Subreddit r/ChangeMyView, which lets users post a view they hold, and let comments try to convince them of another viewpoint. They inserted themselves into the discussion, but instead of trying out various argumentation techniques, they simply let an LLM generate a response with some predefined view, and copy-pasted that generated text into their comment. Their IRB indicated some concerns with the study design (German source), but they proceeded to do it anyway. Once the study appeared as a preprint, users of r/ChangeMyView saw it, and they were furious.
To get an impression of how bad the situation was, take a look at the official comment from the research team explaining and defending their research to the users of that Subreddit. And then, take a look at the comments to that explanation. Here’s a small selection:
The mods and the members of this Community affirmatively revoked consent to interact with AI comments.
“We decided that didn’t matter” is simply not good enough.
This is a gross breach of ethics.
Or:
After completing data collection, we proactively reached out to the CMV moderators
That's not what "proactively" means.
And, finally:
Please cite any other studies where researchers use psychological manipulation techniques on participants who did not consent.
You have confirmed that we now no longer know if these posts and comments are just bots or real people, which leads to the inevitable reverse, where real people facing difficult situations are dismissed as bots. It potentially destabilizes an extremely well moderated and effective community. That is real harm.
You say your study was guided by your so-called principles, including user safety. Frankly I think you are lying. You didn't give a damn about others to do this study, because if you had you would have easily followed the "user safety" principle to it's logical conclusion, given your choice of topics to have the LLM comment about.
How do you think a real user who was dealing with serious trauma from sexual assault would feel after finding comfort or agreement with your bot comments, now finding out that was fake. That is real harm.
You even tried to convince users that the current situation in the US isn't really a big deal, we should focus on other problems. That is political manipulation, and while I understand this is a small community when compared to the global population, this could impact voters. Done at the wrong time of year, that's foreign election interference, a crime.
I'll be reporting your paper on every platform that I see it published.
As a scientist myself, you should be ashamed.
As you can see: The research team really did get burned. And I believe they had it coming. In the remainder of this post, I will talk about the fact that there are already studies that follow a very similar pattern, which resulted in similar outrage; what I think this shows about the approach of many researchers to ethics; and how we can safely navigate research ethics as a community.
One reason for why I am writing this post now, two weeks after the incident, is that over the past two weeks I have observed something concerning. Several people I’ve seen commenting online seemed very surprised by the amount of backlash. However, I don’t believe any of this is surprising, given the gross ethical negligence at play. But it concerns me that I regularly see other social scientists surprised about the harm that ethical misconduct can cause. It shows to me that we are still not in a place where ethics is taken as seriously as it should. I will provide some additional examples that show that sometimes, researchers can be surprisingly naïve when it comes to regulatory frameworks that apply to their research.
The bottom line is: While we researchers often can feel like we’re playing in a sandbox, the real world is not a sandbox. And if we don’t “eat our vegetables” (thanks to Jacob Habinek for providing me with this apt metaphor), then we can’t have fries either.
An Almost Exact Predecessor: The “Hypocrite Commits” Case
First and foremost, if you were surprised by this incident, you should know that it is not the first one of its kind. In fact, just a few years ago, a study has caused similar outrage that used, essentially, structurally the same study design. It is known as the “hypocrite commits” study. Essentially, researchers were tricking real people – Linux Kernel maintainers – by sending them faulty patches, to check how vulnerable the Linux Kernel is to adversaries. Since it’s humans who review patches, of course there is some form of vulnerability. We aren’t machines, so the outcome of this study was entirely predictable: Some of the faulty patches made it through, and were only stopped by unblinding the research subjects. Which then caused backlash.
Essentially, the research team has achieved to get the entire University of Minnesota (not just their own team) banned from ever contributing to the Linux Kernel again. The community was outraged, because none of this was communicated in advance.
Except for the research question, the recent Reddit experiment followed exactly the same study design.
No “Respect for Persons”
“But how can we avoid this backlash?!” you may now ask. Essentially, if one doesn’t know much about ethics, it may now appear as if this could happen to everyone. But that would be very detrimental, because if we are scared of our research subjects, we won’t be doing much of the necessary research to understand human society. So how can we avoid becoming the target of such a backlash?
It’s actually very simple: Show “respect for persons.” That’s it.
Okay, it’s not that simple, and it requires an explanation. “Respect for persons” is the first ethical principle outlined in the Belmont Report. The Belmont Report is one of the ethical guidelines published about 50 years ago, in 1978. If you are somewhat trained in social science methods, you will know the principle “respect for persons” as the principle of “informed consent.” Informed consent requires researchers to tell any research subject beforehand what they are doing, and why, and asking them whether they agree to participate in that study. It’s just one example of the broader principle of “respect for persons,” but an easy one to remember. Essentially, respect for persons means: Imagine you were a research subject in your own study — would you feel respected after you read the published paper? If you can’t answer this with a confident “yes,” this already indicates that there might be an issue.
And essentially this is what has happened to the hypocrite commits and CMV papers. If these researchers had reflected before committing to the paper, and thought “What would I think if I learned of this research, and I was a Linux Kernel member/CMV member?”, they would probably have already guessed that conducting a study without informing anyone in advance is a bad idea.
The CMV paper folks, however, had a reason for why they never informed their research subjects: They argued that conducting a lab experiment with informed consent would’ve tainted the experimental results. Essentially, they said, “we can’t get to the real data if we do all of that in a lab.” Which is simply wrong. No, I believe that them conducting this on Reddit had two other reasons. First, using Reddit makes the research catchier and easier to communicate, because a lot of people know Reddit, and it highlights how this could happen in the real world, not just in a lab. But second, I also suspect that they simply didn’t want to set up appropriate lab conditions. Because that takes time and money. Reddit already has a platform, they wouldn’t have to invent an experimental setup.
From this view, I think I even find the argumentation from the hypocrite commits team more believable: They argued they didn’t think about the fact that it was real humans who would read and parse their emails. It’s still a stretch, but it’s more believable than the argumentation of the CMV paper authors. Made worse by the fact that the CMV paper authors are social scientists, the hypocrite commits authors are computer scientists. The former should definitely have enjoyed better ethical training.
Ethical Principles are not Hard
This leads me to the next section. I strongly believe that both papers wouldn’t have experienced this backlash if the researchers took ethical principles seriously. Ethics is not actually that hard. Sure, it adds about a week of deep reflection to the research process, and time is money. But I believe that even a superficial knowledge of the Belmont and Menlo Reports (the Menlo Report updated the Belmont Report in 2012 with a fourth ethical principle) would’ve captured many of the most egregious ethical violations. Here, let’s quickly walk through the four central ethical principles:
- Respect for Persons: We already talked about that. Get informed consent where possible. When that is not possible, think about why. If you can come up with a very good reason for why you can’t get informed consent, think about “What would I think if I were a research subject in my own study?” If that thought seems fine for you, good!
- Beneficence: This one is a bit harder, but essentially it means: minimize risks, maximize benefits. Risks here is any harm that may be caused by your research. This again requires imagining yourselves as a research subject in your own study. What harm may you get? Also, not all risks are created equal. Some risks are perfectly fine to imbue on your research subjects. Others are much more detrimental. Sometimes it’s more important to minimize risks than maximizing benefits.
- Justice: This connects to the second principle. Whatever risks or benefits your study comes with, ensure that you distribute this equally on all parts of your study population. Simple example: If you are conducting a study among black and white Americans, ensure that it’s not just the black study population who bears all risk, and that it’s not just the white study population that reaps all the benefits.
- Respect for Law and Public Interest: This is the principle the Menlo Report added to the three of the Belmont Report. It contains three elements. First, don’t do illegal research. Second, be accountable for your research. Third, be transparent. I personally find two questions very apt for thinking about whether you conform to this principle (aside from, you know, literally reading the law and consulting a lawyer). First, would you be fine with defending your research in front of your research subjects? And second, would you be fine with having to defend it in court? If you answer both questions with a confident yes, you probably aren’t about to commit egregious ethical misconduct. If not, you absolutely need to rethink your study design.
With a very basic understanding of these four principles, I believe you can avoid the most detrimental ethical issues you could face. If you truly think about these issues, and adapt your study design accordingly, I don’t believe any research subject would be very mad at you. A few are always mad, but at least I don’t think your entire study population would hate you for it.
That being said, didn’t the researchers have better options? Let’s think about this next.
What They Could’ve Done Better
First and foremost: I’m not an experimental sociologist. I have never conducted an experiment myself. So take the following with a grain of salt. But I did read many experimental studies, and I participated in quite some experiments myself.
I believe that both teams would’ve avoided getting roasted by their study population with one very simple change to their study design, without yielding unusable data. The first thing they should’ve done was to actually conduct an experiment in a lab. The reason for this is twofold. First, I do not believe that doing so would’ve invalidated the data in any way, and second, this would’ve made basically everything easier, and would have prevented the two papers from getting retracted.
The CMV paper authors have argued that a lab setting would’ve tainted the data. I believe that this is a lie. Think about it: The authors say that they wanted to know if people can get convinced by a machine. You can replicate this easily in a lab. Simply tell people: “We want you to tell us a belief you have, and be open to let yourself be convinced of another viewpoint. Then, we are going to show you a few counter-arguments, and you have to select how much you feel convinced by each one. Some of these may be AI generated, but most are human generated.” This way you retain the core of the research question (the participants do not know in advance which comments are AI generated), and get an unbiased estimand, you are transparent, and you even get informed consent. And, you would even get an actual proper random sample of the population, not just the heavily biased Reddit community. If you so wanted, you could also have replicated the CMV community setup specifically, and set up a very simple forum, where the fact that “there are some AI generated comments” is clearly communicated, but which still essentially replicates the CMV community. If you still believe that this wouldn’t be as “true” data as you want, then you should probably not adapt your study design, but your idea of man. The exact same holds true for the hypocrite commits paper.
Now, again, I am not an expert in experimental study design, but I don’t think that “If we don’t do it in the real world, we won’t get the correct data” is a valid argument to make. And so I fully side with the outraged study populations in both cases. Essentially, the argument of the researchers implies: “We don’t trust other human beings of taking us seriously, and so we need to lie to them in order to get correct responses. If we invited them into our lab, they would just deceive us.” And this is a very concerning idea of man.
Other Instances of Naïvety
Ethical and legal conundrums don’t stop there. Another thing that I observe frequently across the board are legal and ethical violations of a more benign form. Specifically, with the rise of generative AI, many social scientists are now using ChatGPT, Mistral, Claude, or Gemini in their studies. This very often involves sending some user data to these proprietary service providers. And there are already several studies that do that. What these researchers frequently do not realize is that any user-generated data is either personal data (and as such worthy of extra protection), or plainly copyright-protected. Every piece of text anybody anywhere on earth produces is, by default, copyrighted, meaning that you have no right to send it to a third-party provider without consent.
A common counter-argument is: “If these people participate in my experiment, they can be implied to have agreed to that.” No, it’s not. What they often just agree to is: “You can use my data for research purposes.” They didn’t agree to “OpenAI can have my data.” And this argument becomes even weaker now that we have very capable models that we can run locally. I doubt users would object to you using generative AI in your research as long as they have consented to you using the data. But they are right in objecting to you sending the data to a third-party service provider unless explicitly granted in the consent statement. They granted you the right to use the data for research, they didn’t grant you the right to send it to someone else. Especially since we’re in Europe, the legal boundaries are very clear, and very strict. And if you think it’s annoying, then again, I don’t think it’s a legal issue, it’s an issue with respect for persons.
And this issue isn’t even new. Ten years ago, when I first was unleashed on Bachelor students at my former university, the “TurnItIn” service was becoming popular as a quick way to check for plagiarism in student essays. This service works by collecting a vast database of text and basically doing an advanced form of substring matching. However, for that to actually work, it requires researchers to upload the student essays to the service. And TurnItIn tells very plainly that they would indefinitely store those essays. Many of my colleagues were using the service very liberally, simply uploading all student essays from a course. They were aware of the potential privacy issues, but you know what their solution was? Simply strip away any personally identifying information. Which, in some way, made the problem even worse. Removing author information ensured that no student could actually exercise their GDPR rights once the GDPR was finally enacted. All of these student essays are still part of this database. And even though the students can today exercise proper data protection rights, they actually can’t because their teacher has stripped all their authorship information from the PDF file before uploading.
You know how many plagiators we caught this way that we would not have caught based on other clues and experience with a close reading of the essays? Zero. Now, there is an argument to be made here that university teaching conditions are so bad by now that it is impossible for teachers to properly do all the grading manually. But I don’t know if violating students’ rights is the proper way to solve these issues.
Concluding Thoughts
I believe social scientists need much better training in the ethical and legal surroundings of our research in order to be able to properly understand what they can and cannot do. Just as law is not optional, ethical reflections aren’t either. Just because you are in a legal gray zone or your IRB has said “Whatever, go ahead” doesn’t mean you should. Our actions have consequences, and as such we must make sure we do due diligence in order to avoid these consequences to be detrimental to either us or our research subjects. We may be experts in our field, but we are not experts in human relations. And our research subjects deserve to be treated with respect, regardless of whether they are university professors or unemployed high-school dropouts.
The carelessness demonstrated by the two studies I discussed harms not just their specific research subjects (and, ultimately themselves). No, this carelessness harms research in general. If this becomes a pattern — research does something, subjects are outraged, lawsuits follow — this will erode trust in scientists. It will establish our reputation as people who think we’re something better than our research subjects; as people who treat other people as literal lab rats. And this will reduce the general populations’ willingness to engage with our research, regardless of whether we actually do our ethical due diligence, or not. This kind of behavior is simply not acceptable.
I expect every researcher to take the basic established ethical principles to heart, and act accordingly. You don’t have to catch every ethical conundrum. We all make mistakes, and I myself have done mistakes. We all can’t become ethical experts. But there is a very big range between “Don’t care” and “Due diligence.”
So: Read the Belmont and Menlo reports (it takes you one afternoon), think about them, and apply these principles from hereon. It’s really not that difficult.
In a previous version of this article, I falsly claimed that the University of Zürich's IRB has granted a waiver for the research. These parts are now corrected. Thanks to Sebastian Gießler for pointing this out.