A few days ago, I stumbled upon a discussion in the Open Source Subreddit. It was started by an employee of a small startup that was creating an Open Source application that people could also pay for to receive enterprise support. The employee mentioned that the community was in uproar, and they weren’t sure who was right and who was wrong. He then linked to the corresponding discussion on their GitHub repository.
Here’s what has apparently happened: they built in telemetry and realized that, if people can turn it off, they will do so. The company didn’t like that and at some point just disabled the users’ ability to turn off telemetry. This in turn led to an outcry with users sharing tips on how to forcefully turn off telemetry.
The argument by the employee is that, without telemetry, they have no idea how their software is being used and thus have a much slower development speed. So the argument is fundamentally “efficiency in the development process which would ultimately also benefit users” versus “basic privacy”.
I tend to agree with the users here.
“But don’t they have a point in that telemetry will indeed help them develop faster?”, you may now say. And indeed, if you have large amounts of data, you can conveniently query the database and look for potential issues and avenues for improvement. But after having been lead on a very large software project myself for more than half a decade, I can tell you that it is a false sense of security, believing that telemetry will save your software.
And I am not saying this because I’ve had my fair share of this debate as well (which is why Zettlr comes with zero telemetry; I tried this myself and failed – rightfully so). I am also saying this because I had to come to the realization that data is rarely your friend, and no amount of data can replace a good communication strategy.
There are three parts to this conundrum, which I will introduce in the following before concluding why telemetry won’t save your company.
User privacy
The first part is user privacy. Telemetry fundamentally violates the (human) right to be left alone (see UDHR, § 12). Yes, there are ways to anonymize the data, but research has shown that even perfectly anonymized data is still never anonymized (Montjoye et al. 2015).
Research on what is called “differential privacy” (DP; Dwork et al. 2013) was able to quantify the tradeoff between anonymity and usefulness of the data. Specifically, the better you anonymize your data, the less useful it becomes to you, because anonymization directly correlates with the resolution of it. Remove resolution, and you’ll be left with only averages that don’t tell you anything. At the same time, researchers are getting more and more successful in deanonymizing data, because beyond a certain threshold, the amount of data points is sufficient to track every single human being in earth (Thompson and Warzel 2020).
Now, companies will at this point often say that they obviously will keep your data secure and not mishandle it. How well this works can be seen roughly every two weeks when another data breach becomes public. (If you’re too lazy to search, here is one of the top Google results.)
As a general rule of thumb, every single bit of information that leaves your computer, phone, or tablet must be considered public for all intents and purposes. Users know that, and that is why they always turn telemetry off if they can. Because users are smarter than us developers commonly like to admit. I will come back to that.
Especially egregious is this data collection if it is opt-out only, legally in a gray area, and the corresponding settings are hidden behind dark patterns.
Incidentally, Microsoft is the best example here. Most of you who use a Windows computer will probably have already complained that Microsoft is really pushing its copilot thingy down your throat, exploiting muscle memory to make you open and thus use it.
Even tech-savvy users aren’t safe: VS Code is full of telemetry, A/B testing and other experiments that I do not wish to partake in. But it’s excruciatingly difficult to turn all the various settings off because they’re (a) opt-out only; (b) very cleverly worded; and (c) well hidden within the litany of other settings.
With software giants, we have arranged ourselves because they are monopolies. There is not really an alternative to VS Code, Photoshop, or Google. So we begrudgingly accept that we trade data against convenience.
But small startups don’t have this power, and this is what leads to these (justified) outcries, if one of them does this.
Data Poverty
But say you have figured out a way to collect perfectly anonymous data and know for a cryptographic fact that the data are indeed 100% safe — let’s just assume this on a hypothetical basis.
Then the data will still not help you deliver a better product. The reason is data poverty.
I’ve been doing large scale statistics for the past three years, and let me tell you a secret: data are so sparse that even with a benevolent setup, you’ll have a hard time producing findings. All data on the world are inherently very poor, or sparse – however you want to call it.
The thing is, a datum is merely a fact, and a single number says nothing. 128 could mean a color value, a piece of memory, or just a street address. Without context, you will never know.
And before you now shout “Well, but that’s why we collect so much! So that the context of all values taken together gives us unambiguous indicators of what the user’s intentions are!”
Oh, really? So the information when a certain button is clicked at a specific point and then a few keys are being pressed tells you anything about a user’s intentions? Entertain me, because if that’s true you have literally played through sociology. Here’s your trophy.
No for real: no data analysis ever escapes the state of educated guessing and, as I have been figuring out through a very painstaking process that is still far from over, the biggest feat is not collecting data or even running the analyses — it’s figuring out how to run an analysis that brings into connection data points based on theoretically grounded insights to answer a question as good as possible, knowing that you’re still probably off.
That’s basic science. Collecting telemetry is similar to working with digital trace data, and once you start thinking about it, it is extremely difficult to approximate user intentions. If you’re not convinced, feel free to talk to my colleagues who are doing this on a daily basis, I’m sure they’d happily tell you how wrong you probably are.
Long story short: even if you collect mountains of telemetry, you’ll probably draw the wrong conclusions and, in the worst case, even make your app worse to use. The reason is that what you need for proper software development is to capture a user’s intentions, and no telemetry data can tell you this. Nothing comes close to the actual thought processes that happen when we use an app. There is always a barrier between our thoughts and the imprint we leave on the world (Vaisey 2009; Hedström 2021). The only way to get to this is to ask a user to reflect on how they’re using the app, and if that makes sense to them.
The User is Smarter Than You Think
This brings me to the second to last point: your users are smarter than you think. And they are creative. This is something that is hard to fully grasp as long as you don’t have a good communication strategy, but it is something I continuously find when I talk to my users. (By the way; I realized that I keep making the impression that I have a good communication strategy. I need to mention that my “strategy” is “talk often and talk much” – I doubt that this is the most efficient method.)
First, as a developer, it’s easy to think that your users probably expect to get their asses wiped with golden tissues because if this or that button does not exactly do what the user expects, they’re gonna grab the pitchfork and storm your office. But it turns out that this is not true – especially not for Open Source software, and not even for commercial products.
The thing is, users are more forgiving that we maybe like to admit. Specifically, we can expect a few things from our users:
- They are aware that they’re using free software
- They do not expect everything to work perfectly; only a “reasonable” working state
- They are prepared to do some “hacks” to make those parts work that don’t natively work for them while they wait for you to improve that
- They are patient
Now, of course there are always people who will complain about anything, but these should not be your guiding light. Instead, focus on the vast majority of people who fulfill the criteria above.
Let me give you just two random examples from my experience.
Link Titles Workaround
Zettlr allows you to implement a basic “Zettelkasten” methodology, meaning: Something that works a bit like a Wiki where you can inter-link various files (notes) on your computer using square brackets. This then allows you to write something like this: An OLS-Regression is a statistical [[method]] that allows you to test data for linear relationships. The “method” word is surrounded in square brackets and can contain a filename so that you can click that link and go to the “method” note. However, you probably don’t have a file named “method”. It’s probably called “Methods” or “List of Methods” that serves as a table of contents for all your notes on various methods that you have collected.
Now, as with Wikipedia, oftentimes the grammatical structure of a file’s name doesn’t align with what the grammar of the name within a sentence needs to be. This is when link titles come into play. For example, you could write An OLS-Regression is a statistical [[method|List of Methods]] that allows you to test data for linear relationships., where “method” is the title that will appear, and “List of Methods” is the file name.
Well, at the time of writing, Zettlr still does not support this. Yes, this is not great, but you know what? Users were creative. For the better part of the past 6 years, users simply used a hack that was working in any Zettlr version prior to v3 — they wrote something like this: An OLS-Regression is a statistical [method]([[List of Methods]]) that allows you to test data for linear relationships.
In the context of Zettlr, this means that you will by default only see the word “method” which makes the sentence grammatically correct, but if you enter the link with your cursor, you would then see the actual Wikilink to “List of Methods” which you could then follow.
This was never intended, but it worked for an astonishing amount of people. And it turns out that users were recommending this trick among each others without my knowledge (sneaky!). Just a week ago, I found out that even a colleague of mine two rooms down my corridor was also using this hack, and I have definitely never told her that.
But you know how I found out how many did this? In Zettlr v3 I re-worked the entire parser of the main editor, which also included making it impossible to write Wikilinks within Markdown links. And that was when suddenly mountains of people flooded the GitHub issues section saying that their links were broken!
Now my priority list has slightly shifted. But it’s fine. And you know why? Because once I told the users opening the corresponding issues that this was never intended and that there is a better way, they responded: “Ah alright, good to know! I’ll wait.” and are now waiting for me to actually implement the promised change. Because they know it’s Open Source; I have an actual full-time job; and they are happy that I even put up with this work in my free time without any remuneration.
And this is something that no amount of telemetry can tell you, except you literally siphon off the exact contents of users’ documents, and I believe it would be hard to justify this within any existing privacy law.
Long Backlog
Another story is just fresh from a few days ago. I have just installed Linux on my work computer. This also included installing and setting up Zettlr. However, there is some data that is not synchronized via a cloud service, which includes file templates that I have grown accustomed to. And I realized that this is actually a real pain, because there is no simple way of keeping this info in sync.
So I complained on the Zettlr Discord server that I will probably implement something like this sooner rather than later. Immediately, one user linked to a GitHub issue that basically asks for this feature. I quickly said thanks, clicked the link … and found the issue had been opened over three years ago. It turns out it is not a new feature request, but I never got around to implementing this.
But you know what? Said user said: “Hey, don’t stress, we all know you’re busy!” And that is the attitude I see in most of Zettlr’s users: they are all acutely aware that it’s only one person working on the app in their free time, and they are fine with this.
Long story short: Users are very forgiving when it comes to free software. And they will happily cooperate with you to make things work (or make things work for themselves until you provide a proper way of doing so), but only if one crucial condition is met: communication.
Conclusion: Telemetry is not a Replacement for a Good Communication Strategy
So what does this show us? First, if you develop Open Source, your users will have the corresponding mindset. They will not expect you to show constant development speed, and they will not complain if their issue sits open for a few years. As long as you don’t charge for the software, how could they?
Second, your users will manage, even if the app doesn’t do exactly what they need it to. As long as it works reasonably good, they will not suddenly abandon it just because there is some tiny thing that bugs them.
Third, there are gazillions of issues that may arise with your app, all of which can never be captured with mountains of telemetry data, because that data cannot capture a user’s intentions. There is no way around actually talking to your users. I know that it sounds appealing not having to talk to people — I don’t like being on the phone either —, but it is a simple fact that only communicating with people will tell you (a) their expectations; (b) where they need improvements; and (c) where they are perfectly fine with the state of affairs.
Telemetry is not a replacement for a good communication strategy. And I would think that developing a good communication strategy takes just as much time as tweaking your telemetry collection. Only that one of the two gives you much richer information and, more importantly, tells your users: You’re there. You are aware of their issues.
References
- Dwork, Cynthia, and Aaron Roth. 2013. ‘The Algorithmic Foundations of Differential Privacy’. Foundations and Trends® in Theoretical Computer Science 9 (3–4): 211–407. https://doi.org/10.1561/0400000042.
- Hedström, Peter. 2021. ‘Coda - The Past and Future of Analytical Sociology’. In Research Handbook on Analytical Sociology, edited by Gianluca Manzo. Edward Elgar Publishing. https://www.elgaronline.com/view/edcoll/9781789906844/9781789906844.00035.xml.
- Montjoye, Yves-Alexandre de, Laura Radaelli, Vivek Kumar Singh, and Alex “Sandy” Pentland. 2015. ‘Unique in the Shopping Mall: On the Reidentifiability of Credit Card Metadata’. Science 347 (6221): 536–39. https://doi.org/10.1126/science.1256297.
- Thompson, Stuart A, and Charlie Warzel. 2020. ‘12 Million Phones, One Dataset, Zero Privacy’. New York Times, 26 January 2020, sec. F. https://www.nytimes.com/2020/01/03/opinion/twelve-million-phones-one-dataset-zero-privacy.html.
- Vaisey, Stephen. 2009. ‘Motivation and Justification: A Dual‐Process Model of Culture in Action’. American Journal of Sociology 114 (6): 1675–1715. https://doi.org/10.1086/597179.