Listen. I know I’m not an engineer. I have never studied or learned anything formally that I’d be legally allowed to call myself an engineer. But engineering is something everyone can do, sometimes better, sometimes worse. Hell, even vibe coding is a form of software engineering.1 And I have been doing engineering quite a lot over my lifetime. And one constant throughout these years, from my initial contact with PHP through a lot of JavaScript, and the Python I’ve been using during my PhD degree, is that there is always more to learn.
Disclaimer: In case some actual systems engineer stumbles upon this text at some point, and sees all the many flaws in my amateur explanation below, I’d be interested in your feedback, because I have this feeling this won’t be the last time I’ll have to design a system from scratch. So please be gentle with me; I am sharing some experiences, not actual lecture material, of which I am mostly oblivious.
One of the most important lessons my amateur engineering adventures have taught me is that the actual writing of code is just a tiny fraction of the work. Most work when you program something – be it a data analysis pipeline, a small script, or an entire desktop grade application – is to think about how you want to solve an issue you are facing. There are many roads to Rome. But only a few of those roads end up in an algorithm that doesn’t make you yell at your computer why the code runs slow. Thus, the most critical part about software engineering is to understand the environment in which your code will run, and the data it will have to churn through, and the tools the chosen programming language gives you to solve the issues at hand.
But there is another perspective that I didn’t have to think about too much while working on my small little scripts. I believe there are three “layers” to software engineering. First the actual code that should perform the work. That’s what most people who don’t code themselves think software engineering is all about. Second is the thinking about the code, reasoning about what your programs actually do, how to optimize it, and exploiting patterns in data and usage to make it more efficient. This is what most people who start to code slowly come to realize as the core part of engineering. And then there is a third layer, which moves the second layer from a single application to having to think about an entire system of software components that all have to interact to facilitate a working service. And that layer is what I wish to talk about today.
Stumbling into Systems Engineering
During the past year – the in-between PhD defense and the start of my Postdoc – I have been working in a project at my former university. A major part of that project was to build up a research data infrastructure that would allow the project to share a large corpus of about 100 GB of PDF files of program archives from the Swedish Folk Education System (“Folkhögskolor”) with researchers and the interested public. And, for the better or worse, my PI believed I’d be the ideal candidate to lead this effort. What neither him nor me expected was the third layer that I have mentioned above, and which has made this much more difficult than either one of us anticipated.
So, what is a “research data infrastructure”? While not a formal term, it describes a system of various software components that facilitate storage, retrieval, and transformation of a large amount of data for research purposes. Such a system should allow large amounts of data to be ingested, include a frontend to allow others to browse the data, and include mechanisms to produce additional data from the corpus. Lastly, having an R package as a way to interact with the data more directly from statistical analysis code would also be nice to have. That’s it. That’s what this infrastructure should be about.
Infrastructure, here, is a key term. Because in this instance, we aren’t talking about a single piece of software. We are talking about multiple interdependent software components that have to interact with each other to make the system go. We are talking about a storage solution that can store and provide large quantities of data (in this economy!); a server capable of serving possibly hundreds of researchers pegging some API with thousands of requests a second without buckling; and then of course authentication mechanisms because a lot of data isn’t in the public domain – it includes personally identifying information and other legally relevant types of data. Suddenly, we aren’t talking about some medium-sized monolith2 anymore, we are talking about multiple pieces of software and hardware that all have to come together.
The three layers of engineering as I see them right now are simply writing some code (easy), developing a sense for understanding how to write efficient code (medium), and finally planning an entire software environment that fulfills a set of requirement as best as possible (hard). These layers are hierarchical, with each subsequent layer implying the previous. You don’t have to understand all the various environments in which some desktop application will run in, but you will have to write code. When designing an entire system, you will at some point have to develop the individual pieces of software, which involves both coding and writing efficient code.
Attempting to Design Some Research Data Infrastructure
I have always said that writing code and writing papers, while in many ways comparable, requires two entirely different logics. A computer is deterministic, and as such there are right and wrong ways to program, while humans are not, and as such you can tell various stories in a paper, none of which are objectively wrong. But it turns out that designing an entire software system requires yet another type of thinking, which I simply never did before.
To be clear, I did have to interact with the environments in which my applications and servers run quite a bit. And, over the years, I have slowly built up a basic understanding of the hardware, and why some algorithms run better on certain architectures than on others. But all of this was mostly prompted if my primary application, Zettlr, was experiencing a hiccup that I could not explain.3 There was no systematic effort on my side to understand entire systems, because I simply didn’t have to.
But now I had a set of requirements at hand, and had to understand all components before I even wrote the first line of code. This is entirely different from having a single programming problem and having to devise an algorithm to solve this. Now I had to figure out what type of server we would need, how to store all the data, and how the various components would interact with each other. Over the first three to four months of the project I slowly but steadily came to the realization that I, quite literally, had no idea what I was doing.
So I took a step back, and started thinking about this methodically. Of great help were some colleagues from DIE in Bonn, who employ actual software engineers. Luckily for us, they were facing the exact same issue and had to devise a similar system. Fortunately for us, they already did the planning and thankfully shared a detailed infographic with us that I could essentially just adopt.
After a few weeks of sitting in front of various whiteboards at my university, and emailing various people employed in the IT department, I had a rough understanding of what hardware we would need, and the software components.
The system also includes a software I have dubbed “data analysis pipelines.” The idea here is that one can write some Python modules which read in the raw corpus data, run some analysis code, and write their results into the main database, from which the API server can then provide this information in addition to the PDF files. This way, people can browse this information comfortably without having to reinvent the wheel. And for that I designed a software-development-kit (SDK), so that researchers can write their analysis code against a piece of software that would ensure that their code runs the same locally on their computer and on the server, reducing the potential for bugs and issues, and allowing the maintainers of the infrastructure to change the location of the files easily without the researchers having to re-write their analysis code. That’s a specialty of the system I was designing.
All of these explanations merely scratch the surface of what I actually planned, and what is now live, which shows you just how incomprehensibly complex this all is. So instead of boring you with more details, let’s end this article with some surprising lessons I learned about designing entire systems.
Lessons Learned
The very first thing you may have already noticed is that I never planned for caching. Very quickly after the first components went online, it dawned upon me that we would have to somehow set up aggressive caching, lest the server will at some point buckle under the load of just a few dozen researchers browsing the data. So a first lesson I learned is that you will always forget something, and it will come back to bite you.
The next lesson is that you actually need software engineers (like, the studied type) to properly plan such a system. Yes, I can do some engineering myself, but there is so much I have missed both due to time constraints and because I, well, have no formal experience with this. Without the work done in Bonn that accidentally happened to align with our requirements, we probably would have nothing online by now.
The third lesson is that you really need an entire team. There are a lot of person-hours going into actually pulling this off, and the team we had was simply too small. It was essentially me plus two interns on 50% FTE positions. That was plainly too little. Suddenly, terms such as “product owners” that always sounded alien to me made sense: you really need one person per software component, and additionally some manager-type in between that facilitates communication. Otherwise, it is going to end in a huge mess. I did my best to create such a setup ad-hoc as it dawned upon me, but by the time it was slowly coming together, my time at LiU was coming to an end, and so the system remains to date only about 80% functional. It’s online, and we have authentication, but it is nowhere near a state where non-technical people could comfortably interact with the system.
This brings up a very memorable quote from a colleague, Anna Weichselbraun, has once said many years ago: the humanities never really arrived in the digital age, precisely because all of this knowledge eludes us. I didn’t study sociology because I fancied the vision of sitting for 40 hours a week in an office producing code.4 And the same counts for all my colleagues. While computational social scientists are getting better at writing code, our infrastructure, and the entire software we have at our disposal is hilariously ill-equipped for the type of work we actually need to get done. And what being in this position has really taught me is that designing the infrastructure we need to study our society, especially on social media, is an incredibly hard feat, and we as a discipline have a long way to go.
Final Thoughts
I am writing these lines as I just concluded my position at LiU. The system is mostly done but, as I said, nowhere near its final form. There is still a lot to do, and I will not be witness to the fruition of my labor.
However, as fate dictates, my new position demands of me exactly the same task. In my new project, I again have to design research data infrastructure whose requirements sound eerily similar to what I have just concluded. The only difference? Now I’m part of a much larger team, and we have three years, not just one, to make this happen. The system is also a bit more complex, but with all the mistakes that I have made in my previous position, I feel a bit better equipped to handle the task this time around.
Alas, it remains true what I said several years ago: if we are to reap the possibilities that the digital age promises us, computational social scientists must become much better engineers than we are, and truly marry sociology with software engineering. Otherwise, we will see this same difficult and strenuous process unfold over and over again.
Since this is probably going to be taken as a hot take, let me clarify what I mean in this footnote to not clutter the main argument. There are, in my experience, two types of vibe coders. One that does not care about engineering at all. Those folks merely prompt some LLM to code them up some dashboard but ignore anything that comes afterwards. Here, the goal is “something that appears to work.” Depending on the use-case, this can be fine, especially for personal use, but I wouldn’t call that engineering. That’s where the second type of vibe coder comes in: someone who does prompt some LLM to code up something, but who then spends considerable time on thinking about its UI, UX, and fixing bugs. In short, as long as someone “gives a damn” about what they do, I think there is an argument to be made that this is at least a form of engineering, even if it still deprives the vibe coder of the learning experience. ↩
A “single” (mono) piece of software that works as one. Oftentimes juxtaposed to “microservices,” where each individual function is its own app. ↩
As an interesting example that plagues the app right now: On UNIXoid systems (read: macOS and Linux) there is actually a limit to how many files can be open at the same time. This limit sits at around 10,000 file descriptors. How I know? Well, there’s a component in Zettlr that watches files for external changes, and it has a bug that causes it to open infinite file descriptors. As a user loads more than ca. 10,000 files into Zettlr, the app will just crash. This is one of these ways where I accidentally stumble upon new knowledge about the environment in which my code runs. But aside of this anecdote, the point remains that I never did that purposefully and systematically. ↩
There is this adage we used to tell ourselves in undergrad that we wouldn’t have studied a humanities subject if we were good at math. I always laughed with my friends, but that was a lie because I specialized in math, physics, and chemistry in school and was one of the best students in STEM type subjects. ↩