Electron, chokidar, and native Node.js modules: A horror story from integration hell

Today, I want to tell a story. A story, whose beginning lies somewhere in the ancient past and which has followed me for years — until today it finally found its bloody conclusion. It is a story that has nothing to do with my research but in which I was emotionally invested so much that I just have to tell it, to spare you a similar fate as the one which entrenched myself for so long.

It is a story that began some time in 2018, led me through the fire and flames, gave me a few sleepless nights and where I unexpectedly found myself facing the same problems as one of the biggest software companies on the planet: Microsoft. It is a story that highlights the word “hell” in “integration hell” and that can demonstrate quite vividly the insane amount of monkey-patching and poking-in-the-dark that is sometimes required to make software work.

So let us embark on a journey, shall we?

The Early Beginnings

This story originates sometime in early 2018. To give some context: In summer of 2017 I was writing my Master thesis and was also experimenting with a new way to take reading notes and write more generally. Yes, I was captivated by the back then rather unknown trend of creating my own Zettelkasten and hence was experimenting with Markdown editors that (supposedly) make it very easy to do so.

Growing dissatisfied with the choice of editors, I started dabbling a little bit with writing my own editor (yes I know, but back then this statement made absolute sense in my head). I was trying out a brand-new framework that was just new on the market: Electron.

I realized that with Electron it was extremely easy to create a full-fledged program to write Markdown myself. So I pulled together a code editor and, after a few months of heavy development (I didn’t have too much to do after I finished my Master thesis), it was a perfectly usable editor and I switched to using it full-time.

However, as an editor for files, what I needed was an integration with the file system. My dream was to enable people to see a file tree with all their notes directly in the app, something I was sorely missing from Word and other similar programs. As I soon realized, those files needed to be updated even if someone changed a file outside of the application. It turned out a program doesn’t do anything unless you told it to – who could’ve thought that? So, how can you watch all the files that are displayed in a program and process any external change? This leads to a fateful decision.

Call to Adventure: Introducing chokidar

After a bit of googling, I found a module that was supposed to make all of that easy: chokidar. Now, chokidar is a small module that enables you to watch some directory on the computer for changes. As soon as a file or directory gets created, deleted, or changed in any other way, it would give you a notification so that you could then do whatever you needed to do to process that event. On February 14, 2018, I added that fateful module to the dependencies of my application. Normally, February 14 is a day to celebrate a wonderful relationship, not to shackle yourself to the abusive sh** that was about to go down.

A few hours of reading the documentation and integrating the module into Zettlr, and it could detect all those little external changes. I could do whatever I wanted to do to my files outside of Zettlr and the app would dutifully update the file tree to reflect those changes. Wonderful! But this peace of mind was not supposed to last very long.

A Surprising Finding

In the Apple ecosystem, the year 2018 was still firmly in the hands of Intel. Apple Silicon was still far away. This means that it wasn’t unheard of that the system fans of a MacBook Pro would start to spin if it had to work a little bit harder than usual. That was never something to worry about. Additionally, I had a 2012 MacBook Pro which was already a little bit on the older side of things, so when my fans started to spin while I was doing some heavy writing in Zettlr, I didn’t think too much about that. After all, I have read numerous reports that Electron is super heavy and resource-intensive, so that is to be expected. After all, you’re running a full browser on your system — right?

Well, all that was good, until I started to receive more and more reports by users who were using Zettlr on a MacBook and complained that using the app would set their computer on fire. So apparently it wasn’t to be expected after all. I already noticed that the heavy CPU consumption by Zettlr was possibly not normal as I switched to Microsoft’s Visual Studio Code for development – an app that also utilizes chokidar under the hood but didn’t come even close to the stress that my small Markdown editor put on my computer.

But I’m getting ahead of myself. Back then, I obviously didn’t know that the culprit was chokidar. The only thing I knew was: Electron apps could be relatively resource-friendly, only Zettlr wasn’t. Indeed, if you read the comments on the issue I linked above, not with a single word did I mention chokidar.

Identifying the Cause

At first I thought that the issue was more general: I just needed to improve the overall performance of the app; go through the whole code base a few times to optimize the code, and then it’ll all be good. After all, when I started developing Zettlr, I had close to no experience with JavaScript, so it was only natural to assume that the reason for the heavy resource-usage of Zettlr was my own inability to write good code.

Of course, that was not the reason. For the better part of two years, the app drained a ton of energy and it was basically impossible to use it without having the MacBook hooked to a power outlet. What I knew, however, is that the issue somehow only affected macOS. I never heard of any power consumption problems on Windows or Linux.

At some point, however, someone found the issue. I don’t know exactly when that was, but I remember that at some point, someone commented on an issue over on GitHub and mentioned that the power consumption goes up exponentially with the amount of files loaded into the app. The power drain was related to the amount of files being loaded in the app. And the amount of files being loaded equals the amount of files being watched. I re-read the documentation of chokidar and saw that it utilizes something called fsevents on macOS but, if that is not available, it fell back to polling the file system continuously which could increase the CPU utilization and, by extension, drain the battery and put the computer on fire.

As the documentation told me:

usePolling (default: false). Whether to use fs.watchFile (backed by polling), or fs.watch. If polling leads to high CPU utilization, consider setting this to false.

Alright, so: polling leads to high CPU utilization, so we should avoid this at all costs. However, why did chokidar use polling in the first place on macOS? I double checked and, no: fsevents was in my node_modules folder and I didn’t receive any error message, so it was definitely being used.

Monkey Patching with the largest software company on earth

After months of intermittently thinking about the problem, I chose to turn to the code base of Visual Studio Code. After all, it was being developed by Microsoft, the largest software company of the world, and it also uses chokidar. If someone knew how to properly write software, they should know it, right? So I looked up how VSCode was using chokidar. And, to my surprise, I found this piece of code:¹

const watcherOpts: chokidar.WatchOptions = {
    ignoreInitial: true,
    ignorePermissionErrors: true,
    followSymlinks: true, // this is the default of chokidar and supports file events through symlinks
    interval: pollingInterval, // while not used in normal cases, if any error causes chokidar to fallback to polling, increase its intervals
    binaryInterval: pollingInterval,
    usePolling: usePolling,
    disableGlobbing: true // fix https://github.com/microsoft/vscode/issues/4586
};

So even Microsoft was dumbfounded as to what was happening! Instead of properly utilizing fsevents on macOS, they simply decreased the speed of polling from “always” to “every 5 seconds”, which reduced the CPU utilization and “fixed” that problem for them.

“Alright,” I thought, “let’s just implement polling and call it a day.” So I went, implemented a setting that allowed users to adjust the polling parameters and called it a day. That was on November 17, 2020.

But at the back of my head, I knew that this was not the ideal solution. It was just a monkey patch. It may be used by the largest software company on earth, but it is still not “how it’s supposed to work”. But, having no better idea, I couldn’t do much about it. But my brain continued to think about this issue for almost two years.

At this point, we are nearing the conclusion of this tale, but we first need to introduce an unexpected companion who crossed my path just yesterday that proved to be the key to solving the issues that ailed Zettlr for almost four years at this point.

An Unexpected Companion

A completely unrelated issue that the app was dealing with for the past four years was spellchecking. Spellchecking is supposed to be easy: Just take a word, compare it to a list of words marked as “correct” and underline it with a small, squiggly red line if that word is not in the list. Easy, right?

The problem is that the most comprehensive spellchecker around is Hunspell and most dictionaries are in Hunspell’s format. So if you don’t want to generate dictionaries yourself, you’re stuck with Hunspell. And Hunspell is not written in JavaScript, but C++. So I’m at a loss, right? Well, not quite.

There is another package called nspell that I used since February 1, 2019. It is a JavaScript-only adaptation of Hunspell, albeit with less functionality. What it could do was check the spelling of a word and provide suggestions. So that was perfectly fine. I installed the package, and began shipping Zettlr with it.

It soon turned out, however, that nspell has one extremely problematic limitation: Larger and more complex dictionaries simply do not work and make the whole application crash. This applied specifically to the Portuguese and Italian dictionaries. So one of my long-term goals was to replace nspell with something else. But nothing came across the corner. It was either nspell or nothing.

Yesterday I started another look into potential alternatives. First, I thought that I could just ship the Hunspell binary and use it in a similar fashion as Zettlr uses Pandoc: Start it as a separate process, feed it words and parse the output of the program. But that felt hacky to me.

But then I saw Nodehun in the search results again. I knew Nodehun from earlier searches and I knew that it was out there, but I never really thought about using it. But yesterday was the day that I decided to just give it a try. After all, what Nodehun does is it actually bundles the real Hunspell binary and offers an API for Node.js that I could use. The API it offers looks very similar to nspell so I realized that I should just get rid of my fear of native extensions (*cough* fsevents *cough*) and give it a try.

Descend into hell

So I followed the usual procedure of adding a new module to Zettlr and exchanged nspell with Nodehun. Easy! I started the app in testing mode, and, lo and behold: it worked! The next step was to quickly check the problematic dictionaries, and to my satisfaction I discovered that Nodehun had no problems whatsoever to load and utilize them. Wonderful! So we can finally put this issue to rest, right?

Just to double check, I quickly bundled the app together to verify that everything works fine even when the app will be distributed to users, launched it, and … crash. “Module nodehun was not found.” What?

I knew that the Nodehun library worked flawlessly as I had verified in testing, but it somehow wasn’t bundled with the app when I created the actual program that users could run. I suspected that the problem was with my build process, which utilizes webpack.

After two hours of trial and error and double checking with the other library I had, fsevents (which was indeed correctly included in my app bundle), I found the problem: The Nodehun package includes the library without the extension that native Node.js modules have: .node. The problem is that webpack (and other bundlers) try to minimize the amount of code and therefore just put all the code into a single file. This works well for JavaScript, but Nodehun was a binary file, so you can’t add that into some JavaScript file.

After including a reference to the library itself instead of referencing the package name (import 'Nodehun.node' instead of import 'nodehun'), webpack recognized that it was not a simple JavaScript file but a native Node module and did what it had to do to include the library in the bundle.

Wonderful! So let’s build again, start it, and … crash. WHAT?

Now the library has been found, but I received a weird issue that I have never seen before: macOS complained that the library hasn’t been code signed. I suspected something fishy was going on.

Fast forward another couple hours and I noticed that, in order to actually code-sign the library, I needed to adapt my build configuration. I realized that everything worked flawlessly when I bundled the app without its ASAR package (which is basically a glorified ZIP-file that includes all the code) but broke when I bundled it with an ASAR package.

I then realized that I needed to direct the build process to not include those dynamic node libraries in the ASAR package by providing an option called unpack. Then, it worked flawlessly — both during testing as well as with the final, bundled app. Heureka!

However, remember that this article is not about spell checking, but about chokidar?

A few years ago, once I found out that chokidar was using polling instead of fsevents to watch files, I fiddled around a little bit with the code and noticed that one can actually check whether chokidar actually uses fsevents or not. I didn’t know how to solve the issue I had, but I added a line of code that would always log whether chokidar uses fsevents or fell back to the CPU-intensive polling. Ever since, whenever I started the app and looked into the logs, I could see the same disappointing message: “chokidar falls back to CPU polling.” This was incredibly frustrating, especially since during development it was actually using fsevents. Only in the actual app it didn’t work.

But this morning, once I got Nodehun to work, I opened the logs again. And I saw something. For the first time in four years, a new message was written to the log: “chokidar is utilizing fsevents.”

I was baffled. In fact, I was so excited to see that message that I have been hoping to see for four years that I – and I kid you not – literally jumped off the chair and danced through the flat. It felt as if a large weight was just taken off my back. For the first time in four years, everything just worked as it was supposed to.

And you know what the reason was? Since I added the unpack option, not only did that cause Nodehun to be code-signed. At the same time, the fsevents library was also signed. For the first time in four years.

Pulling it All Together

Now, after one hell of a ride, I was ready to finally understand how native modules work with Electron. So here’s what I figured out after going through integration hell:

When you install a Node module that includes source code for a dynamic library, a.k.a. a .node-file, the library will automatically be compiled from source upon install – for the correct architecture and the correct operating system. Since Electron works a little bit different than Node.js itself, it utilizes electron-rebuild for that rather than the node-gyp-package. But, if you use Electron forge as I do, that module will be already installed alongside Electron forge and it will work out of the box.

At this point, you will have the compiled library sitting inside your node_modules folder. Now, when you start the app in development mode, webpack will compile all your source code. Since fsevents is included with its filename extension (somewhere in the chokidar library there is a statement called require('fsevents.node')), webpack already knows that it needs to copy that file over manually without including it in the JavaScript file. For this, it uses a plugin called node-loader. (Electron forge additionally needs a plugin called asset-relocator, but that is described sufficiently in the documentation.)

However, Nodehun was not included with its filename extension, so webpack didn’t know what to do with it. During development, this was not a problem because of the way Node.js searches for modules. Webpack simply included the statement require('nodehun'), and what Node.js then did was it searched for a match. This includes having a look inside the node_modules folder where it then found the correct module and loaded the .node-file without any issue.

When you build an app, however, the node_modules folder will not be included (because all the JavaScript code will be dumped into a single file), so while the fsevents.node library was still present in the app bundle (as webpack knew what to do with it), the Nodehun library was not. This time, when Node.js saw the statement require('nodehun'), it did not find a node_modules folder and thus could not locate the library. After specifying the filename extension, this problem was solved (because webpack saw that it had to treat Nodehun analogously to fsevents) and both Nodehun.node as well as fsevents.node were included in the final application.

But this still didn’t explain the code signing issue. What then dawned upon me is that, whenever I start the app in development mode, I start it directly from the command line. And, when you start a program from the command line, macOS does not require that program to be signed (probably because if you are capable of using the command line you can be assumed to know enough about computers not to be scammed?). That explains why chokidar used the fsevents library during development: It was actually loaded without a problem.

However, as soon as you include .node-modules with a regular app bundle, you need to code sign those libraries as well. But that doesn’t work out of the box. Electron forge uses a module called osx-sign under the hood to actually perform the code-signing. osx-sign will automatically code-sign every file that is included as-is with the application, but it will skip any file that is going to end up in the app.asar-file. (Remember, that is basically just a ZIP-file with all the JavaScript code.)

So when I disabled the asar-option, osx-sign saw that there were two additional .node-files and it proceeded to sign these as well without any issue. In order to make osx-sign to sign these files even with the asar-option activated, I had to add an exception to the build configuration to exclude .node-files from the final app.asar file.

Funny enough: Even when you do so, the .node-files still end up within the app.asar-file, but now at least they’ll be signed. Don’t ask me why that is.

Anyway, at this point you will end up with your .node-libraries working both within development and within a normal application package. And chokidar won’t fall back to polling, even on macOS.

The main reason why this remained obscure for almost four years (!) is that since chokidar wants to also work when fsevents is not available, it provided a fallback mechanism. In other words: When loading the library fails out of any reason, it silently swallows that error message and simply switches to CPU polling.

For four years, fsevents simply wasn’t code-signed, and I didn’t know because I never saw that error until I added a second native module that was required with no fallback mechanism.

Conclusion

The documentation of Electron Forge, which I use to develop Zettlr, boldly states:

If you used the Webpack or TypeScript + Webpack templates to create your application, native modules will mostly work out of the box.

Which could not be farther from the truth. The same documentation also states, referring to another, seemingly unrelated plugin:

This plugin will automatically add all native Node modules in your node_modules folder to the asar.unpack config option in your packagerConfig. If your app uses native Node modules, you should probably use this to reduce loading times and disk consumption on your users' machines.

Reading this, it appears as if adding native Node modules to this unpack key was optional and just a convenience. But this is not true: You must always, without any exception add Node modules to the unpack option. Otherwise, they will not be code-signed and hence will never work.

In the end, including native modules to an Electron app is relatively straight forward – if you know what is required to do so. If you are facing a similar problem as I did here, always make sure to fulfill the following two steps apart from reading the official documentation: First, if you use some bundler such as webpack, add the corresponding plugins to handle .node-files and also ensure they get processed by requiring them with their filename extension. And second, always add the following option to your build process to ensure these files get code-signed: asar: { unpack: '*.{node,dll}' }.

This is how integration hell looks like. And that even Microsoft could not pinpoint the same exact issue I was having for several years proves that nobody is safe from it – not even the largest software company on earth.