As a Linux sysadmin in the 2010s, it's hard not to have an opinion on systemd. But what I find baffling about it is how divisive it is; nearly everyone (or at least the most vocal crowd) seems to either love it or hate it. When I tell people that systemd was the catalyst for my defection to OpenBSD last year, their usual reaction is to assume that I am part of the "hate it" group. Nope.
In truth, systemd itself was a very small part of the reason I jumped ship. Its introduction made me realise two important things. First, the design problems with modern Linux run deeper than any one piece of software, I just hadn't noticed until I had a fresh one to learn. Second, and this is specific to Debian, the "universal operating system" mantra is fundamentally flawed; you cannot support all use cases when confronted with two alternatives that each function best when adopted to the exclusion of all others.
But you didn't come here to read my life story. Back to systemd.
Let me be clear. systemd is not the work of the devil, it probably doesn't contain any NSA backdoors, and it isn't the worst piece of software the big Linux distributions are shipping by default. It does address some real problems with Linux, and its approach does have its merits.
It's often said that a half-truth is worse than a lie. In the case of systemd, a half-improvement is worse than a complete flop. People are focusing on the features they want to use and ignoring the ones that will come back to bite them later.
No, my biggest problem with systemd is that it is repeating the mistakes of System V all over again. It's just given them a change of clothes to appeal to the 21st century.
In order to understand how we got here, we need to cast our minds (or, for people under the age of 50 like myself, our imaginations) back to the mid-1960s. Time-sharing operating systems were a new and innovative concept, and many people still used batch processing as the established method for getting things done. Many different organisations were experimenting with many different approaches to time-sharing. One of these was Multics.
Multics was the conceptual ancestor of all UNIX and UNIX-like systems. The problems that made Multics a failure were not its core design principles, but the design and implementation of its parts. This is much more effectively communicated using a concrete example, from a paper by Dennis Ritchie first presented in 1979.
The very convenient notation for IO redirection, using the `>’ and `<‘ characters, was not present from the very beginning of the PDP-7 Unix system, but it did appear quite early. Like much else in Unix, it was inspired by an idea from Multics. Multics has a rather general IO redirection mechanism embodying named IO streams that can be dynamically redirected to various devices, files, and even through special stream-processing modules. Even in the version of Multics we were familiar with a decade ago, there existed a command that switched subsequent output normally destined for the terminal to a file, and another command to reattach output to the terminal. Where under Unix one might say
ls >xxto get a listing of the names of one’s files in xx, on Multics the notation was
iocall attach user_output file xx list iocall attach user_output syn user_i/oEven though this very clumsy sequence was used often during the Multics days, and would have been utterly straightforward to integrate into the Multics shell, the idea did not occur to us or anyone else at the time. I speculate that the reason it did not was the sheer size of the Multics project: the implementors of the IO system were at Bell Labs in Murray Hill, while the shell was done at MIT. We didn’t consider making changes to the shell (it was their program); correspondingly, the keepers of the shell may not even have known of the usefulness, albeit clumsiness, of iocall. (The 1969 Multics manual lists iocall as an `author-maintained,’ that is non-standard, command.) Because both the Unix IO system and its shell were under the exclusive control of Thompson, when the right idea finally surfaced, it was a matter of an hour or so to implement it.
What we have here is, fundamentally, a trade-off. Ritchie's speculation on the reason for the trade-off aside, Multics tried to be everything to everyone by providing the most general interface possible. As a result, it was inefficient to develop and cumbersome to use. It got the job done by making the 99% majority of use cases more difficult in order to make the 1% minority simpler. This is a design pattern I think is being repeated in systemd.
Most importantly for the point I'm making, an interface like iocall is so obviously wrong to us because we have seen what a better interface looks like. To the designers of Multics, it was a perfectly reasonable approach to solving a very real problem. That is not the same thing as being a good solution.
Fast-forward to the 1980s. AT&T wanted to commercialise UNIX, ended the practice of distributing source code along with their binaries, and began implementing features they thought would make their product sell. The culmination of these efforts is the System V family of operating systems, of which Linux is a member (albeit as a clone, not a direct descendant). For the purposes of this discussion, I'll be referring to System III as a member of the System V family, as the distinction isn't too relevant.
One of the things System V inherited from Research UNIX was an init that provided inflexible process monitoring, and invoked an ever-growing shell script (/etc/rc) to perform service startup. Its developers decided that the best way to tackle this problem was to invent a new configuration language which would enable init to perform more complex and customisable tasks, along with a way of allowing vendors to drop in their own service startup routines and have them run automatically at boot.
Sound familiar? It should. The fact that System V's init failed so spectacularly at scaling to the problems of today, while modern BSD systems continue to manage just fine by way of incremental improvements to the pre-existing components, should serve as a lesson in itself.
Okay, enough with the history lesson already. You came here to read a rant. So here it is.
A common point raised in favour of systemd is that its unit files are "simpler" than init scripts. Superficially, this is true. What's really going on, though, is that the complexity has been moved from the logic into the language.
In stark contrast to Bourne shell, so many of systemd's unit directives are highly specialised building blocks which aren't easily reusable. This isn't necessarily a bad thing, except that because systemd units aren't real programs, there's no way to escape to more basic primitives as required. Inevitably, what this means is that more and more directives will be added over time as more and more use cases are discovered.
Don't believe me? Wait until 2025, and if the number of unit directives hasn't at least tripled compared to today (and people aren't working around their absence by invoking wrapper scripts from ExecStart), I'll buy you a drink. Hint: At least one Debian package (memcached) is already invoking a wrapper script from ExecStart.
If you start with complex logic in a simple language, you can always simplify your implementation. If you start with simple logic in a complex language, you're stuck with that complexity as soon as anyone else starts using it.
This complaint is far from unique to systemd, but is pandemic to a disturbingly increasing number of programs, especially on Linux. Why is systemctl edit even a thing?
No, you're just a user. You couldn't possibly manage the job of invoking an editor on a configuration file without bricking your computer, so here's a wrapper that does it for you. Oh, you edited the file directly anyway? Here, let me helpfully tell you that you should run systemctl daemon-reload, but perform the action you requested using the old configuration anyway.
Speaking of which, why is systemctl daemon-reload even a thing? When did filesystems become so uncool that not only are they demoted to second-class citizens beneath some other primary source of truth, but you don't even reload files when you know they've been changed?
Now, I'm well aware that I'm giving an oversimplification of the issue here. There are deeper design issues that make systemctl edit quite different from opening an editor on a unit file, but my point is that the user doesn't care. All they see is an overly complex CLI that does little other than edit a file. However, this does make a convenient segue into my next point...
One of the fundamental differences between Multics and UNIX is the way programs interact. Multics had programs that were developed independently, and while they were each workable solutions that could be made to complement each other when necessary, they were far from efficient at doing so. UNIX turned this on its head, with one of its most basic design principles to be to "write programs to work together".
Now, systemd does rely heavily on D-Bus, and in doing so makes itself controllable by other applications. You might consider that "working together", but I see systemd as more akin to iocall than UNIX's shell redirection; it provides facilities other programs can use without integrating into their workflow.
The previous example is an ideal illustration of this. Using an editor on unit files directly will work, but it's clumsy, and you need to remember to invoke daemon-reload after you're done. Yes, it works, but that's not the point. The point is that it's inefficient and counterintuitive to edit unit files except through systemd's own interfaces. That is not what I would call "working together".
I'll repeat what I said before, because it's important: To the designers of Multics, it was a perfectly reasonable approach to solving a very real problem. That is not the same thing as being a good solution.
Log into a computer running systemd. Try to find the answer to "which units are going to be started on next boot?". This is not quite the same thing as "which units were started on this boot?", and because of the complex dependency relationships between units, it's actually a very difficult question to answer. This is bad if you're a security-conscious sysadmin.
The failing of dependency-driven service initialisation is that it works far better in theory than it does in practice. In theory, you just have a directed graph of nodes, and you would like those depended upon by default.target to start. In practice, that's not how systems administration works.
As a sysadmin, when I reboot a system, I want to be able to ask "is foo going to start up after a reboot?". I want to know, with reasonable confidence, what the state of my system is going to be when it comes onto the network. This is one area where sysvinit outperforms systemd immeasurably.
And yes, you can avoid the whole issue just by masking any units you don't want to start, but that's missing the point. Again, the point I'm making isn't that it doesn't work, just that it doesn't work efficiently. Your tools should make your job easier, not get in your way.
One of the "features" of systemd is that it allows you to boot a system without needing a shell at all. This seems like such a senseless manoeuvre that I can't help but think of it as a knee-jerk reaction to the perception of Too Much Shell in sysv init scripts.
In exactly which universe is it reasonable to assume that you have a running D-Bus service (or kdbus) and a filesystem containing unit files, all the binaries they refer to, all the libraries they link against, and all the configuration files any of them reference, but that you lack that most ubiquitous of UNIX binaries, /bin/sh?
The use case often cited for this is managing services inside a container. I don't see why the init on my desktop needs to be complicated and restricted for the sake of a feature used by a minority of people with specialised use cases. By all means, write a tool for bootstrapping containers that doesn't rely on a shell, but don't shoehorn that into a one-size-fits-all init.
What makes this especially annoying, though, is that systemd also includes a dumbed-down shell-like parser to handle "EnvironmentFile"s (which usually don't actually set environment variables when sourced from a shell, but that's the way systemd's parser treats them). Also, service units have a pseudo-shell syntax for argument lists. One particularly bizarre feature that breaks expectations for users of pretty much any other software is that $FOO is word-split into multiple arguments, whereas ${FOO} isn't.
Is avoiding ever having to execute a binary really so useful in the real world that these complications are justified?
I said it in my introduction, and I'll say it again: systemd's approach has its merits. But its shortcomings are representative of a philosophy that seems to be in fashion at the moment on Linux: ignoring history and assuming we can do it better now, instead of familiarising ourselves with historical mistakes and learning from them.
There is no one thing about it that makes it bad software, but taken as a whole, I believe it creates more problems than it solves. Sadly, many of those problems will take years to become apparent, by which time all the major Linux distributions will have fully integrated it. That mess will be left up to some poor soul in the 2040s to deal with, who will hopefully learn from these mistakes and break the vicious cycle.
While this article was focused on systemd, it is really just one example of an endemic trend on Linux, and across System V-family systems in general. It's easy to look at an existing tool, identify its deficiencies, write a replacement and label it as progress. It takes considerably more skill to look at an existing tool, identify its strong points, and modify it to work better. That is where Linux stands to learn a lot from BSD.
If you think I'm full of shit, feel free to rant at me about it on Twitter or something, but don't expect me to care. I've made my choice to move to OpenBSD, and I couldn't care less whether systemd works well for others or not. I'm simply documenting my experiences because I feel there are important lessons to be learned which I'd rather not forget about, and maybe others would like to read as well.