+ +

Why I dislike systemd

+ +

+As a Linux sysadmin in the 2010s, it's hard not to have an opinion on +systemd. But what I find baffling about it is how divisive it is; +nearly everyone (or at least the most vocal crowd) seems to either love +it or hate it. When I tell people that systemd was the catalyst for my +defection to OpenBSD last year, their usual reaction is to assume that +I am part of the "hate it" group. Nope. +

+ +

+In truth, systemd itself was a very small part of the reason I jumped +ship. Its introduction made me realise two important things. First, the +design problems with modern Linux run deeper than any one piece of +software, I just hadn't noticed until I had a fresh one to learn. +Second, and this is specific to Debian, the "universal operating +system" mantra is fundamentally flawed; you cannot support all use +cases when confronted with two alternatives that each function best +when adopted to the exclusion of all others. +

+ +

+But you didn't come here to read my life story. Back to systemd. +

+ +

+Let me be clear. systemd is not the work of the devil, it probably +doesn't contain any NSA backdoors, and it isn't the worst piece of +software the big Linux distributions are shipping by default. It does +address some real problems with Linux, and its approach does have its +merits. +

+ +

+It's often said that a half-truth is worse than a lie. In the case of +systemd, a half-improvement is worse than a complete flop. People are +focusing on the features they want to use and ignoring the ones that +will come back to bite them later. +

+ +

+No, my biggest problem with systemd is that it is repeating the +mistakes of System V all over again. It's just given them a change of +clothes to appeal to the 21st century. +

+ +

A bit of history

+ +

+In order to understand how we got here, we need to cast our minds (or, +for people under the age of 50 like myself, our imaginations) back to +the mid-1960s. Time-sharing operating systems were a new and innovative +concept, and many people still used batch processing as the established +method for getting things done. Many different organisations were +experimenting with many different approaches to time-sharing. One of +these was Multics. +

+ +

+Multics was the conceptual ancestor of all UNIX and UNIX-like systems. +The problems that made Multics a failure were not its core design +principles, but the design and implementation of its parts. This is +much more effectively communicated using a concrete example, from +a paper by Dennis Ritchie first presented in 1979. +

+ +

+
+The very convenient notation for IO redirection, using the `>â and +`<â characters, was not present from the very beginning of the PDP-7 +Unix system, but it did appear quite early. Like much else in Unix, it +was inspired by an idea from Multics. Multics has a rather general IO +redirection mechanism embodying named IO streams that can be +dynamically redirected to various devices, files, and even through +special stream-processing modules. Even in the version of Multics we +were familiar with a decade ago, there existed a command that switched +subsequent output normally destined for the terminal to a file, and +another command to reattach output to the terminal. Where under Unix +one might say +
+ +
ls >xx
+ +
+to get a listing of the names of oneâs files in xx, on Multics the +notation was +
+ +
iocall attach user_output file xx
+list
+iocall attach user_output syn user_i/o
+ +
+Even though this very clumsy sequence was used often during the Multics +days, and would have been utterly straightforward to integrate into the +Multics shell, the idea did not occur to us or anyone else at the time. +I speculate that the reason it did not was the sheer size of the +Multics project: the implementors of the IO system were at Bell Labs in +Murray Hill, while the shell was done at MIT. We didnât consider making +changes to the shell (it was their program); correspondingly, the +keepers of the shell may not even have known of the usefulness, albeit +clumsiness, of iocall. (The 1969 Multics manual lists iocall as an +`author-maintained,â that is non-standard, command.) Because both the +Unix IO system and its shell were under the exclusive control of +Thompson, when the right idea finally surfaced, it was a matter of an +hour or so to implement it. +
+

+ +

+What we have here is, fundamentally, a trade-off. Ritchie's speculation +on the reason for the trade-off aside, Multics tried to be everything +to everyone by providing the most general interface possible. As a +result, it was inefficient to develop and cumbersome to use. It got the +job done by making the 99% majority of use cases more difficult in +order to make the 1% minority simpler. This is a design pattern I think +is being repeated in systemd. +

+ +

+Most importantly for the point I'm making, an interface like iocall is +so obviously wrong to us because we have seen what a better +interface looks like. To the designers of Multics, it was a +perfectly reasonable approach to solving a very real problem. That is +not the same thing as being a good solution. +

+ +

The commercialisation of UNIX

+ +

+Fast-forward to the 1980s. AT&T wanted to commercialise UNIX, ended +the practice of distributing source code along with their binaries, and +began implementing features they thought would make their product sell. +The culmination of these efforts is the System V family of operating +systems, of which Linux is a member (albeit as a clone, not a direct +descendant). For the purposes of this discussion, I'll be referring to +System III as a member of the System V family, as the distinction isn't +too relevant. +

+ +

+One of the things System V inherited from Research UNIX was an init +that provided inflexible process monitoring, and invoked an +ever-growing shell script (/etc/rc) to perform service startup. Its +developers decided that the best way to tackle this problem was to +invent a new configuration language which would enable init to perform +more complex and customisable tasks, along with a way of allowing +vendors to drop in their own service startup routines and have them run +automatically at boot. +

+ +

+Sound familiar? It should. The fact that System V's init failed so +spectacularly at scaling to the problems of today, while modern BSD +systems continue to manage just fine by way of incremental improvements +to the pre-existing components, should serve as a lesson in itself. +

+ +

Introducing systemd

+ +

+Okay, enough with the history lesson already. You came here to read a +rant. So here it is. +

+ +

It shuffles complexity around

+ +

+A common point raised in favour of systemd is that its unit files are +"simpler" than init scripts. Superficially, this is true. What's really +going on, though, is that the complexity has been moved from the logic +into the language. +

+ +

+In stark contrast to Bourne shell, so many of systemd's unit directives +are highly specialised building blocks which aren't easily reusable. +This isn't necessarily a bad thing, except that because systemd units +aren't real programs, there's no way to escape to more basic primitives +as required. Inevitably, what this means is that more and more +directives will be added over time as more and more use cases are +discovered. +

+ +

+Don't believe me? Wait until 2025, and if the number of unit directives +hasn't at least tripled compared to today (and people aren't working +around their absence by invoking wrapper scripts from ExecStart), I'll +buy you a drink. Hint: At least one Debian package (memcached) is +already invoking a wrapper script from ExecStart. +

+ +

+If you start with complex logic in a simple language, you can always +simplify your implementation. If you start with simple logic in a +complex language, you're stuck with that complexity as soon as anyone +else starts using it. +

+ +

It babysits its users

+ +

+This complaint is far from unique to systemd, but is pandemic to a +disturbingly increasing number of programs, especially on Linux. Why is +systemctl edit even a thing? +

+ +

+No, you're just a user. You couldn't possibly manage +the job of invoking an editor on a configuration file without bricking +your computer, so here's a wrapper that does it for you. Oh, you edited +the file directly anyway? Here, let me helpfully tell you that +you should run systemctl daemon-reload, but perform the action +you requested using the old configuration anyway. +

+ +

+Speaking of which, why is systemctl daemon-reload even a +thing? When did filesystems become so uncool that not only are they +demoted to second-class citizens beneath some other primary source of +truth, but you don't even reload files when you know +they've been changed? +

+ +

+Now, I'm well aware that I'm giving an oversimplification of the issue +here. There are deeper design issues that make systemctl edit +quite different from opening an editor on a unit file, but my point is +that the user doesn't care. All they see is an overly +complex CLI that does little other than edit a file. However, this does +make a convenient segue into my next point... +

+ +

It takes the Multics approach to interoperability

+ +

+One of the fundamental differences between Multics and UNIX is the way +programs interact. Multics had programs that were developed +independently, and while they were each workable solutions that could +be made to complement each other when necessary, they were far from +efficient at doing so. UNIX turned this on its head, with one of its +most basic design principles to be to "write programs to work +together". +

+ +

+Now, systemd does rely heavily on D-Bus, and in doing so makes itself +controllable by other applications. You might consider that "working +together", but I see systemd as more akin to iocall than UNIX's shell +redirection; it provides facilities other programs can use without +integrating into their workflow. +

+ +

+The previous example is an ideal illustration of this. Using an editor +on unit files directly will work, but it's clumsy, and you need to +remember to invoke daemon-reload after you're done. Yes, it +works, but that's not the point. The point is that it's inefficient and +counterintuitive to edit unit files except through systemd's own +interfaces. That is not what I would call "working together". +

+ +

+I'll repeat what I said before, because it's important: To the +designers of Multics, it was a perfectly reasonable approach to solving +a very real problem. That is not the same thing as being a good +solution. +

+ +

It's unpredictable

+ +

+Log into a computer running systemd. Try to find the answer to "which +units are going to be started on next boot?". This is not quite the +same thing as "which units were started on this boot?", and because of +the complex dependency relationships between units, it's actually a +very difficult question to answer. This is bad if you're a +security-conscious sysadmin. +

+ +

+The failing of dependency-driven service initialisation is that it +works far better in theory than it does in practice. In theory, you +just have a directed graph of nodes, and you would like those depended +upon by default.target to start. In practice, that's not how systems +administration works. +

+ +

+As a sysadmin, when I reboot a system, I want to be able to ask "is foo +going to start up after a reboot?". I want to know, with reasonable +confidence, what the state of my system is going to be when it comes +onto the network. This is one area where sysvinit outperforms systemd +immeasurably. +

+ +

+And yes, you can avoid the whole issue just by masking any units you +don't want to start, but that's missing the point. Again, the point I'm +making isn't that it doesn't work, just that it doesn't work +efficiently. Your tools should make your job easier, not get in your +way. +

+ +

Its priorities are warped

+ +

+One of the "features" of systemd is that it allows you to boot a system +without needing a shell at all. This seems like such a senseless +manoeuvre that I can't help but think of it as a knee-jerk reaction to +the perception of Too Much Shell in sysv init scripts. +

+ +

+In exactly which universe is it reasonable to assume that you have a +running D-Bus service (or kdbus) and a filesystem containing unit +files, all the binaries they refer to, all the libraries they link +against, and all the configuration files any of them reference, but +that you lack that most ubiquitous of UNIX binaries, /bin/sh? +

+ +

+The use case often cited for this is managing services inside a +container. I don't see why the init on my desktop needs to be +complicated and restricted for the sake of a feature used by a minority +of people with specialised use cases. By all means, write a tool for +bootstrapping containers that doesn't rely on a shell, but don't +shoehorn that into a one-size-fits-all init. +

+ +

+What makes this especially annoying, though, is that systemd also +includes a dumbed-down shell-like parser to handle "EnvironmentFile"s +(which usually don't actually set environment variables when sourced +from a shell, but that's the way systemd's parser treats them). Also, +service units have a pseudo-shell syntax for argument lists. One +particularly bizarre feature that breaks expectations for users of +pretty much any other software is that $FOO is word-split into multiple +arguments, whereas ${FOO} isn't. +

+ +

+Is avoiding ever having to execute a binary really so useful in the +real world that these complications are justified? +

+ +

Conclusions

+ +

+I said it in my introduction, and I'll say it again: systemd's approach +has its merits. But its shortcomings are representative of a philosophy +that seems to be in fashion at the moment on Linux: ignoring history +and assuming we can do it better now, instead of familiarising +ourselves with historical mistakes and learning from them. +

+ +

+There is no one thing about it that makes it bad software, but taken as +a whole, I believe it creates more problems than it solves. Sadly, many +of those problems will take years to become apparent, by which time all +the major Linux distributions will have fully integrated it. That mess +will be left up to some poor soul in the 2040s to deal with, who will +hopefully learn from these mistakes and break the vicious cycle. +

+ +

+While this article was focused on systemd, it is really just one +example of an endemic trend on Linux, and across System V-family +systems in general. It's easy to look at an existing tool, identify its +deficiencies, write a replacement and label it as progress. It takes +considerably more skill to look at an existing tool, identify its +strong points, and modify it to work better. That is where Linux stands +to learn a lot from BSD. +

+ +

+If you think I'm full of shit, feel free to rant at me about it on +Twitter or something, but don't expect me to care. I've made my choice +to move to OpenBSD, and I couldn't care less whether systemd works well +for others or not. I'm simply documenting my experiences because I feel +there are important lessons to be learned which I'd rather not forget +about, and maybe others would like to read as well. +

+ +

Articles