1 <!--#include virtual="/doctype.txt"-->
4 <!--#set var="pagetitle" value="Why I dislike systemd"-->
5 <!--#include virtual="/head.shtml"-->
9 <!--#include virtual="/header.shtml"-->
11 <!--#include virtual="/navigation.shtml"-->
15 <h1>Why I dislike systemd</h1>
17 <p><em>(Published 2015-06-14)</em></p>
20 As a Linux sysadmin in the 2010s, it's hard not to have an opinion on
21 systemd. But what I find baffling about it is how divisive it is;
22 nearly everyone (or at least the most vocal crowd) seems to either love
23 it or hate it. When I tell people that systemd was the catalyst for my
24 defection to OpenBSD last year, their usual reaction is to assume that
25 I am part of the "hate it" group. Nope.
29 In truth, systemd itself was a very small part of the reason I jumped
30 ship. Its introduction made me realise two important things. First, the
31 design problems with modern Linux run deeper than any one piece of
32 software, I just hadn't noticed until I had a fresh one to learn.
33 Second, and this is specific to Debian, the "universal operating
34 system" mantra is fundamentally flawed; you cannot support all use
35 cases when confronted with two alternatives that each function best
36 when adopted to the exclusion of all others.
40 But you didn't come here to read my life story. Back to systemd.
44 Let me be clear. systemd is not the work of the devil, it probably
45 doesn't contain any NSA backdoors, and it isn't the worst piece of
46 software the big Linux distributions are shipping by default. It does
47 address some real problems with Linux, and its approach does have its
52 It's often said that a half-truth is worse than a lie. In the case of
53 systemd, a half-improvement is worse than a complete flop. People are
54 focusing on the features they want to use and ignoring the ones that
55 will come back to bite them later.
59 No, my biggest problem with systemd is that it is repeating the
60 mistakes of System V all over again. It's just given them a change of
61 clothes to appeal to the 21st century.
64 <h2>A bit of history</h2>
67 In order to understand how we got here, we need to cast our minds (or,
68 for people under the age of 50 like myself, our imaginations) back to
69 the mid-1960s. Time-sharing operating systems were a new and innovative
70 concept, and many people still used batch processing as the established
71 method for getting things done. Many different organisations were
72 experimenting with many different approaches to time-sharing. One of
77 Multics was the conceptual ancestor of all UNIX and UNIX-like systems.
78 The problems that made Multics a failure were not its core design
79 principles, but the design and implementation of its parts. This is
80 much more effectively communicated using a concrete example, from <a
81 href="http://www.utterpower.com/10132011-dennis-ritchie-bell-labs-rip-c-language-and-unix/">
82 a paper</a> by Dennis Ritchie first presented in 1979.
86 cite="http://www.utterpower.com/10132011-dennis-ritchie-bell-labs-rip-c-language-and-unix/">
88 The very convenient notation for IO redirection, using the `>’ and
89 `<‘ characters, was not present from the very beginning of the PDP-7
90 Unix system, but it did appear quite early. Like much else in Unix, it
91 was inspired by an idea from Multics. Multics has a rather general IO
92 redirection mechanism embodying named IO streams that can be
93 dynamically redirected to various devices, files, and even through
94 special stream-processing modules. Even in the version of Multics we
95 were familiar with a decade ago, there existed a command that switched
96 subsequent output normally destined for the terminal to a file, and
97 another command to reattach output to the terminal. Where under Unix
101 <pre><code>ls >xx</code></pre>
104 to get a listing of the names of one’s files in xx, on Multics the
108 <pre><code>iocall attach user_output file xx
110 iocall attach user_output syn user_i/o</code></pre>
113 Even though this very clumsy sequence was used often during the Multics
114 days, and would have been utterly straightforward to integrate into the
115 Multics shell, the idea did not occur to us or anyone else at the time.
116 I speculate that the reason it did not was the sheer size of the
117 Multics project: the implementors of the IO system were at Bell Labs in
118 Murray Hill, while the shell was done at MIT. We didn’t consider making
119 changes to the shell (it was their program); correspondingly, the
120 keepers of the shell may not even have known of the usefulness, albeit
121 clumsiness, of iocall. (The 1969 Multics manual lists iocall as an
122 `author-maintained,’ that is non-standard, command.) Because both the
123 Unix IO system and its shell were under the exclusive control of
124 Thompson, when the right idea finally surfaced, it was a matter of an
125 hour or so to implement it.
130 What we have here is, fundamentally, a trade-off. Ritchie's speculation
131 on the reason for the trade-off aside, Multics tried to be everything
132 to everyone by providing the most general interface possible. As a
133 result, it was inefficient to develop and cumbersome to use. It got the
134 job done by making the 99% majority of use cases more difficult in
135 order to make the 1% minority simpler. This is a design pattern I think
136 is being repeated in systemd.
140 Most importantly for the point I'm making, an interface like iocall is
141 so obviously wrong to us <strong>because we have seen what a better
142 interface looks like</strong>. To the designers of Multics, it was a
143 perfectly reasonable approach to solving a very real problem. That is
144 not the same thing as being a good solution.
147 <h2>The commercialisation of UNIX</h2>
150 Fast-forward to the 1980s. AT&T wanted to commercialise UNIX, ended
151 the practice of distributing source code along with their binaries, and
152 began implementing features they thought would make their product sell.
153 The culmination of these efforts is the System V family of operating
154 systems, of which Linux is a member (albeit as a clone, not a direct
155 descendant). For the purposes of this discussion, I'll be referring to
156 System III as a member of the System V family, as the distinction isn't
161 One of the things System V inherited from Research UNIX was an init
162 that provided inflexible process monitoring, and invoked an
163 ever-growing shell script (/etc/rc) to perform service startup. Its
164 developers decided that the best way to tackle this problem was to
165 invent a new configuration language which would enable init to perform
166 more complex and customisable tasks, along with a way of allowing
167 vendors to drop in their own service startup routines and have them run
168 automatically at boot.
172 Sound familiar? It should. The fact that System V's init failed so
173 spectacularly at scaling to the problems of today, while modern BSD
174 systems continue to manage just fine by way of incremental improvements
175 to the pre-existing components, should serve as a lesson in itself.
178 <h2>Introducing systemd</h2>
181 Okay, enough with the history lesson already. You came here to read a
185 <h3>It shuffles complexity around</h3>
188 A common point raised in favour of systemd is that its unit files are
189 "simpler" than init scripts. Superficially, this is true. What's really
190 going on, though, is that the complexity has been moved from the logic
195 In stark contrast to Bourne shell, so many of systemd's unit directives
196 are highly specialised building blocks which aren't easily reusable.
197 This isn't necessarily a bad thing, except that because systemd units
198 aren't real programs, there's no way to escape to more basic primitives
199 as required. Inevitably, what this means is that more and more
200 directives will be added over time as more and more use cases are
205 Don't believe me? Wait until 2025, and if the number of unit directives
206 hasn't at least tripled compared to today (and people aren't working
207 around their absence by invoking wrapper scripts from ExecStart), I'll
208 buy you a drink. Hint: At least one Debian package (memcached) is
209 already invoking a wrapper script from ExecStart.
213 If you start with complex logic in a simple language, you can always
214 simplify your implementation. If you start with simple logic in a
215 complex language, you're stuck with that complexity as soon as anyone
216 else starts using it.
219 <h3>It babysits its users</h3>
222 This complaint is far from unique to systemd, but is pandemic to a
223 disturbingly increasing number of programs, especially on Linux. Why is
224 <tt>systemctl edit</tt> even a thing?
228 No, you're just a <strong>user</strong>. You couldn't possibly manage
229 the job of invoking an editor on a configuration file without bricking
230 your computer, so here's a wrapper that does it for you. Oh, you edited
231 the file directly anyway? Here, let me helpfully <strong>tell you that
232 you should run <tt>systemctl daemon-reload</tt>, but perform the action
233 you requested using the old configuration anyway</strong>.
237 Speaking of which, why is <tt>systemctl daemon-reload</tt> even a
238 thing? When did filesystems become so uncool that not only are they
239 demoted to second-class citizens beneath some other primary source of
240 truth, but you don't even reload files when you <strong>know</strong>
241 they've been changed?
245 Now, I'm well aware that I'm giving an oversimplification of the issue
246 here. There are deeper design issues that make <tt>systemctl edit</tt>
247 quite different from opening an editor on a unit file, but my point is
248 that <strong>the user doesn't care</strong>. All they see is an overly
249 complex CLI that does little other than edit a file. However, this does
250 make a convenient segue into my next point...
253 <h3>It takes the Multics approach to interoperability</h3>
256 One of the fundamental differences between Multics and UNIX is the way
257 programs interact. Multics had programs that were developed
258 independently, and while they were each workable solutions that could
259 be made to complement each other when necessary, they were far from
260 efficient at doing so. UNIX turned this on its head, with one of its
261 most basic design principles to be to "write programs to work
266 Now, systemd does rely heavily on D-Bus, and in doing so makes itself
267 controllable by other applications. You might consider that "working
268 together", but I see systemd as more akin to iocall than UNIX's shell
269 redirection; it provides facilities other programs can use without
270 integrating into their workflow.
274 The previous example is an ideal illustration of this. Using an editor
275 on unit files directly will work, but it's clumsy, and you need to
276 remember to invoke <tt>daemon-reload</tt> after you're done. Yes, it
277 works, but that's not the point. The point is that it's inefficient and
278 counterintuitive to edit unit files except through systemd's own
279 interfaces. That is not what I would call "working together".
283 I'll repeat what I said before, because it's important: To the
284 designers of Multics, it was a perfectly reasonable approach to solving
285 a very real problem. That is not the same thing as being a good
289 <h3>It's unpredictable</h3>
292 Log into a computer running systemd. Try to find the answer to "which
293 units are going to be started on next boot?". This is not quite the
294 same thing as "which units were started on this boot?", and because of
295 the complex dependency relationships between units, it's actually a
296 very difficult question to answer. This is bad if you're a
297 security-conscious sysadmin.
301 The failing of dependency-driven service initialisation is that it
302 works far better in theory than it does in practice. In theory, you
303 just have a directed graph of nodes, and you would like those depended
304 upon by default.target to start. In practice, that's not how systems
305 administration works.
309 As a sysadmin, when I reboot a system, I want to be able to ask "is foo
310 going to start up after a reboot?". I want to know, with reasonable
311 confidence, what the state of my system is going to be when it comes
312 onto the network. This is one area where sysvinit outperforms systemd
317 And yes, you can avoid the whole issue just by masking any units you
318 don't want to start, but that's missing the point. Again, the point I'm
319 making isn't that it doesn't work, just that it doesn't work
320 efficiently. Your tools should make your job easier, not get in your
324 <h3>Its priorities are warped</h3>
327 One of the "features" of systemd is that it allows you to boot a system
328 without needing a shell at all. This seems like such a senseless
329 manoeuvre that I can't help but think of it as a knee-jerk reaction to
330 the perception of Too Much Shell in sysv init scripts.
334 In exactly which universe is it reasonable to assume that you have a
335 running D-Bus service (or kdbus) and a filesystem containing unit
336 files, all the binaries they refer to, all the libraries they link
337 against, and all the configuration files any of them reference, but
338 that you lack that most ubiquitous of UNIX binaries, /bin/sh?
342 The use case often cited for this is managing services inside a
343 container. I don't see why the init on my desktop needs to be
344 complicated and restricted for the sake of a feature used by a minority
345 of people with specialised use cases. By all means, write a tool for
346 bootstrapping containers that doesn't rely on a shell, but don't
347 shoehorn that into a one-size-fits-all init.
351 What makes this especially annoying, though, is that systemd also
352 includes a dumbed-down shell-like parser to handle "EnvironmentFile"s
353 (which usually don't actually set environment variables when sourced
354 from a shell, but that's the way systemd's parser treats them). Also,
355 service units have a pseudo-shell syntax for argument lists. One
356 particularly bizarre feature that breaks expectations for users of
357 pretty much any other software is that $FOO is word-split into multiple
358 arguments, whereas ${FOO} isn't.
362 Is avoiding ever having to execute a binary really so useful in the
363 real world that these complications are justified?
369 I said it in my introduction, and I'll say it again: systemd's approach
370 has its merits. But its shortcomings are representative of a philosophy
371 that seems to be in fashion at the moment on Linux: ignoring history
372 and assuming we can do it better now, instead of familiarising
373 ourselves with historical mistakes and learning from them.
377 There is no one thing about it that makes it bad software, but taken as
378 a whole, I believe it creates more problems than it solves. Sadly, many
379 of those problems will take years to become apparent, by which time all
380 the major Linux distributions will have fully integrated it. That mess
381 will be left up to some poor soul in the 2040s to deal with, who will
382 hopefully learn from these mistakes and break the vicious cycle.
386 While this article was focused on systemd, it is really just one
387 example of an endemic trend on Linux, and across System V-family
388 systems in general. It's easy to look at an existing tool, identify its
389 deficiencies, write a replacement and label it as progress. It takes
390 considerably more skill to look at an existing tool, identify its
391 strong points, and modify it to work better. That is where Linux stands
392 to learn a lot from BSD.
396 If you think I'm full of shit, feel free to rant at me about it on
397 Twitter or something, but don't expect me to care. I've made my choice
398 to move to OpenBSD, and I couldn't care less whether systemd works well
399 for others or not. I'm simply documenting my experiences because I feel
400 there are important lessons to be learned which I'd rather not forget
401 about, and maybe others would like to read as well.
406 <!--#include virtual="/footer.shtml"-->
projects / www.git / blob