1 <!--#include virtual="/doctype.txt"-->
4 <!--#set var="pagetitle" value="Why I dislike systemd"-->
5 <!--#include virtual="/head.shtml"-->
9 <!--#include virtual="/header.shtml"-->
11 <!--#include virtual="/navigation.shtml"-->
15 <h1>Why I dislike systemd</h1>
17 <p><em>(Published 2015-06-14)</em></p>
20 As a Linux sysadmin in the 2010s, it's hard not to have an opinion on
21 systemd. But what I find baffling about it is how divisive it is;
22 nearly everyone (or at least the most vocal crowd) seems to either love
23 it or hate it. When I tell people that systemd was the catalyst for my
24 defection to OpenBSD last year, their usual reaction is to assume that
25 I am part of the "hate it" group. Nope.
29 In truth, systemd itself was a very small part of the reason I jumped
30 ship. Its introduction made me realise two important things. First, the
31 design problems with modern Linux run deeper than any one piece of
32 software, I just hadn't noticed until I had a fresh one to learn.
33 Second, and this is specific to Debian, the "universal operating
34 system" mantra is fundamentally flawed; you cannot support all use
35 cases when confronted with two alternatives that each function best
36 when adopted to the exclusion of all others.
40 But you didn't come here to read my life story. Back to systemd.
44 Let me be clear. systemd is not the work of the devil, it probably
45 doesn't contain any NSA backdoors, and it isn't the worst piece of
46 software the big Linux distributions are shipping by default. It does
47 address some real problems with Linux, and its approach does have its
52 It's often said that a half-truth is worse than a lie. In the case of
53 systemd, a half-improvement is worse than a complete flop. People are
54 focusing on the features they want to use and ignoring the ones that
55 will come back to bite them later.
59 No, my biggest problem with systemd is that it is repeating the
60 mistakes of System V all over again. It's just given them a change of
61 clothes to appeal to the 21st century.
64 <h2>A bit of history</h2>
67 In order to understand how we got here, we need to cast our minds (or,
68 for people under the age of 50 like myself, our imaginations) back to
69 the mid-1960s. Time-sharing operating systems were a new and innovative
70 concept, and many people still used batch processing as the established
71 method for getting things done. Many different organisations were
72 experimenting with many different approaches to time-sharing. One of
77 Multics was the conceptual ancestor of all UNIX and UNIX-like systems.
78 The problems that made Multics a failure were not its core design
79 principles, but the design and implementation of its parts. This is
80 much more effectively communicated using a concrete example, from <a
81 href="http://www.utterpower.com/10132011-dennis-ritchie-bell-labs-rip-c-language-and-unix/">
82 a paper</a> by Dennis Ritchie first presented in 1979.
86 cite="http://www.utterpower.com/10132011-dennis-ritchie-bell-labs-rip-c-language-and-unix/">
88 The very convenient notation for IO redirection, using the `>’ and
89 `<‘ characters, was not present from the very beginning of the PDP-7
90 Unix system, but it did appear quite early. Like much else in Unix, it
91 was inspired by an idea from Multics. Multics has a rather general IO
92 redirection mechanism embodying named IO streams that can be
93 dynamically redirected to various devices, files, and even through
94 special stream-processing modules. Even in the version of Multics we
95 were familiar with a decade ago, there existed a command that switched
96 subsequent output normally destined for the terminal to a file, and
97 another command to reattach output to the terminal. Where under Unix
101 <pre><code>ls >xx</code></pre>
104 to get a listing of the names of one’s files in xx, on Multics the
108 <pre><code>iocall attach user_output file xx
110 iocall attach user_output syn user_i/o</code></pre>
113 Even though this very clumsy sequence was used often during the Multics
114 days, and would have been utterly straightforward to integrate into the
115 Multics shell, the idea did not occur to us or anyone else at the time.
116 I speculate that the reason it did not was the sheer size of the
117 Multics project: the implementors of the IO system were at Bell Labs in
118 Murray Hill, while the shell was done at MIT. We didn’t consider making
119 changes to the shell (it was their program); correspondingly, the
120 keepers of the shell may not even have known of the usefulness, albeit
121 clumsiness, of iocall. (The 1969 Multics manual lists iocall as an
122 `author-maintained,’ that is non-standard, command.) Because both the
123 Unix IO system and its shell were under the exclusive control of
124 Thompson, when the right idea finally surfaced, it was a matter of an
125 hour or so to implement it.
130 What we have here is, fundamentally, a trade-off. Ritchie's speculation
131 on the reason for the trade-off aside, Multics tried to be everything
132 to everyone by providing the most general interface possible. As a
133 result, it was inefficient to develop and cumbersome to use. It got the
134 job done by making the 99% majority of use cases more difficult in
135 order to make the 1% minority simpler. This is a design pattern I think
136 is being repeated in systemd.
140 Most importantly for the point I'm making, an interface like iocall is
141 so obviously wrong to us <strong>because we have seen what a better
142 interface looks like</strong>. To the designers of Multics, it was a
143 perfectly reasonable approach to solving a very real problem. That is
144 not the same thing as being a good solution.
147 <h2>The commercialisation of UNIX</h2>
150 Fast-forward to the 1980s. AT&T wanted to commercialise UNIX, ended
151 the practice of distributing source code along with their binaries, and
152 began implementing features they thought would make their product sell.
153 The culmination of these efforts is the System V family of operating
154 systems, of which Linux is a member (albeit as a clone, not a direct
155 descendant). For the purposes of this discussion, I'll be referring to
156 System III as a member of the System V family, as the distinction isn't
161 One of the things System V inherited from Research UNIX was an init
162 that provided inflexible process monitoring, and invoked an
163 ever-growing shell script (/etc/rc) to perform service startup. Its
164 developers decided that the best way to tackle this problem was to
165 invent a new configuration language which would enable init to perform
166 more complex and customisable tasks, along with a way of allowing
167 vendors to drop in their own service startup routines and have them run
168 automatically at boot. [Ed: I was actually slightly mistaken, and this
169 came from SunOS. See <a
170 href="https://news.ycombinator.com/item?id=9717758">this clarification
171 </a> from someone who's been alive longer than I have.]
175 Sound familiar? It should. The fact that System V's init failed so
176 spectacularly at scaling to the problems of today, while modern BSD
177 systems continue to manage just fine by way of incremental improvements
178 to the pre-existing components, should serve as a lesson in itself.
181 <h2>Introducing systemd</h2>
184 Okay, enough with the history lesson already. You came here to read a
188 <h3>It shuffles complexity around</h3>
191 A common point raised in favour of systemd is that its unit files are
192 "simpler" than init scripts. Superficially, this is true. What's really
193 going on, though, is that the complexity has been moved from the logic
198 In stark contrast to Bourne shell, so many of systemd's unit directives
199 are highly specialised building blocks which aren't easily reusable.
200 This isn't necessarily a bad thing, except that because systemd units
201 aren't real programs, there's no way to escape to more basic primitives
202 as required. Inevitably, what this means is that more and more
203 directives will be added over time as more and more use cases are
208 Don't believe me? Wait until 2025, and if the number of unit directives
209 hasn't at least tripled compared to today (and people aren't working
210 around their absence by invoking wrapper scripts from ExecStart), I'll
211 buy you a drink. Hint: At least one Debian package (memcached) is
212 already invoking a wrapper script from ExecStart.
216 If you start with complex logic in a simple language, you can always
217 simplify your implementation. If you start with simple logic in a
218 complex language, you're stuck with that complexity as soon as anyone
219 else starts using it.
222 <h3>It babysits its users</h3>
225 This complaint is far from unique to systemd, but is pandemic to a
226 disturbingly increasing number of programs, especially on Linux. Why is
227 <tt>systemctl edit</tt> even a thing?
231 No, you're just a <strong>user</strong>. You couldn't possibly manage
232 the job of invoking an editor on a configuration file without bricking
233 your computer, so here's a wrapper that does it for you. Oh, you edited
234 the file directly anyway? Here, let me helpfully <strong>tell you that
235 you should run <tt>systemctl daemon-reload</tt>, but perform the action
236 you requested using the old configuration anyway</strong>.
240 Speaking of which, why is <tt>systemctl daemon-reload</tt> even a
241 thing? When did filesystems become so uncool that not only are they
242 demoted to second-class citizens beneath some other primary source of
243 truth, but you don't even reload files when you <strong>know</strong>
244 they've been changed?
248 Now, I'm well aware that I'm giving an oversimplification of the issue
249 here. There are deeper design issues that make <tt>systemctl edit</tt>
250 quite different from opening an editor on a unit file, but my point is
251 that <strong>the user doesn't care</strong>. All they see is an overly
252 complex CLI that does little other than edit a file. However, this does
253 make a convenient segue into my next point...
256 <h3>It takes the Multics approach to interoperability</h3>
259 One of the fundamental differences between Multics and UNIX is the way
260 programs interact. Multics had programs that were developed
261 independently, and while they were each workable solutions that could
262 be made to complement each other when necessary, they were far from
263 efficient at doing so. UNIX turned this on its head, with one of its
264 most basic design principles to be to "write programs to work
269 Now, systemd does rely heavily on D-Bus, and in doing so makes itself
270 controllable by other applications. You might consider that "working
271 together", but I see systemd as more akin to iocall than UNIX's shell
272 redirection; it provides facilities other programs can use without
273 integrating into their workflow.
277 The previous example is an ideal illustration of this. Using an editor
278 on unit files directly will work, but it's clumsy, and you need to
279 remember to invoke <tt>daemon-reload</tt> after you're done. Yes, it
280 works, but that's not the point. The point is that it's inefficient and
281 counterintuitive to edit unit files except through systemd's own
282 interfaces. That is not what I would call "working together".
286 I'll repeat what I said before, because it's important: To the
287 designers of Multics, it was a perfectly reasonable approach to solving
288 a very real problem. That is not the same thing as being a good
292 <h3>It's unpredictable</h3>
295 Log into a computer running systemd. Try to find the answer to "which
296 units are going to be started on next boot?". This is not quite the
297 same thing as "which units were started on this boot?", and because of
298 the complex dependency relationships between units, it's actually a
299 very difficult question to answer. This is bad if you're a
300 security-conscious sysadmin.
304 The failing of dependency-driven service initialisation is that it
305 works far better in theory than it does in practice. In theory, you
306 just have a directed graph of nodes, and you would like those depended
307 upon by default.target to start. In practice, that's not how systems
308 administration works.
312 As a sysadmin, when I reboot a system, I want to be able to ask "is foo
313 going to start up after a reboot?". I want to know, with reasonable
314 confidence, what the state of my system is going to be when it comes
315 onto the network. This is one area where sysvinit outperforms systemd
320 And yes, you can avoid the whole issue just by masking any units you
321 don't want to start, but that's missing the point. Again, the point I'm
322 making isn't that it doesn't work, just that it doesn't work
323 efficiently. Your tools should make your job easier, not get in your
327 <h3>Its priorities are warped</h3>
330 One of the "features" of systemd is that it allows you to boot a system
331 without needing a shell at all. This seems like such a senseless
332 manoeuvre that I can't help but think of it as a knee-jerk reaction to
333 the perception of Too Much Shell in sysv init scripts.
337 In exactly which universe is it reasonable to assume that you have a
338 running D-Bus service (or kdbus) and a filesystem containing unit
339 files, all the binaries they refer to, all the libraries they link
340 against, and all the configuration files any of them reference, but
341 that you lack that most ubiquitous of UNIX binaries, /bin/sh?
345 The use case often cited for this is managing services inside a
346 container. I don't see why the init on my desktop needs to be
347 complicated and restricted for the sake of a feature used by a minority
348 of people with specialised use cases. By all means, write a tool for
349 bootstrapping containers that doesn't rely on a shell, but don't
350 shoehorn that into a one-size-fits-all init.
354 What makes this especially annoying, though, is that systemd also
355 includes a dumbed-down shell-like parser to handle "EnvironmentFile"s
356 (which usually don't actually set environment variables when sourced
357 from a shell, but that's the way systemd's parser treats them). Also,
358 service units have a pseudo-shell syntax for argument lists. One
359 particularly bizarre feature that breaks expectations for users of
360 pretty much any other software is that $FOO is word-split into multiple
361 arguments, whereas ${FOO} isn't.
365 Is avoiding ever having to execute a binary really so useful in the
366 real world that these complications are justified?
372 I said it in my introduction, and I'll say it again: systemd's approach
373 has its merits. But its shortcomings are representative of a philosophy
374 that seems to be in fashion at the moment on Linux: ignoring history
375 and assuming we can do it better now, instead of familiarising
376 ourselves with historical mistakes and learning from them.
380 There is no one thing about it that makes it bad software, but taken as
381 a whole, I believe it creates more problems than it solves. Sadly, many
382 of those problems will take years to become apparent, by which time all
383 the major Linux distributions will have fully integrated it. That mess
384 will be left up to some poor soul in the 2040s to deal with, who will
385 hopefully learn from these mistakes and break the vicious cycle.
389 While this article was focused on systemd, it is really just one
390 example of an endemic trend on Linux, and across System V-family
391 systems in general. It's easy to look at an existing tool, identify its
392 deficiencies, write a replacement and label it as progress. It takes
393 considerably more skill to look at an existing tool, identify its
394 strong points, and modify it to work better. That is where Linux stands
395 to learn a lot from BSD.
399 If you think I'm full of shit, feel free to rant at me about it on
400 Twitter or something, but don't expect me to care. I've made my choice
401 to move to OpenBSD, and I couldn't care less whether systemd works well
402 for others or not. I'm simply documenting my experiences because I feel
403 there are important lessons to be learned which I'd rather not forget
404 about, and maybe others would like to read as well.
409 <!--#include virtual="/footer.shtml"-->
projects / www.git / blob