1 <!--#include virtual="/doctype.txt"-->
4 <!--#set var="pagetitle" value="Why I dislike systemd"-->
5 <!--#include virtual="/head.shtml"-->
9 <!--#include virtual="/header.shtml"-->
11 <!--#include virtual="/navigation.shtml"-->
15 <h1>Why I dislike systemd</h1>
18 As a Linux sysadmin in the 2010s, it's hard not to have an opinion on
19 systemd. But what I find baffling about it is how divisive it is;
20 nearly everyone (or at least the most vocal crowd) seems to either love
21 it or hate it. When I tell people that systemd was the catalyst for my
22 defection to OpenBSD last year, their usual reaction is to assume that
23 I am part of the "hate it" group. Nope.
27 In truth, systemd itself was a very small part of the reason I jumped
28 ship. Its introduction made me realise two important things. First, the
29 design problems with modern Linux run deeper than any one piece of
30 software, I just hadn't noticed until I had a fresh one to learn.
31 Second, and this is specific to Debian, the "universal operating
32 system" mantra is fundamentally flawed; you cannot support all use
33 cases when confronted with two alternatives that each function best
34 when adopted to the exclusion of all others.
38 But you didn't come here to read my life story. Back to systemd.
42 Let me be clear. systemd is not the work of the devil, it probably
43 doesn't contain any NSA backdoors, and it isn't the worst piece of
44 software the big Linux distributions are shipping by default. It does
45 address some real problems with Linux, and its approach does have its
50 It's often said that a half-truth is worse than a lie. In the case of
51 systemd, a half-improvement is worse than a complete flop. People are
52 focusing on the features they want to use and ignoring the ones that
53 will come back to bite them later.
57 No, my biggest problem with systemd is that it is repeating the
58 mistakes of System V all over again. It's just given them a change of
59 clothes to appeal to the 21st century.
62 <h2>A bit of history</h2>
65 In order to understand how we got here, we need to cast our minds (or,
66 for people under the age of 50 like myself, our imaginations) back to
67 the mid-1960s. Time-sharing operating systems were a new and innovative
68 concept, and many people still used batch processing as the established
69 method for getting things done. Many different organisations were
70 experimenting with many different approaches to time-sharing. One of
75 Multics was the conceptual ancestor of all UNIX and UNIX-like systems.
76 The problems that made Multics a failure were not its core design
77 principles, but the design and implementation of its parts. This is
78 much more effectively communicated using a concrete example, from <a
79 href="http://www.utterpower.com/10132011-dennis-ritchie-bell-labs-rip-c-language-and-unix/">
80 a paper</a> by Dennis Ritchie first presented in 1979.
84 cite="http://www.utterpower.com/10132011-dennis-ritchie-bell-labs-rip-c-language-and-unix/">
86 The very convenient notation for IO redirection, using the `>’ and
87 `<‘ characters, was not present from the very beginning of the PDP-7
88 Unix system, but it did appear quite early. Like much else in Unix, it
89 was inspired by an idea from Multics. Multics has a rather general IO
90 redirection mechanism embodying named IO streams that can be
91 dynamically redirected to various devices, files, and even through
92 special stream-processing modules. Even in the version of Multics we
93 were familiar with a decade ago, there existed a command that switched
94 subsequent output normally destined for the terminal to a file, and
95 another command to reattach output to the terminal. Where under Unix
99 <pre><code>ls >xx</code></pre>
102 to get a listing of the names of one’s files in xx, on Multics the
106 <pre><code>iocall attach user_output file xx
108 iocall attach user_output syn user_i/o</code></pre>
111 Even though this very clumsy sequence was used often during the Multics
112 days, and would have been utterly straightforward to integrate into the
113 Multics shell, the idea did not occur to us or anyone else at the time.
114 I speculate that the reason it did not was the sheer size of the
115 Multics project: the implementors of the IO system were at Bell Labs in
116 Murray Hill, while the shell was done at MIT. We didn’t consider making
117 changes to the shell (it was their program); correspondingly, the
118 keepers of the shell may not even have known of the usefulness, albeit
119 clumsiness, of iocall. (The 1969 Multics manual lists iocall as an
120 `author-maintained,’ that is non-standard, command.) Because both the
121 Unix IO system and its shell were under the exclusive control of
122 Thompson, when the right idea finally surfaced, it was a matter of an
123 hour or so to implement it.
128 What we have here is, fundamentally, a trade-off. Ritchie's speculation
129 on the reason for the trade-off aside, Multics tried to be everything
130 to everyone by providing the most general interface possible. As a
131 result, it was inefficient to develop and cumbersome to use. It got the
132 job done by making the 99% majority of use cases more difficult in
133 order to make the 1% minority simpler. This is a design pattern I think
134 is being repeated in systemd.
138 Most importantly for the point I'm making, an interface like iocall is
139 so obviously wrong to us <strong>because we have seen what a better
140 interface looks like</strong>. To the designers of Multics, it was a
141 perfectly reasonable approach to solving a very real problem. That is
142 not the same thing as being a good solution.
145 <h2>The commercialisation of UNIX</h2>
148 Fast-forward to the 1980s. AT&T wanted to commercialise UNIX, ended
149 the practice of distributing source code along with their binaries, and
150 began implementing features they thought would make their product sell.
151 The culmination of these efforts is the System V family of operating
152 systems, of which Linux is a member (albeit as a clone, not a direct
153 descendant). For the purposes of this discussion, I'll be referring to
154 System III as a member of the System V family, as the distinction isn't
159 One of the things System V inherited from Research UNIX was an init
160 that provided inflexible process monitoring, and invoked an
161 ever-growing shell script (/etc/rc) to perform service startup. Its
162 developers decided that the best way to tackle this problem was to
163 invent a new configuration language which would enable init to perform
164 more complex and customisable tasks, along with a way of allowing
165 vendors to drop in their own service startup routines and have them run
166 automatically at boot.
170 Sound familiar? It should. The fact that System V's init failed so
171 spectacularly at scaling to the problems of today, while modern BSD
172 systems continue to manage just fine by way of incremental improvements
173 to the pre-existing components, should serve as a lesson in itself.
176 <h2>Introducing systemd</h2>
179 Okay, enough with the history lesson already. You came here to read a
183 <h3>It shuffles complexity around</h3>
186 A common point raised in favour of systemd is that its unit files are
187 "simpler" than init scripts. Superficially, this is true. What's really
188 going on, though, is that the complexity has been moved from the logic
193 In stark contrast to Bourne shell, so many of systemd's unit directives
194 are highly specialised building blocks which aren't easily reusable.
195 This isn't necessarily a bad thing, except that because systemd units
196 aren't real programs, there's no way to escape to more basic primitives
197 as required. Inevitably, what this means is that more and more
198 directives will be added over time as more and more use cases are
203 Don't believe me? Wait until 2025, and if the number of unit directives
204 hasn't at least tripled compared to today (and people aren't working
205 around their absence by invoking wrapper scripts from ExecStart), I'll
206 buy you a drink. Hint: At least one Debian package (memcached) is
207 already invoking a wrapper script from ExecStart.
211 If you start with complex logic in a simple language, you can always
212 simplify your implementation. If you start with simple logic in a
213 complex language, you're stuck with that complexity as soon as anyone
214 else starts using it.
217 <h3>It babysits its users</h3>
220 This complaint is far from unique to systemd, but is pandemic to a
221 disturbingly increasing number of programs, especially on Linux. Why is
222 <tt>systemctl edit</tt> even a thing?
226 No, you're just a <strong>user</strong>. You couldn't possibly manage
227 the job of invoking an editor on a configuration file without bricking
228 your computer, so here's a wrapper that does it for you. Oh, you edited
229 the file directly anyway? Here, let me helpfully <strong>tell you that
230 you should run <tt>systemctl daemon-reload</tt>, but perform the action
231 you requested using the old configuration anyway</strong>.
235 Speaking of which, why is <tt>systemctl daemon-reload</tt> even a
236 thing? When did filesystems become so uncool that not only are they
237 demoted to second-class citizens beneath some other primary source of
238 truth, but you don't even reload files when you <strong>know</strong>
239 they've been changed?
243 Now, I'm well aware that I'm giving an oversimplification of the issue
244 here. There are deeper design issues that make <tt>systemctl edit</tt>
245 quite different from opening an editor on a unit file, but my point is
246 that <strong>the user doesn't care</strong>. All they see is an overly
247 complex CLI that does little other than edit a file. However, this does
248 make a convenient segue into my next point...
251 <h3>It takes the Multics approach to interoperability</h3>
254 One of the fundamental differences between Multics and UNIX is the way
255 programs interact. Multics had programs that were developed
256 independently, and while they were each workable solutions that could
257 be made to complement each other when necessary, they were far from
258 efficient at doing so. UNIX turned this on its head, with one of its
259 most basic design principles to be to "write programs to work
264 Now, systemd does rely heavily on D-Bus, and in doing so makes itself
265 controllable by other applications. You might consider that "working
266 together", but I see systemd as more akin to iocall than UNIX's shell
267 redirection; it provides facilities other programs can use without
268 integrating into their workflow.
272 The previous example is an ideal illustration of this. Using an editor
273 on unit files directly will work, but it's clumsy, and you need to
274 remember to invoke <tt>daemon-reload</tt> after you're done. Yes, it
275 works, but that's not the point. The point is that it's inefficient and
276 counterintuitive to edit unit files except through systemd's own
277 interfaces. That is not what I would call "working together".
281 I'll repeat what I said before, because it's important: To the
282 designers of Multics, it was a perfectly reasonable approach to solving
283 a very real problem. That is not the same thing as being a good
287 <h3>It's unpredictable</h3>
290 Log into a computer running systemd. Try to find the answer to "which
291 units are going to be started on next boot?". This is not quite the
292 same thing as "which units were started on this boot?", and because of
293 the complex dependency relationships between units, it's actually a
294 very difficult question to answer. This is bad if you're a
295 security-conscious sysadmin.
299 The failing of dependency-driven service initialisation is that it
300 works far better in theory than it does in practice. In theory, you
301 just have a directed graph of nodes, and you would like those depended
302 upon by default.target to start. In practice, that's not how systems
303 administration works.
307 As a sysadmin, when I reboot a system, I want to be able to ask "is foo
308 going to start up after a reboot?". I want to know, with reasonable
309 confidence, what the state of my system is going to be when it comes
310 onto the network. This is one area where sysvinit outperforms systemd
315 And yes, you can avoid the whole issue just by masking any units you
316 don't want to start, but that's missing the point. Again, the point I'm
317 making isn't that it doesn't work, just that it doesn't work
318 efficiently. Your tools should make your job easier, not get in your
322 <h3>Its priorities are warped</h3>
325 One of the "features" of systemd is that it allows you to boot a system
326 without needing a shell at all. This seems like such a senseless
327 manoeuvre that I can't help but think of it as a knee-jerk reaction to
328 the perception of Too Much Shell in sysv init scripts.
332 In exactly which universe is it reasonable to assume that you have a
333 running D-Bus service (or kdbus) and a filesystem containing unit
334 files, all the binaries they refer to, all the libraries they link
335 against, and all the configuration files any of them reference, but
336 that you lack that most ubiquitous of UNIX binaries, /bin/sh?
340 The use case often cited for this is managing services inside a
341 container. I don't see why the init on my desktop needs to be
342 complicated and restricted for the sake of a feature used by a minority
343 of people with specialised use cases. By all means, write a tool for
344 bootstrapping containers that doesn't rely on a shell, but don't
345 shoehorn that into a one-size-fits-all init.
349 What makes this especially annoying, though, is that systemd also
350 includes a dumbed-down shell-like parser to handle "EnvironmentFile"s
351 (which usually don't actually set environment variables when sourced
352 from a shell, but that's the way systemd's parser treats them). Also,
353 service units have a pseudo-shell syntax for argument lists. One
354 particularly bizarre feature that breaks expectations for users of
355 pretty much any other software is that $FOO is word-split into multiple
356 arguments, whereas ${FOO} isn't.
360 Is avoiding ever having to execute a binary really so useful in the
361 real world that these complications are justified?
367 I said it in my introduction, and I'll say it again: systemd's approach
368 has its merits. But its shortcomings are representative of a philosophy
369 that seems to be in fashion at the moment on Linux: ignoring history
370 and assuming we can do it better now, instead of familiarising
371 ourselves with historical mistakes and learning from them.
375 There is no one thing about it that makes it bad software, but taken as
376 a whole, I believe it creates more problems than it solves. Sadly, many
377 of those problems will take years to become apparent, by which time all
378 the major Linux distributions will have fully integrated it. That mess
379 will be left up to some poor soul in the 2040s to deal with, who will
380 hopefully learn from these mistakes and break the vicious cycle.
384 While this article was focused on systemd, it is really just one
385 example of an endemic trend on Linux, and across System V-family
386 systems in general. It's easy to look at an existing tool, identify its
387 deficiencies, write a replacement and label it as progress. It takes
388 considerably more skill to look at an existing tool, identify its
389 strong points, and modify it to work better. That is where Linux stands
390 to learn a lot from BSD.
394 If you think I'm full of shit, feel free to rant at me about it on
395 Twitter or something, but don't expect me to care. I've made my choice
396 to move to OpenBSD, and I couldn't care less whether systemd works well
397 for others or not. I'm simply documenting my experiences because I feel
398 there are important lessons to be learned which I'd rather not forget
399 about, and maybe others would like to read as well.
404 <!--#include virtual="/footer.shtml"-->
projects / www.git / blob