My beef with systemd
Systemd is one of the most contientious software suites to ever grace the open-source community. It came in at a time where pretty much all the alternatives were just unsustainable at scale and the well-integrated suite and comparatively pleasant management for sysadmins led pretty much every distro worth their salt to migrate slowly towards it.
However, we're at a point now where there are several competent alternative init systems that often times can serve you just fine. Systemd absolutely has its place in the industry, but I have some gripes and issues with the suite as a whole and certain things it's done to the way Linux is managed.
First of all,
What even IS systemd?
In a nutshell, systemd (system daemon) is a Linux system management suite. It handles init (the first process that starts when a Linux system boots), and a bunch of other things, including but not limited to:
- service management
- logs
- service/unit dependencies
- resource control (cgroups)
- running containers
- socket activation
- timers (replacing cron)
- DNS resolving
- UEFI chainloading
- networking
- mounting disks
What it replaced
Systemd replaced the SysV init system used on most Linux distros. Think of SysV as a nightmare of shell scripts thrown in random places. You had better cross your fingers and pray that you wrote your scripts well enough, because the entire boot process and services are dependent around it.
- Systemd's dependencies allowed for more sensible structuring of services in a different order.
- For example, running a filesystem balance on your RAID array doesn't make sense if it's done BEFORE the service that mounts your array.
The centralized syntax and configuration allowed for less weird distro-isms and quirks between different flavors of Linux. Suddenly, things became a lot more unified.
So, what's the problem?
A large chunk of systemd's optional services are entrenched within the Linux ecosystem and are treated as a default of sorts. A lot of software just expects it, but not all of the suite is of equal quality.
- systemd-resolved is a really mediocre resolver. It, like a lot of other networking services on Linux, symlinks the absolutely crucial /etc/resolv.conf file that defines domain name servers to its own generated file on boot. This is done for the sake of accounting for VPN connections, but it becomes incredibly annoying when you just want to have a dormant system point to a specific DNS server. Modern features like DNS-over-HTTPS and DNSSEC which Windows 11 has recently embraced are nowhere to be found. It's also a pain to get working with tools like dnsmasq.
The problem is less that it's doing stuff with DNS, it's more that it's not particularly transparent about it. The amount of Linux veterans that have to work around their resolv.conf file changing is insane.
-
journald (logging) is not great. Making log files binary offers a theoretical benefit in speed, but in my opinion it's just not worth the extra headache. What happens if you're accessing a system offline? Text-based logs are just way more pleasant to work with.
-
It's got a massive attack surface. It's the first process running on your Linux system and it's attached to a million different other services that are probably also running on your system. It's all written in memory-unsafe C.
Keep it simple, stupid
One of my beliefs is that Linux system administration and development should aim for simplicity within reason. Solutions should be easy to understand, straightforward, and minimal by default. When you structure a system around these ideas, not only does it become a lot more pleasant to maintain as a distro maintainer or developer, but it becomes more feasible for end users and everyone in between to have a better grasp of how their system works. This is the same reason why I'm not a huge fan of mandatory access control systems like SELinux -- while they definitely have their place, even experienced users will resort to switching SELinux off just because the way it restricts access can be frustrating. Often times, you can just be running services just to have them inexplicably silently fail. Add on top that you need to sit down to really understand the system and that it's not reasonable to define and write your own SELinux policies (most people just rely on RHEL/Fedora's good defaults) and you can see why so many people just... turn it off. Even in production.
systemd is a lot easier to work with than SELinux, but the idea still stands. Simpler systems like runit, s6, and OpenRC could potentially be more suitable options for more modest configurations -- think smaller workgroups and homelabs.
I wish it were smaller
I like a lot of the concepts and ideas of systemd. I think unit files are a cool idea, and the declarative management puts it a step above the competition. However, the project is just doing way too much and as a result there's just a lot of shit that's unfortunately attached to an otherwise very interesting init system.
Real world, recent concerns
When writing this blog post I came across and learned about the xz/lzma backdoor that's been chilling on Linux systems for a decent amount of time now. Make sure you update your systems, by the way. In a nutshell, OpenSSH (secure shell daemon for logging in remotely) is compiled and patched to support systemd notifications. While obviously it's a little more nuanced than "systemd allowed for this!!", it makes you think just how tightly ingrained the ecosystem is on Linux.