Systemd's Nuts and Bolts – A Visual Guide to Systemd

If you’re an intermediate or advanced Linux user or sysadmin, you might have felt an odd fascination with the myth of systemd. I invite you to this deep dive into systemd's nuts and bolts. I'm not gonna beat around the bush: It's a hairy business, it will be hard, but I promise juicy and satisfying rewards if you keep it up.

Let’s start by uncovering the “D” of systemd, the secret sauce that doesn't get the love it deserves: D-Bus.

Warning: This guide starts from systemd’s internals, and only towards the end reaches the user-facing interface. By this we aim for a deeper, “bottom-up,” solid understanding. Readers in urgent need of getting familiar with systemd are advised to look elsewhere (but feel free to skim and look at the pretty visual graphics, whose likeness is yet to be replicated elsewhere by any other systemd guide).

So, what is this D-Bus thing? D-Bus (short for “Desktop Bus”) is a high-level message-passing mechanism for Inter-Process Communication (IPC).

If that sounds like a mouthful, think of it as a fancy, typed Remote Procedure Call (RPC) protocol, not unlike gRPC, GraphQL, or ZeroMQ, but heavily optimized for processes talking to each other on a single machine.

Originally designed to keep everything in sync on desktop environments like GNOME and KDE, its usefulness has expanded to all sorts of system services.

Sure, you could roll your own IPC with a log file and a lock, or a mess of UNIX sockets, but you’d have to invent a protocol and a server to manage it all. D-Bus gives you a standard framework for this, and it’s typically implemented over those very same UNIX sockets.

D-Bus also pioneered the concept of “service activation,” a concept eventually extended (and extinguished?) by systemd itself. Systemd later created a similar new trick, “socket activation” (more on those later, but it’s basically about performance).

Downsides of D-Bus? It’s a bit verbose, a bit like Java (more on that later), and Linux-only. But then again, Linux is the best OS in the world, so it gets a pass. Spoiler alert: systemd is therefore also Linux-only, and its stubbornness in this respect has been a point of contention for some, and of contentment for others (such as one Lennart Poettering, the mastermind of this ordeal).

‘systemctl’ vs ‘busctl’ as D-Bus clients (visual guide)

D-Bus’ anatomy is as beautiful as the compositions of the young French composer Claude Debussy.

It’s a client/server architecture. One server, many clients usually. There are plenty o’ libraries to build D-Bus entities yourself.

The D-Bus daemon runs a “bus,” a virtual channel where messages fly back and forth. Actually, there are usually two buses running on your system:

• The System Bus: For system-wide services. This is where the big players hang out: NetworkManager, and even systemd itself, which is actually a D-Bus service this whole time! (Surprise!)
• The Session Bus: A private bus for each logged-in user, handling things like desktop notifications and application communication.

When a program connects to a bus, it gets a unique ID like :1.1553. But since nobody wants to remember that, it can also claim a friendly, "well-known name” (aka “bus name”), which by convention looks suspiciously like a Java package name (e.g., org.freedesktop.NetworkManager, ugh). This is how you can talk to the "Network Manager service" without caring about its PID, its D-Bus ID, or what particular implementation of the D-Bus interface is actually running the client process. It’s basically DNS for D-Bus.

The above flexibility is part of the claim made that systemd is just a collection of tools, and they are all replaceable. But naturally, no one does that because everyone likes to complain but no-one has a level of virtuous masochism that would make them want to replace even a part of systemd.

The real fun begins with the D-Bus object model. It’s got a whole OOP-ish vibe. A service on the bus doesn’t just send random messages; it exposes objects. These objects live at “paths” that look like filesystem paths but are just a representation of the object’s hierarchy (e.g., /org/freedesktop/systemd1). These objects have methods you can call and signals you can listen for, which are grouped into interfaces (again with the Java names, like org.freedesktop.systemd1.Manager, but not to be confused with the D-Bus well-known name mentioned above, despite them being the same sometimes.)

Yes, I know, you lost the will to live by this point. But if you think about it, every typed RPC protocol feels like looking at the void of the abyss. If you stick around, I’ll show you a future where this might not be that much of a problem.

“This is getting way too abstract,” you’re probably thinking. Fine, let’s make it concrete. The systemctl command you use all the time is basically a fancy, user-friendly wrapper for making D-Bus calls.

When you type systemctl start my-cool-app.service, you're not writing arcane magic. You're just sending a D-Bus message to the systemd process (PID 1), which is a D-Bus service. In fact, you can do it yourself with the busctl tool. This command does the exact same thing:

Do you need to know anything at all about D-Bus to use systemd? Not really, but it helps to ground the monstrosity of it all.

Before moving on, the winds of change are talking about a D-Bus alternative: Varlink. It’s a newer RPC protocol that uses JSON instead of XML, but works basically the same. Fear not, it’s been said that “systemd deeply integrates with D-Bus, and that's not going to change any time soon." Currently, systemd exposes its APIs both as D-Bus and as Varlink.

Now that we got a taste of all sorts of Ds, let’s go down the rabbit hole into the C.

Cgroup Hierarchy with Systemd (visual guide)

Ever heard of Docker containers? They are how we solved the problem of running safe and isolated software on any OS: If the OS is Linux, use the containerization features of the OS (a key one, but not sufficient for full containerization, is “cgroups”) to run your software. If the OS is NOT Linux, run Linux in a VM and repeat the previous step. Easy.

So, what are these “cgroups” which found their way into the cornerstone of cross-platform software deployment?

You can think of cgroups (aka “control groups”) as providing a vital part of the full containerization experience. Namely the isolation of hardware resources (CPU, Memory, I/O, etc.) between processes. And to keep processes somewhat isolated from each other.

And, more relevant for us, systemd uses cgrups extensively to provide a level of security and isolation to all processes on your machine.

Basically, if your Linux has systemd, you can't escape cgroups. Every single process runs in one cgroup. Some cgroups contain many processes (there is one cgroup for each user and its processes), but if you use systemd services, each of them gets its own cgroup by default. This is a huge win. In the old days, a daemon could fork itself into oblivion, detaching from its parent and becoming an untraceable ghost in the machine.

You can get a bird’s-eye view of the whole cgroup hierarchy with systemd-cgls, or peek at the raw data yourself, since the kernel conveniently mounts them as a virtual filesystem under /sys/fs/cgroup. (Everything's a file on Linux, even your dignity!)

Example of a cgroup hierarchy managed by systemd:

Systemd organizes this hierarchy with its own naming convention, which also happens to correspond to its own “unit” types (more on that later, I promise). You’ll see things ending in:

• .slice: A group of cgroups. Think of it as a folder for organizing other cgroups, primarily for resource management. The whole system starts in the root slice (-.slice), with user processes in user.slice and system services in system.slice.
• .service: A cgroup for a service unit, like your nginx.service. This is the most common one you'll see.
• .scope: A cgroup for grouping “foreign” processes that systemd didn't start itself, but still wants to keep tabs on. Your user login session, for example, lives in a scope.

“But wait,” you say, “I see a systemd process nested inside another service!" Yes, you've found the nested systemd --user instance.

Each logged-in user gets their own private systemd manager that runs services meant for the user. It has its own D-Bus bus and manages its own cgroup hierarchy under your user slice. And yes, it's confusing that your graphical session apps live in a .scope managed by the main PID 1, while other user background services live under the systemd --user service. The rationale is... well, I want to believe there's a rationale.

To clarify things a little bit: Just as there is a “System D-Bus” and “user-specific D-Buses” running at the same time, there is a “system systemd” and “user-specific systemds” running at the same time.

You can get in on the cgroup action yourself. The systemd-run command lets you launch any command in its own transient .service or .scope unit, effectively giving you on-the-fly cgroups. It's like your own personal nice or nohup, allowing you to set resource limits for a single command.

Then there’s the related systemd-run0, which is meant to become the new sudo, but we don't talk about that because sudo is a sacred spell passed unto us eons ago.

Alright, we’ve met the referee (D-Bus) and the field (cgroups). Now it’s time to meet the actual players (Sorry American readers, we in the real world use soccer metaphors, we are manly like that, even our women).

In the world of systemd, everything is a "unit."

They are declarative little configuration files that tell systemd what you want done, not how to do it. You don't write a script that says "start process A, wait 5 seconds, check if it's alive, then start process B." Instead, you create two unit files and tell systemd, "Hey, unit B needs unit A to be running first." systemd cleans up the mess. It's like SQL for exorcising your machine, but with no EXPLAIN keyword.

Now I know you can’t wait to get your hands on a unit, but first let’s explore the different types of units. There are 11 of them, but let’s focus on the ones people actually use:

• .service: This unit type starts and controls a daemon or any long-running process (or any process you want really). It’s the most common type of unit you’ll write or interact with.
• .socket: This unit encapsulates a local IPC or network socket. When traffic arrives on this socket, systemd will activate the corresponding .service unit. This is "socket activation," an idea heavily inspired by macOS's launchd. It allows services to be started on-demand, reducing boot time and resource usage (likely the best selling point of systemd is how much of the system initialization can be done in parallel due to this).
• .target: A way to group other units and create synchronization points. Targets don’t do much themselves; they are just hooks for dependencies. When you start a target, you’re telling systemd to start all the units that are WantedBy or RequiredBy that target. Some targets correspond to the old SysV init runlevels; for example, multi-user.target is roughly equivalent to runlevel 3. In any case, targets usually encapsulate the units needed to consider that the boot process reached a particular milestone. On boot, systemd activates the default.target unit, which is usually a symlink to graphical.target or multi-user.target (yes, this is UNIX, we use symlinks to configure stuff, but also that's not the only way. Yes this is tricky, we'll get to that later. Help, I'm trapped in a systemd article and I can't get out!).
• .timer: For triggering the activation of other units based on timers. This is systemd's answer to cron. You can define "realtime" timers that run on a calendar schedule (e.g., every Monday at 2 AM) or "monotonic" timers that run after a certain amount of time has passed since an event (e.g., 15 minutes after boot).
• .slice and .scope: These relate directly back to cgroups. A .slice unit is used to group other units together in the cgroup tree for resource management. A .scope unit is used to manage "foreign" processes that systemd didn't start itself but wants to track.

Systemd Unit Directories (Visual Guide)

Units are configured in simple .ini-style text files (a classic Windows shenanigan). These files live in a few specific places, with a clear hierarchy of precedence. For system-wide units, systemd looks in:

• /etc/systemd/system: Highest precedence. For sysadmins' custom units, symlinks to enabled units, and overrides.
• /run/systemd/system: For runtime-generated units.
• /usr/lib/systemd/system: Lowest precedence. For units installed by packages (e.g., via pacman or apt).

The rule of thumb is: Be like MC Hammer, and don’t touch this: /usr/lib/systemd/system. If you need to change a package-provided unit, you should create an override file in /etc/systemd/system instead. We'll get to that.

A unit file typically has three sections: [Unit], [Install], and a type-specific section like [Service] or [Timer].

Systemd Unit File Sections (Visual Guide)

This section contains generic information about the unit and, most importantly, its relationship with other units.

• Description=: A short, human-readable string describing the unit. This is what you'll see in the output of systemctl status.
• Wants=: A space-separated list of units that should be started along with this one. This is a "weak" dependency. If a unit listed in Wants= fails to start, systemd doesn't care and will start this unit anyway.
• Requires=: A "strong" dependency. If a unit listed here fails to start, this unit will not be started. Furthermore, if a required unit is stopped later, this unit will be stopped too. In general, it's better to use Wants= to build a more robust system.
• After= and Before=: These define the startup order. After=other.service means this unit will only be started after other.service has successfully started. Note that this is completely separate from Wants= or Requires=. If you want unit B to start after unit A, you need to specify both a requirement and an ordering: Requires=A.service and After=A.service.
• Conflicts=: The opposite of Requires=. If this unit is started, any unit listed here will be stopped, and vice versa.

Now this is where things get weird as fuck, so hold on.

This section is not used by systemd at runtime. Instead, it's read by the systemctl enable and systemctl disable commands. It tells systemctl how to hook the unit into the boot process.

So basically, it’s a way of providing “make me bootable” instruction to systemd, but then it's up to the user to request to make it bootable. Making it bootable typically consists of saying which target this depends on using options in this section (which systemd will use to create some weird ass symlinks in a particularly named folder which is equivalent to declaring the dependency explicitly on the target, except we DON'T do that because we want to keep the original target files clean. You got that? Good, you're starting to believe in the D).

• WantedBy=: This is the most common directive here. WantedBy=multi-user.target tells systemctl enable to create a symlink to this unit file inside the /etc/systemd/system/multi-user.target.wants/ directory. This effectively makes our unit a dependency of multi-user.target, ensuring it gets started during a normal boot. (I'm sorry, I don't like this any more than you do).

This section is specific to .service units and defines how to run the process.

• Type=: This tells systemd how to track the service's startup process. Common types are:
• simple: (The default) The service is considered "started" as soon as the main process is forked. exec is generally preferred.
• exec: (What you wish were the default) The service is considered started after the execve() system call has successfully completed.
• forking: For traditional UNIX daemons that fork a child process and then have the parent exit. systemd considers the service started once the original parent process exits. This type is best avoided if possible.
• oneshot: For short-lived tasks. The service is considered active until the main process exits. Often used with RemainAfterExit=yes to make other units depend on the fact that this task has completed successfully at least once.
• notify: The service is expected to send a "READY=1" message back to systemd via the sd_notify library call when it's ready. This is the most robust way for a service to signal readiness.
• dbus: The service is considered ready once it acquires a specific D-Bus name.
• ExecStart=: The full command (with arguments) to execute to start the service.
• Restart=: Configures if and when the service should be automatically restarted. Common values are no (default), on-failure (if it exits with a non-zero code), and always.
• StartLimitIntervalSec= and StartLimitBurst=: A rate-limiting mechanism to prevent a service from getting stuck in a rapid crash-restart loop. For example, StartLimitIntervalSec=10 and StartLimitBurst=5 means systemd will stop trying to restart the service if it fails 5 times within a 10-second window.

Execution Environment

You can control the environment of the executed process with directives that are common to several unit types:

• WorkingDirectory=: Sets the current working directory for the process.
• User= and Group=: Specifies the UNIX user and group the process should run as.
• Environment=: Sets environment variables directly, e.g., Environment="VAR1=foo" "VAR2=bar".
• EnvironmentFile=: Specifies a file to read environment variables from (one KEY=VALUE pair per line).

Overriding Unit Files

As mentioned, you shouldn’t directly edit files in /usr/lib/systemd/system. The proper way to modify a unit is to create a "drop-in" file. systemctl edit my-app.service is the magic command for this. It will open an editor for a new file, typically at /etc/systemd/system/my-app.service.d/override.conf.

In this file, you only need to specify the sections and directives you want to add or change. For example, to change the restart policy of my-app.service, your override.conf might just contain:

To clear a list-based option like Wants=, you can set it to an empty value: Wants=. When you're done, systemd merges this drop-in file with the original, with your changes taking precedence. After making any changes to unit files, you must run systemctl daemon-reload to make systemd aware of them. (systemctl edit does this for you automatically).

Socket Units

Socket units (or “suck it” units) are the key to on-demand service startup. A .socket file defines a socket, and for each one, there must be a matching .service file (e.g., my-app.socket and my-app.service).

The .socket file has a [Socket] section:

• ListenStream=, ListenDatagram=: Define the address to listen on. For a TCP socket, this would be an IP address and port (127.0.0.1:8080). For a UNIX domain socket, it's a file path (/run/my-app.sock).
• Accept=: If no (the default), systemd passes all listening sockets to a single instance of the started service. If yes, systemd accepts each connection itself and spawns a new instance of the service for each one. This requires the service unit to be a template (e.g., [email protected]).

When you enable and start the .socket unit, systemd creates the socket and listens on it. The .service unit isn't started yet. Only when the first connection arrives does systemd start the service and pass it the ready-to-go socket file descriptor.

Timer Units

Timer units replace cron. A .timer file activates a matching .service file. The timer unit contains a [Timer] section with directives that define the schedule.

• Realtime (wallclock) timers: Use OnCalendar= with a specific calendar event format.
• OnCalendar=weekly runs once a week at midnight on Monday.
• OnCalendar=*-*-* 14:00:00 runs daily at 2 PM.
• OnCalendar=Mon,Tue *-*-01..05 12:00 runs at noon on the first five days of the month, if they are a Monday or Tuesday.
• Monotonic timers: Activate after a time span relative to a starting point.
• OnBootSec=15min runs 15 minutes after the system boots.
• OnUnitActiveSec=1w runs one week after the unit was last activated.
• Persistent=true: If the system was down when the timer should have fired, this setting makes it run as soon as possible after the system is next booted.

You enable and start the .timer unit, not the .service unit it controls. You can see all active timers and when they're next scheduled to run with systemctl list-timers.

systemctl is your command-line interface to the systemd manager. We've mentioned it a lot, so here's a more structured look at its main commands.

Unit Inspection Commands

• systemctl status [UNIT...]: Shows detailed runtime status of one or more units, including its state (active, inactive, failed), cgroup, recent log entries, and more. If no unit is given, it shows the overall system status.
• systemctl list-units [--all] [--type=TYPE]: Lists units that systemd has loaded into memory. By default, it only shows active units.
• systemctl list-unit-files: Lists all available unit files found on the system and their state (enabled, disabled, static).
• systemctl cat UNIT...: Shows the contents of a unit file, including any drop-in overrides, so you can see the final, merged configuration.
• systemctl list-dependencies UNIT: Shows a tree of the requirement dependencies for a unit.

Unit Management Commands

• systemctl start UNIT...: Starts (activates) one or more units.
• systemctl stop UNIT...: Stops (deactivates) one or more units.
• systemctl restart UNIT...: Stops and then starts a unit.
• systemctl reload UNIT...: Asks a service to reload its configuration without a full restart. The service must be designed to support this (e.g., via ExecReload=).
• systemctl enable UNIT...: Enables a unit to start at boot, by creating the symlinks defined in the [Install] section. This does not start the unit right now.
• systemctl disable UNIT...: The opposite of enable; removes the symlinks.
• To do both at once, use the --now flag: systemctl enable --now my-app.service.
• systemctl daemon-reload: Reloads all unit configuration files from disk. You must run this after manually creating or editing a unit file (but not when using systemctl edit).

The final piece of the core systemd puzzle is its logging system, the journal. The systemd-journald daemon collects log messages from a variety of sources:

• Kernel messages (dmesg)
• Standard syslog messages (the old system, cross-Unix, extended and extinguished by the D)
• Standard output and standard error from all services managed by systemd
• Messages written directly to the Journal API

All of this data is stored in a structured, indexed binary format in /var/log/journal/ (if configured for persistence) or /run/log/journal/ (non-persistent). It's also cryptographically protected to prevent tampering. The tool to read it is journalctl.

Using journalctl

journalctl by itself will dump the entire journal, which is usually too much. The real power is in its filtering options.

• Follow logs live: journalctl -f
• Show logs for a specific unit: journalctl -u nginx.service
• Show logs from the current boot: journalctl -b (or -b -1 for the previous boot).
• Filter by time: journalctl --since "2023-10-26" --until "1 hour ago"
• Filter by priority: journalctl -p err shows all messages with a priority of "error" or higher (err, crit, alert, emerg).
• Change output format: journalctl -o verbose shows all structured fields for each log entry.
• Kernel messages: journalctl -k is a shortcut for viewing just the kernel ring buffer.

You can also use systemd-cat to pipe the output of any command directly into the journal. For example: ls -l / | systemd-cat -p info -t my-script will run the command and log its output with an "info" priority and the identifier "my-script".

This concludes our tour of systemd. God I'm tired. From D-Bus IPC and cgroup process management to the declarative unit system and the structured journal, systemd provides a comprehensive, integrated suite of tools for managing a modern Linux system. While it has a steep learning curve and its share of controversies, understanding its core components gives you immense power and control over your machine's lifecycle.

Now go sit back, relax, and drink the alcoholic beverage of your choice. You earned it.