Migrating My NAS from CoreOS/Flatcar Linux to NixOS

Table of contents

• Context/History
• My NAS Software Requirements
• Why migrate from CoreOS/Flatcar to NixOS?
• Prototyping in a VM
• Migrating
  • M1. Install NixOS
  • M2. Set up remote disk unlock
  • M3. Set up Samba for access
  • M4. Set up SSH/rsync for backups
• Nice-to-haves
  • N1. Prometheus Node Exporter
  • N2. Reliable mounting
  • N3. nginx-healthz
  • N4. NixOS Jellyfin
  • N5. NixOS samba
  • N6. NixOS rrsync
  • N7. sync.pl script
  • N8. Sharing configs
  • N9. Trying immich!
• Conclusion

In this article, I want to show how to migrate an existing Linux server to NixOS — in my case the CoreOS/Flatcar Linux installation on my Network Attached Storage (NAS) PC.

I will show in detail how the previous CoreOS setup looked like (lots of systemd units starting Docker containers), how I migrated it into an intermediate state (using Docker on NixOS) just to get things going, and finally how I migrated all units from Docker to native NixOS modules step-by-step.

If you haven’t heard of NixOS, I recommend you read the first page of the NixOS website to understand what NixOS is and what sort of things it makes possible.

The target audience of this blog post is people interested in trying out NixOS for the use-case of a NAS, who like seeing examples to understand how to configure a system.

You can apply these examples by first following my blog post “How I like to install NixOS (declaratively)”, then making your way through the sections that interest you. If you prefer seeing the full configuration, skip to the conclusion.

Context/History

Over the last decade, I used a number of different operating systems for my NAS needs. Here’s an overview of the 2 NAS systems storage2 and storage3:

My NAS Software Requirements

• (This post is only about software! For my usage patterns and requirements regarding hardware selection, see “Design Goals” in my My all-flash ZFS NAS build post (2023).)
• Remote management: I really like the model of having the configuration of my network storage builds version-controlled and managed on my main PC. It’s a nice property that I can regain access to my backup setup by re-installing my NAS from my PC within minutes.
• Automated updates, with easy rollback: Updating all my installations manually is not my idea of a good time. Hence, automated updates are a must — but when the update breaks, a quick and easy path to recovery is also a must.
  • CoreOS/Flatcar achieved that with an A/B updating scheme (update failed? boot the old partition), whereas NixOS achieves that with its concept of a “generation” (update failed? select the old generation), which is finer-grained.

Why migrate from CoreOS/Flatcar to NixOS?

When I started using CoreOS, Docker was pretty new technology. I liked that using Docker containers allowed you to treat services uniformly — ultimately, they all expose a port of some sort (speaking HTTP, or Postgres, or…), so you got the flexibility to run much more recent versions of software on a stable OS, or older versions in case an update broke something.

Over a decade later, Docker is established tech. People nowadays take for granted the various benefits of the container approach.

So, here’s my list of reasons why I wasn’t satisfied with Flatcar Linux anymore.

R1. cloud-init is deprecated

The CoreOS cloud-init project was deprecated at some point in favor of Ignition, which is clearly more powerful, but also more cumbersome to get started with as a hobbyist. As far as I can tell, I must host my config at some URL that I then provide via a kernel parameter. The old way of just copying a file seems to no longer be supported.

Ignition also seems less convenient in other ways: YAML is no longer supported, only JSON, which I don’t enjoy writing by hand. Also, the format seems to change quite a bit.

As a result, I never made the jump from cloud-init to Ignition, and it’s not good to be reliant on a long-deprecated way to use your OS of choice.

R2. Container Bitrot

At some point, I did an audit of all my containers on the Docker Hub and noticed that most of them were quite outdated. For a while, Docker Hub offered automated builds based on a Dockerfile obtained from GitHub. However, automated builds now require a subscription, and I will not accept a subscription just to use my own computers.

R3. Dependency on a central service

If Docker at some point ceases operation of the Docker Hub, I am unable to deploy software to my NAS. This isn’t a very hypothetical concern: In 2023, Docker Hub announced the end of organizations on the Free tier and then backpedaled after community backlash.

Who knows how long they can still provide free services to hobbyists like myself.

R4. Could not try Immich on Flatcar

The final nail in the coffin was when I noticed that I could not try Immich on my NAS system! Modern web applications like Immich need multiple Docker containers (for Postgres, Redis, etc.) and hence only offer Docker Compose as a supported way of installation.

Unfortunately, Flatcar does not include Docker Compose.

I was not in the mood to re-package Immich for non-Docker-Compose systems on an ongoing basis, so I decided that a system on which I can neither run software like Immich directly, nor even run Docker Compose, is not sufficient for my needs anymore.

Reason Summary

With all of the above reasons, I would have had to set up automated container builds, run my own central registry and would still be unable to run well-known Open Source software like Immich.

Instead, I decided to try NixOS again (after a 10 year break) because it seems like the most popular declarative solution nowadays, with a large community and large selection of packages.

How does NixOS compare for my situation?

• Same: I also need to set up an automated job to update my NixOS systems.
  • I already have such a job for updating my gokrazy devices.
  • Docker push is asynchronous: After a successful push, I still need extra automation for pulling the updated containers on the target host and restarting the affected services, whereas NixOS includes all of that.
• Better: There is no central registry. With NixOS, I can push the build result directly to the target host via SSH.
• Better: The corpus of available software in NixOS is much larger (including Immich, for example) and the NixOS modules generally seem to be expressed at a higher level of abstraction than individual Docker containers, meaning you can configure more features with fewer lines of config.

Prototyping in a VM

My NAS setup needs to work every day, so I wanted to prototype my desired configuration in a VM before making changes to my system. This is not only safer, it also allows me to discover any roadblocks, and what working with NixOS feels like without making any commitments.

I copied my NixOS configuration from a previous test installation (see “How I like to install NixOS (declaratively)”) and used the following command to build a VM image and start it in QEMU:

The configuration instructions below can be tried out in this VM, and once you’re happy enough with what you have, you can repeat the steps on the actual machine to migrate.

Migrating

For the migration of my actual system, I defined the following milestones that should be achievable within a typical session of about an hour (after prototyping them in a VM):

• M1. Install NixOS
• M2. Set up remote disk unlock
• M3. Set up Samba for access
• M4. Set up SSH/rsync for backups
• Everything extra is nice-to-have and could be deferred to a future session on another day.

In practice, this worked out exactly as planned: the actual installation of NixOS and setting up my config to milestone M4 took a little over one hour. All the other nice-to-haves were done over the following days and weeks as time permitted.

Tip: After losing data due to an installer bug in the 2000s, I have adopted the habit of physically disconnecting all data disks (= pulling out the SATA cable) when re-installing the system disk.

M1. Install NixOS

After following “How I like to install NixOS (declaratively)”, this is my initial configuration.nix:

All following sections describe changes within this configuration.nix.

All devices in my home network obtain their IP address via DHCP. If I want to make an IP address static, I configure it accordingly on my router.

My NAS PCs have one specialty with regards to IP addressing: They are reachable via IPv4 and IPv6, and the IPv6 address can be derived from the IPv4 address.

Hence, I changed the systemd-networkd configuration from above such that it configures a static IPv6 address in a dynamically configured IPv6 network:

✅ This fulfills milestone M1.

M2. Set up remote disk unlock

To unlock my encrypted disks on boot, I have a custom systemd service unit that uses wget(1) and cryptsetup(8) to split the key file between the NAS and a remote server (= an attacker needs both pieces to unlock).

With CoreOS/Flatcar, my cloud-init configuration looked as follows:

I converted it into the following NixOS configuration:

We’ll also need to store the custom TLS certificate file on disk. For that, we can use the environment. configuration:

The references like ${pkgs.wget} will be replaced with a path to the Nix store (→ nix.dev documentation). On CoreOS/Flatcar, I was limited to using just the (minimal set of) software included in the base image, or I had to reach for Docker. On NixOS, we can use all packages available in nixpkgs.

After deploying and rebooting, I can access my unlocked disk under /srv! 🎉

% df -h /srv Filesystem Size Used Avail Use% Mounted on /dev/mapper/data-data 15T 14T 342G 98% /srv

When listing my files, I noticed that the group id was different between my old system and the new system. This can be fixed by explicitly specifying the desired group id:

✅ M2 is complete.

M3. Set up Samba for access

Whereas I want to configure remote disk unlock at the systemd service level, for Samba I want to use Docker: I wanted to first transfer my old (working) Docker-based setups as they are, and only later convert them to Nix.

We enable the Docker NixOS module which sets up the daemons that Docker needs and whatever else is needed to make it work:

This is already sufficient for other services to use Docker, but I also want to be able to run the docker command interactively for debugging. Therefore, I added docker to systemPackages:

After deploying this configuration, I can run docker run -ti debian to verify things work.

The cloud-init version of samba looked like this:

We can translate this 1:1 to NixOS:

✅ Now I can manage my files over the network, which completes M3!

See also: Nice-to-haves: N5. samba from NixOS

M4. Set up SSH/rsync for backups

For backing up data, I use rsync over SSH. I restrict this SSH access to run only rsync commands by using rrsync (in a Docker container). To configure the SSH authorized_keys(5) , we set:

✅ A successful test backup run completes milestone M4!

See also: Nice-to-haves: N6. rrsync from NixOS

Nice-to-haves

N1. Prometheus Node Exporter

I like to monitor all my machines with Prometheus (and Grafana). For network connectivity and authentication, I use the Tailscale mesh VPN.

To install Tailscale, I enable its NixOS module and make the tailscale command available:

After deploying, I run sudo tailscale up and open the login link in my browser.

The Prometheus Node Exporter can also easily be enabled through its NixOS module:

However, this isn’t reliable yet: When Tailscale’s startup takes a while during system boot, the Node Exporter might burn through its entire restart budget when it cannot listen on the Tailscale IP address yet. We can enable indefinite restarts for the service to eventually come up:

N2. Reliable mounting

While migrating my setup, I noticed that calling mount(8) from unlock.service directly is not reliable, and it’s better to let systemd manage the mounting:

Afterwards, I could just remove the mount(8) call from unlock.service:

In systemd services, I can now depend on the /srv mount unit:

N3. nginx-healthz

To save power, I turn off my NAS when they are not in use.

My backup orchestration uses Wake-on-LAN to start a wakeup and needs to wait until the NAS is fully booted up and has mounted its /srv mount before it can start backup jobs.

For this purpose, I have configured a web server (without any files) that depends on the /srv mount. So, once the web server responds to HTTP requests, we know /srv is mounted.

The cloud-init config looked as follows:

The Docker version (ported from Flatcar Linux) looks like this:

This configuration gets a lot simpler when migrating it from Docker to NixOS:

N4. NixOS Jellyfin

The Docker version (ported from Flatcar Linux) looks like this:

As before, when using jellyfin from NixOS, the configuration gets simpler:

For a while, I had also set up compatibility symlinks that map the old location (/data/movies, inside the Docker container) to the new location (/srv/data/movies), but I encountered strange issues in Jellyfin and ended up just re-initializing my whole Jellyfin state. While the required configuration had more lines, I found it neat to move it into its own file, so here is how to do that:

Remove the lines above from configuration.nix and move them into jellyfin.nix:

Then, in configuration.nix, add jellyfin.nix to the imports:

N5. NixOS samba

To use Samba from NixOS, I replaced my systemd.services.samba config from M3 with this:

Note: Setting the samba password in the activation script works for small setups, but if you want to keep your samba passwords out of the Nix store, you’ll need to use a different approach. On a different machine, I use sops-nix to manage secrets and found that refactoring the smbpasswd call like so works reliably:

I also noticed that NixOS does not create a group for each user by default, but I am used to managing my permissions like that. We can easily declare a group like so:

N6. NixOS rrsync

The Docker version (ported from Flatcar Linux) looks like this:

To use rrsync from NixOS, I changed the configuration like so:

N7. sync.pl script

The Docker version (ported from Flatcar Linux) looks like this:

I wanted to stop managing the following Dockerfile to ship sync.pl:

To get rid of the Docker container, I translated the sync.pl file into a Nix expression that writes the sync.pl Perl script to the Nix store:

I can then reference this file by importing it in my configuration.nix and pointing it to the pkgs expression of my NixOS configuration:

This works, but is it the best approach? Here are some thoughts:

• By managing this script in a Nix expression, I can no longer use my editor’s Perl support.
  • I could probably also keep sync.pl as a separate file and use string interpolation in my Nix expression to inject an absolute path to the rsync binary into the script.
• Another alternative would be to add a wrapper script to my Nix expression which ensures that $PATH contains rsync and then the script wouldn’t need an absolute path anymore.
• For small glue scripts like this one, I consider it easier to manage the contents “inline” in the Nix expression, because it means one fewer file in my config directory.

N8. Sharing configs

I want to configure all my NixOS systems such that my user settings are identical everywhere.

To achieve that, I can extract parts of my configuration.nix into a user-settings.nix and then declare an accompanying flake.nix that provides this expression as an output.

After publishing these files in a git repository, I can reference said repository in my flake.nix:

Everything declared in the user-settings.nix can now be removed from configuration.nix!

N9. Trying immich!

One of the motivating reasons for switching away from CoreOS/Flatcar was that I couldn’t try Immich, so let’s give it a shot on NixOS:

Conclusion

You can find the full configuration directory on GitHub.

I am pretty happy with this NixOS setup! Previously (with CoreOS/Flatcar), I could declaratively manage my base system, but had to manage tons of Docker containers in addition. With NixOS, I can declaratively manage everything (or as much as makes sense).

Custom configuration like my SSH+rsync-based backup infrastructure can be expressed cleanly, in one place, and structured at the desired level of abstraction/reuse.

If you’re considering managing at least one other system with NixOS, I would recommend it! One of my follow-up projects is to convert storage3 (my other NAS build) from Ubuntu Server to NixOS as well to cut down on manual management. Being able to just copy the entire config to set up another system, or try out an idea in a throwaway VM, is just such a nice workflow 🥰

…but if you have just a single system to manage, probably all of this is too complicated.