.png)
3 hours ago
2
Lots of projects claim to be the “smallest” or “simplest” Kubernetes, but they never provide data to back it up. Let’s look at how these distributions compare to Talos Linux. Note that Talos Linux is not a Kubernetes distribution, but rather a Linux distribution purpose-built for running upstream Kubernetes.
Before we look at the data, we need to define our criteria.
Criteria
All of the options need to scale down as well as up. Meaning we want to look at multi-node Kubernetes options even though we’ll be running them in a single-node configuration. The tests create a single node “cluster” with as many default settings as possible. This includes running the Kubernetes control plane and etcd—or embedded etcd—on each node.
Single-node options such as KubeSolo, KinD, minikube, or K3s with SQLite were not tested because they have a different use case and have made other trade-offs to minimize system usage. KubeSolo is the smallest Kubernetes footprint we tested, but it only works on a single node and removes components (e.g., Kubernetes scheduler). We wanted the smallest distribution capable of using worker nodes and a highly available control plane.
What We Tested
For every metric, we’ll be measuring the average or total usage over one hour of the system being idle. No external Kubernetes API calls, workloads, or remote access (e.g., SSH) to the system.
The metrics we’ll be comparing are:
- CPU usage
- Memory usage
- Disk i/o
- Network i/o
- Disk usage
Nodes are all configured as the same size VM with 2 vCPUs and 4096 GB of memory. For all options, except Talos Linux, Ubuntu 24.04 LTS was picked as the base operating system. The latest Kubernetes version was deployed for each cluster, which was 1.32 or 1.33. Stats are gathered with the Prometheus node exporter deployed as a DaemonSet and graphed in Grafana.
All of these metrics were gathered in the most consistent way we could find and should be repeatable by anyone, but repeating this experiment will not produce the exact same numbers. After running and re-running the tests we found that the comparison of each system was roughly the same even if the numbers didn’t match.
The distros we tested are:
- Kubeadm
- Talos Linux
- K3s
- K0s
- Kairos with k3s
- RKE2
- Canonical Kubernetes
Kubeadm Baseline
Kubeadm is the gold standard for setting up an upstream Kubernetes cluster and it’s where we started. We installed Ubuntu with all updates and manually ran kubeadm init and deployed the Prometheus node exporter. It gave us a respectable baseline to compare the other options.
Metrics don’t tell the whole story, of course. There’s a reason companies don’t run their Kubernetes clusters from Kubeadm. Most of those reasons involve high levels of maintenance between the operating system, certificates, Kubernetes components, and a lack of support.
But, as a measure of only metric usage, it’s a great starting point.
Talos Linux
Talos Linux is the only Linux distribution made specifically for running Kubernetes. It streamlines maintenance for the operating system and Kubernetes and does not have a general-purpose init system. It’s the only option we’ll be examining that includes the OS and Kubernetes in a single API. But how many resources does it use?
Compared to Kubeadm, it requires 7% less memory, 6% more CPU, 49% less disk i/o, 16% more network i/o, and 47% less disk storage. Over the week of testing, we regularly saw CPU usage and Network usage be the same or less on Talos than Kubeadm. This is just a snapshot of 1 hour of usage. Memory usage, disk i/o, and total disk usage were always less on Talos Linux.
Note: The only configuration change made from the default installation was the disabling of the local dashboard. This is also recommended for single-node and edge installations.
K3s
K3s has a reputation for being the smallest Kubernetes distribution. They claim an entire Kubernetes distribution in a single binary, less than 70 MB. In testing, K3s actually downloaded quite a few additional binaries, including containerd, CNI plugins, and kubectl. It is primarily configured and maintained through a bash script, and it’s up to you to figure out how to maintain and secure your operating system.
The main benefit K3s and similar “single binary” distributions bring is an embedded etcd server. By embedding etcd, they end up saving a bit of CPU usage at the cost of more memory.
You’ll see that it’s using 15% more memory than the baseline kubeadm installation, with the trade-off of 19% less CPU. Using an external etcd database is supported, but we didn’t test this configuration. Disk i/o was 50% higher, network i/o 7% lower, and disk usage was 8% lower.
K3s does things in a slightly opinionated way. We didn’t change any of the default settings, except for using etcd. K3s is advertised as being great for lightweight environments, while RKE2 (we’ll cover later) is meant for a more compliant and production-ready Kubernetes.
K0s
K0s is a newer, “single binary” option that takes a lot of inspiration from K3s. It looked like the only thing different was some of the opinions about the included configuration, which have been made slightly easier to change. It’s heavily encouraged to manage K0s via Cluster API, and it seems to be targeted toward running it in Kubernetes (similar to vCluster) more than single-node usage.
K0s also uses embedded etcd and has a very similar footprint to K3s. The only interesting difference was that it appeared to use less network traffic. Every other metric was equal to and sometimes higher or lower than K3s during testing.
It is also installed and maintained with a bash script by default and doesn’t include any maintenance for the underlying Linux distribution.
Kairos
Kairos is a “meta” Linux distribution that repackages other Linux distributions (e.g., Ubuntu) and optionally includes a Kubernetes distribution (i.e., k3s, k0s, or “generic”). It does this in an interesting way, and for me it was the most confusing option to install. Their GitHub releases have over 300 assets, and it’s configured with cloud-init. Besides Talos Linux, this is the only Linux distribution that bundles the OS and Kubernetes.
This was the only distribution I tried that I failed to get working multiple times. It also had a large discrepancy between node exporter metrics and what appeared to actually be happening on the system for disk usage. I say that to let you know some of these metrics are not accurate, but I wanted to include them anyway.
Seeing that the Kubernetes component is K3s, it makes sense that the metrics are similar. Memory usage is almost exactly the same, and CPU usage is just a little bit higher. Disk r/w is 42% lower than K3s and network is 68% higher.
Disk usage was hard to collect because of their “immutable” mounting system. They take traditional Linux distributions and remount portions of the disk into read-only partitions and RAM disks.
The total install is says 124 GB, which is impossible on a 20 GB drive. When I SSH to the system and use df the installation is 6.4 GB on /, /oem, /usr/local, and /run/initramfs/cos-state.
RKE2
RKE2 isn’t advertised for small environment usage, but it is a common “upgrade” from K3s. It’s more Kubernetes conformant with fewer opinions built in. It even uses a stand-alone etcd instance by default, which gives a better idea of what the other “single binary” options would consume if they used a full etcd option.
Almost all of the metrics are 2x higher than the Kubeadm baseline. Memory usage is 150% higher, CPU is 230% higher, disk i/o is 260% higher, network i/o is the same, and installed size is 143% larger.
This still doesn’t include any OS management, configuration, or hardening. Local Kubernetes maintenance is done via a bash script and config file like K3s.
Canonical Kubernetes
Canonical Kubernetes is the official offering from Canonical, and their marketing material claims it’s good for a small footprint Kubernetes.
CPU usage and network i/o were on par with the others, but memory usage was the most—201% more than Kubeadm—and disk i/o was 533% more than the baseline.
Note: We didn’t test MicroK8s because, from what we could tell, it is being replaced with Canonical Kubernetes, and setting it up didn’t work with the node exporter.
Conclusion
Talos Linux is unique. It’s the only option that includes OS management in a purpose-built distribution for running Kubernetes. There’s no compromise for scaling up or down. In terms of small-scale numbers, it “wins” in several of the examined categories, including memory usage, disk r/w, and installation size. But all of these metrics are side effects of Talos Linux’s defining characteristic: It’s simple.
We believe the most important metric is your time. We’ve worked with Kubernetes, and we’ve been there. The real story isn’t how many GB this takes compared to that, but rather how all of the above metrics add up to reflect how you spend your days. Talos is the only option that eliminates traditional OS management.
Next time you’re looking for a small Kubernetes distribution, be sure to look at the whole picture. While adding a single binary to a system comprised of 2700 others may not make a big difference, Talos Linux reimagined what’s possible. It is fundamentally different, with orders of magnitude fewer system binaries than any other option.
This simplicity provides the reliable speed and efficiency Kubernetes users can benefit from on a daily basis. We regularly hear from our customers that the greatest benefit of Talos Linux is time saved, and we have seen it reduce maintenance time by 66%. With fewer CVEs, simpler configuration, and dependable upgrades, you just get more.
Want to run the tests yourself? Here’s ths gist!
