.png)
1 month ago
5
Published: 05/10/2025
4 minute read
TL;DR: Rootless Docker has become easy! You can also easily run the GitLab Runner binary with rootless Docker without impacting workloads with rootless-kit. Even run buildkit, dind and docker build! See the HOW-TO!
GitLab CI is the biggest CI/CD system after GitHub Actions and is the preferred solution for self-hosted and enterprise SCM-Systems. While GitHub gives you a fresh, full-blown VM for each job, GitLab has the concept of different Executors. While the GitLab-Runner binary manages the communication and job setup, the executor does the actual work.
While there are Shell and SSH executors, container-based executors are often preferred for flexibility, security and reproducibility. While Docker can improve security, it is not know for it in its default configuration, since everything runs as root.
Attack Surface Docker
By default, Docker is started as root and everyone in the docker user group can communicate with the root-owned socket in /var/run/docker.sock. Since the user-ids inside a container map 1:1 to the user-ids on the host, a simple docker run --it --privileged -v /:/host debian can overtake the whole system. Even if the user executing it is not root.
ℹ️ Docker and other containers are NOT virtualization! All processes run on the same kernel as the host and are only isolated by process namespace and cgroup capabilities.
This alone is kind of bad. It gets even worse if you run arbitrary, potentially malicious code from unknown systems. A complete host takeover is only one services: [docker:dind] or a stupid runner config away.
But how can you run jobs rootless and what about docker build jobs that need dind?
Securing Docker & CI
This can be mitigated by not running Docker as root at all. Rootless Docker used to be a pain but has become surprisingly easy on modern systems.
NOTE: The instructions are based on a Debian system but can be adapted to others.
First install Docker according to the official docs:
| 1 | # Install pre-requirements and some extra packages |
| 2 | sudo apt update |
| 3 | sudo apt install curl ca-certificates curl uidmap apparmor lsb-release \ |
| 4 | slirp4netns dbus-user-session fuse-overlayfs systemd-container fuse-overlayfs cifs-utils |
| 5 | |
| 6 | # Add the docker apt repo |
| 7 | sudo install -m 0755 -d /etc/apt/keyrings |
| 8 | sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc |
| 9 | sudo chmod a+r /etc/apt/keyrings/docker.asc |
| 10 | echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \ |
| 11 | https://download.docker.com/linux/debian \ |
| 12 | $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ |
| 13 | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null |
| 14 | |
| 15 | # Finally install docker |
| 16 | sudo apt update |
| 17 | sudo apt install docker-ce docker-ce-cli containerd.io \ |
| 18 | docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras |
Now Docker is running as root. Which we don’t want. Disable it via systemd and optionally allow normal users to open ports below 1024.
| 1 | # Disable the root docker service |
| 2 | sudo systemctl restart apparmor.service |
| 3 | sudo systemctl disable --now docker.service docker.socket |
| 4 | sudo rm /var/run/docker.sock |
| 5 | # Allow privileged ports for normal users |
| 6 | sudo sysctl -w net.ipv4.ip_unprivileged_port_start=0 |
Now we need the gitlab-runner binary that communicates with GitLab and our executor. By default, the official install scripts also runs gitlab-runner as root. It should also run as a normal user:
| 1 | # Create gitlab-runner user and install the GitLab Runner |
| 2 | sudo useradd --create-home --comment gitlab-runner --shell /usr/bin/bash --user-group gitlab-runner |
| 3 | curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | \ |
| 4 | sudo bash |
| 5 | sudo apt install gitlab-runner |
| 6 | sudo systemctl stop gitlab-runner |
| 7 | sudo systemctl disable --now gitlab-runner |
We want to run docker rootless, but the users inside of containers are often root or require root. Since we want all our workloads to work with the new setup we use the user-namespace capability of the Linux kernel. This maps the root user form inside a container to an unprivileged user to the host. So even if an attacker manages to break out of a container, he will never have privileges higher than the newly created gitlab-runner user.
| 1 | # Set user-namespace mapping |
| 2 | sudo echo $(id -u gitlab-runner):100000:65536 >> /etc/subuid |
| 3 | sudo echo $(id -u gitlab-runner):100000:65536 >> /etc/subgid |
| 4 | |
| 5 | # If you run everything headless without a login shell you need to enable lingering |
| 6 | sudo loginctl enable-linger |
We can now switch to the new gitlab-runner user and set up docker and the required configs:gitlab-runner.service
| 1 | # gitlab-runner.service |
| 2 | [Unit] |
| 3 | Description="GitLab Runner" |
| 4 | ConditionFileIsExecutable="/usr/bin/gitlab-runner" |
| 5 | After="network.target" |
| 6 | |
| 7 | [Service] |
| 8 | Restart="always" |
| 9 | RestartSec=120 |
| 10 | StartLimitInterval=5 |
| 11 | StartLimitBurst=10 |
| 12 | Environment="DOCKER_HOST=unix:///run/user/<<YOUR_GITLAB_RUNNER_USERID>>/docker.sock" |
| 13 | ExecStart='/usr/bin/gitlab-runner "run" "--config" "/home/gitlab-runner/gitlab-runner-config.toml" "--working-directory" "/home/gitlab-runner" "--service" "gitlab-runner" "--user" "gitlab-runner"' |
| 14 | |
| 15 | [Install] |
| 16 | WantedBy="default.target" |
| 1 | # gitlab-runner-config.toml |
| 2 | concurrent = 4 |
| 3 | check_interval = 0 |
| 4 | shutdown_timeout = 0 |
| 5 | [session_server] |
| 6 | session_timeout = 1800 |
| 7 | [[runners]] |
| 8 | name = "Rootless Docker" |
| 9 | url = "https://gitlab.com" |
| 10 | id = 1234 |
| 11 | token = "<MY_TOKEN>" |
| 12 | executor = "docker" |
| 13 | [runners.docker] |
| 14 | image = "debian" |
| 15 | allowed_pull_policies = ["always", "if-not-present"] |
| 16 | allowed_privileged_images = ["moby/buildkit:*", "docker:dind" ] |
| 17 | disable_entrypoint_overwrite = false |
| 18 | privileged = true |
| 19 | oom_kill_disable = false |
| 20 | disable_cache = false |
| 21 | volumes = ["/cache", "/certs/client"] |
| 22 | shm_size = 0 |
| 23 | network_mtu = 0 |
We now can start rootless docker and configure gitlab-runner to use it:
| 1 | sudo -i && su gitlab-runner |
| 2 | mkdir -p .config/systemd/user /.config/docker |
| 3 | |
| 4 | # Run rootless docker |
| 5 | dockerd-rootless-setuptool.sh install |
| 6 | |
| 7 | # You can now run docker commands |
| 8 | export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock |
| 9 | docker run --rm hello-world |
| 10 | |
| 11 | # Setup GitLab Runner conf and systemd service with the conf above |
| 12 | # Remember to adjust your user-id and set your token |
| 13 | vim .config/systemd/user/gitlab-runner.service |
| 14 | vim gitlab-runner-config.toml |
| 15 | |
| 16 | # Activate the runner |
| 17 | systemctl --user daemon-reload |
| 18 | systemctl --user enable gitlab-runner.service |
| 19 | systemctl --user start gitlab-runner.service |
With the correct token and GitLab server configured you should now see the runner in the GitLab UI.
You now can run any CI-Job with this runner - even dind works!
| 1 | rootless-job: |
| 2 | tags: [my-rootless-runner] |
| 3 | services: [docker:dind] |
| 4 | stage: test |
| 5 | scripts: |
| 6 | - echo "I am running as $(whoami)" |
And now?
This is one measurement to avoid security issues that may overtake your CI runners, but it is by far not the only attack that often used. Supply chain security is a wide and deep topic and the upcoming post will explain how to protect your environment from credential threats and other attacks.
