Automated Installation of Proxmox VE

This article documents the automated installation process of Proxmox projects like Proxmox VE.

Introduction

The automated installation method allows installing a Proxmox solution in an unattended manner. This enables you to fully automate the setup process on bare-metal. Once the installation is complete and the host has booted up, automation tools like Ansible can be used to further configure the installation.

The necessary options for the installer must be provided in an answer file. This file allows using filter rules to determine which disks and network cards should be used.

To use the automated installation, it is first necessary to choose a source from which the answer file is fetched from and then prepare an installation ISO with that choice.

Once the ISO is prepared, its initial boot menu will show a new boot entry named "Automated Installation" which gets automatically selected after a 10-second timeout.

Overview

Assistant Tool

The proxmox-auto-install-assistant tool provides the prepare-iso sub-command which can be used to prepare a new enough, but otherwise standard ISO of a Proxmox project for automated installation. You will have to install it first:

apt install proxmox-auto-install-assistant

Note: The xorriso binary is required for preparing an ISO. On Debian-based systems, you can install it using apt install xorriso.

See the help of the sub-command has more details: proxmox-auto-install-assistant prepare-iso --help.

Note: The prepare-iso --partition-label option is available starting with the following ISO releases: PVE 8.3-1, PBS 3.3-1

Answer File Format

The answer file is a TOML-formatted configuration file that provides the basic configuration for a system, like the root password, the network configuration and the target root disk.

To allow the installation of systems with (for example) many disks and network devices, you can use filters to match on device properties, such as serial numbers, vendor, or other unique details like the MAC address of a network interface.

For more details about the schema and the possible filters, see the Answer File Format section.

Answer File Source

There are several locations where the automatic installer can fetch an answer file from:

• From the ISO directly
• Read from a separate partition, identified by the partition label
• Fetched through HTTP from the network
  • The source URL can be pre-determined or queried from DNS or DHCP

For now, you must choose exactly one source.

Prepare an Installation ISO

See the following for a more detailed description of the possible sources and examples on how to prepare a ISO with the different fetch-from modes.

Answer included in the ISO

It is possible to prepare an ISO so that it includes an answer.toml directly, allowing you to have a unified medium for rolling out automatic installation.

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from iso --answer-file /path/to/answer.toml

Answer on Separate Partition

The ISO can also be prepared to search for an answer file called answer.toml on a separate partition/file system (for example, on a USB flash drive). In this mode, the automatic installer searches for a partition with a specific name, by default either proxmox-ais or PROXMOX-AIS (Automated Installation Source).

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from partition

The partition label for which the automatic installer should search for can be customized by passing the --partition-label parameter to proxmox-auto-install-assistant.

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from partition --partition-label YOURLABEL

Afterward, prepare the USB flash drive, for example on /dev/sdX1. You may then adapt and run the following commands as root:

Answer Fetched via HTTP

To fetch the answer file via HTTP(S), the installer needs to know the URL. The URL may be provided via the following ways:

• URL defined in the ISO
• as DHCP option (250)
• via a DNS TXT record
  • The DNS TXT record needs to be located at proxmox-auto-installer.{search domain}, where {search domain} is the search domain provided by the DHCP server.

Note: Fetching the fingerprint via DHCP or DNS records is only done if the same method is used to retrieve the URL!

The HTTP(S) POST request sends JSON data that can help to identify the physical machine and use that information to generate a custom answer file. A full example of the JSON data can be found under the code listing at System information POST data.

Certificate Fingerprint Matching

It is possible to provide the 'SHA256' certificate fingerprint of the TLS certificate. This is useful in the following situations:

• the URL is using an IP address instead of a FQDN
• a self-signed certificate is used which the installer does not trust
• added security by pinning the known certificate of the server

There are three ways to provide the 'SHA256' fingerprint to the installer:

• Fingerprint defined in the ISO
• as a DHCP option (251)
• via a DNS TXT record
  • The DNS TXT record must be located at proxmox-auto-installer-cert-fingerprint.{search domain}, where {search domain} is the search domain provided by the DHCP server.

Note: Fetching the fingerprint via DHCP or DNS records is only done if the same method is used to retrieve the URL!

Example

Similarly, it is possible to specify the URL from which the installer should fetch an answer file as well as the TLS certificate fingerprint. For example, to specify both:

Answer File Format

The answer file is expected in TOML format.

The following example shows an answer file with all possible sections. It uses the DHCP-provided network settings and will use ZFS in a RAID-1 on disks sda and sdb. It also has a post installation webhook and first-boot hook configured:

Global Section

This section contains the following keys:

• keyboard -- The keyboard layout with the following possible options:

de de-ch dk en-gb en-us es fi fr fr-be fr-ca fr-ch hu is it jp lt mk nl no pl pt pt-br se si tr

• country -- The country code in the two letter variant. For example, at, us or fr.
• fqdn -- Specifies the fully-qualified domain name of the host or the method to retrieve it dynamically.
  • If set to a string, it will be set as fully-qualified domain name of the target. The domain part will be used as the search domain.
  • Alternatively, two sub keys are available. See also the example section for a usage example.
    • source -- Must be set to from-dhcp. Retrieves the host name (option 12) and, if available, domain name (option 15) from the DHCP lease configuration. In this mode, the DHCP server must provide a host name, otherwise the installation will fail.
    • domain -- Fallback (search) domain name to use if the DHCP server does not provide one.
• mailto -- The default email address for the user root.
• timezone -- The timezone in tzdata format. For example, Europe/Vienna or America/New_York.
• root-password -- The password for the root user.
• root-password-hashed -- The pre-hashed password for the root user, which will be written verbatim to /etc/passwd. May be used instead of root_password and can be generated using the mkpasswd tool, for example.
• root-ssh_keys -- Optional. SSH public keys to add to the authorized_keys file of the root user after the installation.
• reboot-on-error -- Optional. If set to true, the installer will reboot automatically when an error is encountered. Defaults to false, giving the administrator a chance to investigate why an installation failed.
• reboot-mode -- Optional. Specifies whether the target machine should be rebooted or powered off after a successful installation. Options are reboot (default) and power-off.

Either root_password or root_password_hashed must be set. Setting none or both will result in a validation error.

Network Section

This section contains the following keys:

• source -- Where to source the static network configuration from. This can be from-dhcp or from-answer. If set to from-dhcp the other network options must not be set. The installer will then use the active NIC and the DHCP settings received during installation to write out a static network configuration.
• cidr -- The IP address in CIDR notation. For example, 192.168.1.10/24
• dns -- The IP address of the DNS server.
• gateway -- The IP address of the default gateway.
• filter -- Filter against the UDEV properties to select the network card. See filters.

Disk Setup Section

This section contains the following keys:

• filesystem -- One of the following options: ext4, xfs, zfs, or btrfs. btrfs is only available for Proxmox VE installations.
• disk-list -- List of disks to use. Useful if you are sure about the disk names. For example: disk_list = ["sda", "sdb"]

Note: The content of disk_list is only a list of disks to be considered, but not required for the installer. If disks are listed but not accessible, the installer will still continue, given that enough disks from the list can be used for the selected filesystem or RAID mode.

• filter -- Filter against UDEV properties to select the disks for the installation. See filters.

Note: Use either disk_list or filter. Defining both is not allowed.

• filter-match -- Can be "any" or "all". Decides if a match of any filter is enough or if all filters need to match for a disk to be selected. The default is "any".
• zfs -- Defines ZFS-specific properties. See ZFS Advanced Options of our documentation. The properties are:
  •  raid -- The RAID level that should be used. Options are raid0, raid1, raid10, raidz-1, raidz-2, or raidz-3.
  •  ashift -- Optional. Specifies the ashift property of the created zpool.
  •  arc-max -- Optional. Specifies the maximum amount of memory in MiB ZFS may use for its ARC in. See also Limit ZFS Memory Usage.
  • checksum -- Optional. Specifies the checksumming algorithm. Options are on (default), fletcher4 and sha256.
  •  compress -- Optional. Specifies whether compression is enabled and if yes, what algorithm to use. Options are on (default), off, lzjb, lz4, zle, gzip and zstd.
  •  copies -- Optional. Specifies the copies>property of the created zpool. See zfsprops.7 for more information.
  •  hdsize -- Optional. Defines the total hard disk size to be used in GB. Only honored for bootable disks, that is only the first disk or mirror for RAID0, RAID1 or RAID10, and all disks in RAID-Z[123].
• lvm -- Advanced properties that can be used with the ext4 or xfs file system. See LVM Advanced Options. The properties are:
  •  hdsize -- Optional. Specifies the total hard disk size to be used in GB. It can be used to reserve free space on the hard disk for further partitioning after the installation.
  •  swapsize -- Optional. Specifies the size of the swap volume in GB. Default is the size of installed memory, clamped to between 4 GB and 8 GB.
  •  maxroot -- Optional. Specifies the maximum size of the root volume in GB. Maximum is hdsize / 4.
  •  maxvz -- Optional. Specifies the maximum size of the data volume in GB.
  •  minfree -- Optional. Specifies the amount of free space that should be left in the LVM volume group.
• btrfs -- Defines BTRFS specific options.

Note: Only available on Proxmox VE installations. When using the HTTP method, it can also be dynamically determined from the system information data using the "product"."enable_btrfs" key.

• •  raid -- The RAID level that should be used. Options are raid0, raid1, and raid10.
  •  hdsize -- Optional. Specifies the total hard disk size to be used in GB.
  • compress -- The compression type to use. Possible options are on, off, zlib, lzo and zstd. Defaults to off. See also the btrfs(5) manpage.

Post Installation Webhook Section

Optional. It can be used to configure a webhook to be called after a successful installation. See Post-installation notification for more information.

This section contains the following keys:

• url -- The URL the information about the installed system should be sent to as HTTP POST request.
• cert-fingerprint -- Optional. SHA256 certificate fingerprint if certificate pinning should be used.

First Boot Hook Section

Optional. It can be used to configure a shell script or a compiled binary to run on the first boot of the new system after a successful installation.

If configured, this installs an additional package named proxmox-first-boot. After booting the new system for the first time, this package can safely be removed using apt purge proxmox-first-boot.

This section contains the following keys:

• source -- Where to source the executable for running at first boot from. It can either be from-iso or from-url. When using from-iso, the executable must be specified when preparing the ISO using proxmox-auto-install-assistant prepare-iso --on-first-boot path/to/file.sh.
• ordering -- Optional. At what stage of the boot to run the hook. It can be one of before-network, network-online or fully-up. - before-network -- Runs before any networking devices are set up. - network-online -- Runs after network connectivity has been established. See also network-online.target. - fully-up -- Default. The system is fully booted and ready for normal operation. Project-dependent APIs will also be available. For example, on Proxmox VE tools like qm, pct and pvesh can be used.
• url -- Required when source = "from-url". The URL of the executable file to download.
• cert-fingerprint -- Optional. SHA256 certificate fingerprint if certificate pinning should be used for the download of the executable file.

See also the example section for a full example.

Note: The maximum executable file size is 1 MiB, for both integrating it into the ISO and fetching it from a URL.

Note: Compiled binaries must target x86_64 Linux and should be statically linked, as there are no guarantees for libraries installed on the first boot.

Note: Scripts must start with a shebang specifying the interpreter. Only interpreters that are already present after installation from the ISO can be used.

Answer File Validation

The proxmox-auto-install-assistant tool can also be used to validate the syntax of an answer file and display the identifying information that will be sent to the HTTP(s) server when fetching the answer file.

For example, to validate an answer file:

Answer file format changes

Changes are backwards-compatible (if not otherwise noted) and do not affect existing setups.

Introduced with ISO release Change description

PVE 8.2-1, PMG 8.2-1, PBS 3.2-1	Initial ISO capable of automated installation.
PVE 8.3-1, PBS 3.3-1	Introduced new Post-installation notification and first-boot hook mechanism sections and global.root-password-hashed answer file option (see Global Section).
PVE 8.4-1, PBS 3.4-1	Introduced global.fqdn sub-options global.reboot-mode option, see Global Section for details. All keys can now also be specified in kebab-case.

Filters

Filters allow you to match against device properties exposed by udevadm.

The proxmox-auto-install-assistant utility can display these properties and allows you to test filters in advance. The utility is available in the installer environment (via its debug mode) and on already existing Proxmox Virtual Environment installation.

For example, to fetch information about the available disks:

The key of the filter decides on which property it should be applied to. For example, in order to match against the vendor and model number of the disk, the filter in the answer file could look like this:

Note: The * globbing symbol at the end is used to match anything after the defined filter!

This will match for all Kioxia disks with that model number. You may verify which disks will be found by the filter by running the following command:

The filter_match parameter controls whether all filters must apply, or if it is enough if any of the filters match. This makes it possible to use different disk models for the installation by using different properties.

For example:

Note: For network cards, only the first match will be used as the installer requires only one network card.

More complex network setups can be configured after the installation. Using properties with unique identifiers will result in the most predictable behavior (for example, the MAC address).

Filter Syntax

The following special characters can be used in filters:

• ? -- matches any single character
• * -- matches any number of characters, can be none
• [a], [abc], [0-9] -- matches any single character inside the brackets, ranges are possible
• [!a] -- negate the filter, any single character but the ones specified

Useful Properties

For network cards, the following properties can be useful:

• ID_NET_NAME
• ID_NET_NAME_MAC
• ID_VENDOR_FROM_DATABASE
• ID_MODEL_FROM_DATABASE

For disks, these properties can be useful:

• DEVNAME
• ID_SERIAL_SHORT
• ID_WWN
• ID_MODEL
• ID_SERIAL

Post-installation notification

Optionally, a webhook can be configured to receive detailed information as JSON data about the new system after a successful installation as HTTP POST request.

The request includes information about disks, network interfaces, machine-id and SSH host public keys, to uniquely identify machines. It can also be used to trigger additional automated setup deployment tools.

A full example of what such JSON data looks like can be found under the code listing at Post-installation webhook JSON example.

Meta schema information

The "$schema" object carries meta-information about the JSON document itself.

Currently, only one field is defined:

• version -- Describes the version and thus structure of the document. Follows the format "<major>.<minor>" and applies semantic versioning meaning for both the major and minor number. The major number will be increased on breaking changes, such as removing an existing field. The minor number will be increased for backwards-compatible changes, for example the addition of new fields.

Schema changes

Schema version Change description Introduced with ISO release

1.0	Initial schema/feature release.	PVE 8.3-1, PMG 8.2-1, PBS 3.3-1
1.1	New top-level key reboot-mode, specifying whether the target machine should reboot or power-off after installation. See also the corresponding key in the answer file global section.	PVE 8.4-1, PBS 3.4-1

Helper Tool

The proxmox-auto-install-assistant tool can be used to validate the syntax of an answer file and display the identifying information that will be sent to the HTTP(s) server when fetching the answer file.

For example, to validate an answer file:

If the syntax of the answer file has errors, it will let us know:

To display the identifying information of the current machine:

How useful the identifying information is depends mainly on the hardware vendor and how much of the information is configured correctly. In this example, you can see that some fields are still configured with placeholder values.

Examples

Answer Files

ZFS Mirror on Samsung Disks

We assume that there are only two Samsung disks present. These should be used for the OS in a ZFS Mirror (RAID-1). Additionally, we want to use only 150 GiB of the disks' capacities and leave the rest empty for further customization.

Ext4 With No Swap and Data LV on /dev/sda

Install on /dev/sda and define swap and data LVs with size 0.

ZFS Mirror With Manual Network Config

Note: There is a * in the filter for the network card. This is necessary because that property usually has enx prefixed.

When displaying this property with proxmox-auto-install-assistant device-info -t network, its full value looks like this:

Retrieving FQDN from DHCP with domain fallback

Retrieving the first-boot executable from a URL during installation

Serving Answer Files via HTTP

Note: These are merely examples! Please ensure that your connection is secured via TLS or that your network is trusted.

Setting up a reverse proxy in front that provides TLS for your server is a good idea.

Serving a Static Answer File via netcat

This variant is doing the bare minimum to provide a single answer file.

Ensure that you have netcat-traditional installed:

Create a directory in which the server will run and give it adequate permissions, for example:

Place your answer.toml file in that directory.

You may then serve the file on https://[HOST IP]:8000 like this:

The process will run in the background.

Note: Please ensure that your connection is secured by TLS through something like a reverse proxy.

To terminate the process, you can kill it via its job ID or its PID. For such background jobs, those can be listed via jobs -l.

For example, if the process's job ID is 1 and its PID is 371012, you can terminate it by running either kill %1 or kill 371012.

Serving a Static Answer File via Python

This option showcases how to use Python to serve a single answer file. It can be a starting point for your solution.

Ensure that you have python3-aiohttp installed:

Create a directory in which the server will run and give it adequate permissions, for example:

Place your answer.toml file in that directory.

Then save the following script as server.py in the server's directory as well:

The server's directory should now look like this:

You may now serve the answer file on https://[HOST IP]:8000/answer by starting the server with Python:

The server will run in the foreground and can be terminated by hitting CTRL+C in the console.

Note: Please ensure that your connection is secured by TLS through something like a reverse proxy.

Serving Answer Files Depending on MAC Address via Python

This is a slightly more advanced example that can dynamically serve answer files depending on the MAC address of the host that made the request. This is achieved by reading the JSON data from the HTTP POST request.

Ensure that you have python3-aiohttp and python3-tomlkit installed:

Create a directory in which the server will run and give it adequate permissions, for example:

Before you can run the server below, you must set up the following:

• Place a file named default.toml in the server's directory.

This is the fallback answer file that will be used if no MAC address match was found.

• Create a directory named answers in the server's directory.

This directory will contain the answer files associated with each MAC address.

You may then add as many answer files to the answers directory as you want.

• Each file must be named after the MAC address it is associated with and also end with .toml.

For example: BC:24:11:AB:12:21.toml

Then save the following script as server.py in the server's directory as well:

The server's directory should now look like this:

You may now serve your answer files on https://[HOST IP]:8000/answer by starting the server with Python:

The server will run in the foreground and can be terminated by hitting CTRL+C in the console.

Note: Please ensure that your connection is secured by TLS through something like a reverse proxy.

Third party tools

The following third-party tools around the Proxmox Automated Installation might be useful:

• https://github.com/natankeddem/autopve Answer file server with web-based GUI

Troubleshooting

If the installation fails for some reason, it will drop into a shell (unless the reboot_on_error option in the answer file is set to true). This gives you the chance to troubleshoot what went wrong.

The log files of interest will most likely be:

• /tmp/fetch_answer.log — the steps to retrieve an answer file
• /tmp/auto_installer — parsing of the answer file, matching of hardware to use
• /tmp/install-low-level-start-session.log — the actual installation process

Code listings

System information POST data

The actual contents of the DMI information ("dmi" key) might vary wildly, depending on the system. The keys itself are guaranteed to be present, but the values depend on the manufacturer and firmware of the system.

Post-installation webhook JSON example

The system has multiple disks and network interfaces, was installed using ext4 as a file system on the first disk and the first network interface is used as management interface.