Upgrading my 25gbit internet router to VyOS

It has been a while since I set up my original router for my 25gbit internet connection. I decided it was time to upgrade, but since I have some services running, I did not want to be down for too long and purchased some new hardware which would allow me to experiment with VyOS without affecting my current setup.

There has been a lot of talk around the MS-01 since it came out about a year ago, so I decided to purchase one. Additionally, I purchased a cheap (CHF 42.-) used Mellanox SFP28 card from AliExpress.

The following post contains specific types of configurations I needed and how I set them up.

My plan was to install Proxmox on the MS-01, just like on the “old” server/router, and run VyOS as a VM. The primary reason for this is to make upgrading and backup easier.

I use PCI passthrough to the VyOS VM for the onboard SFP+ and the Mellanox SFP28 PCIe card. One onboard 2.5G is bridged in via Proxmox as well as a bridge with no physical NICs to connect VMs running on MS-01 with VyOS.

VLANs and bridges

The only reason I am even using VLANs is because one of the WANs is located at another part of the house and I did not want to use more than one fiber (which would require more SFP+ modules etc.) to get that WAN to the router.

I have several VLANs at this time which are:

• 100 LAN
• 150 Swisscom WAN
• 200 DMZ (This one did not exist in the old setup, I used it to connect a trunk between the two instances of Proxmox I now have)
• 900 Management

I decided to use bridges in VyOS to connect VLANs and physical interfaces together as well as apply firewall/routing rules

Unlike the Broadcom card in the other server, the Mellanox initially did not work with the SFP28 module from FlexOptix which the ISP (Init7) provided. One option would have been to re-program the SFP28 using a programmer which we have at work, but I did not want to have to re-program the SFP28 every time I wanted to switch which PC it was in.

The solution was to switch to another firmware version of the Mellanox card which did not care about the SFP28 vendor ID.

Thank you to the Init7 community on reddit for this tip.:)

I downgraded to version: fw-ConnectX4Lx-rel-14_24_1000-MCX4121A-ACA_Ax-UEFI-14.17.11-FlexBoot-3.5.603.bin

Flashing process:

After using the MS-01 for a while I noticed that the heat of the PCIe card is not dealt with correctly. I added Nactua 5V fan which includes a USB adapter to the outside of the case to deal with this.

I went from 108°C to 45°C…

Many thanks to John Howard for the example VyOS setup he posted, which got me going in the right direction.

I am new to VyOS so here are a few things that I “learned” setting up my VM.

I configured VyOS with 4 sockets and 4 cores as well as 4GB of RAM (probably overkill at this time). After booting the live environment of the ISO, I ran the setup process to install VyOS to disk.

In order to make my life easier and not having to type my entire config into the KVM shell, I added a virtiofs file system. This allows me to edit the file via an SSH session to the Proxmox server and then load it into VyOS until I have SSH access directly into the VyOS VM.

I only need to enter sudo mount -t virtiofs vyos-config /mnt in VyOS via the shell and then run load /mnt/myconfig in the configure mode.

I also added it to the fstab so it is mounted on reboot:

vyos-config /mnt virtiofs rw,relatime 0 0

I remapped my interfaces to match the order I wanted, and if for any reason VyOS detects them in a different order after an update, I am not affected.

In order for this to work, I had to copy out the auto-generated /config/config.boot, edit the MAC addresses and names, and then copy it back. After a restart, the interfaces were then correctly assigned as I wanted them.

interfaces { ethernet eth0 { hw-id 58:47:ca:xx:xx:xx offload { gro gso sg tso } } ... } ...

I set up several bridges (non-VLAN aware) which connected the VLANs together.

For example, the br200, which is my DMZ, connected the trunk port going to my “old server” as well as the trunk going to Proxmox itself on which VyOS is running so I can connect services running on there as well.

Initaliy I ran into an issue where I could not route traffic between for example eth4.200 and eth6.200 but traffic did go from either one of those to vyos. After several days arguing with ChatGPT to Deepseek I found my mistake (without any AI being even close) which was my load-balancing rule, it was missing the exclusion for br200…

For backup at this time, I have two cronjobs in the VyOS VM to export the structure config and the commands config. These also go into the virtiofs mount.

The second export can be loaded into VyOS via load command

I do not have a fixed IP (although it doesn’t really change very often) so I also send updates to an external DDNS/DNS server of mine.

For the services I am hosting directly from my connection, like Matrix server and Mastodon, I have a DMZ setup. All the services are behind an nginx reverse proxy which also deals with SSL certificates.

I have a NAT rule going to the proxy as well as the appropriate firewall settings

Preserve IPs for nginx proxy x-forwarded-for header

For these sites to work from within my LAN there are two options.

The first one would be to use Hairpin NAT but unlike opnsense it does not work as well especially if the port is 443 or 80 as in my case.

For this reason I decided to use Split DNS which is just defining the interal IP of a service on my local DNS server which I use on all my clients.

So for example:

This caused an issue with my GitLab server when using SSH as it would send traffic to my proxy and not my actual GitLab server.

To solve this, I set up a “SSH Proxy” on my nginx proxy like so:

I have 3 WANs which I have specific order in which they should be used. The following will use the Init7 primarily and failover to the slower connections in order.

For traffic from vyos like ddns calls and wireguard the next-hop distance needs to be adjusted or you will have traffic going out random routes. Since all my WANs are DHCP I can not set static routes so I set the route distance in the DHCP client.

This will result in routes like this:

As I mentioned above, I used John’s blog as a starting point and added a DMZ zone for my hosted services. I also applied my firewall rules to bridges instead of directly the interfaces. Thank you John for the great write up.

Standard road-warrior setup for laptops and phones as well as some remote fixed locations.

Like under opnsense I would like to be able to connect to my WAN2 and WAN3 routers even if the current WAN is Init7.

I set the following static routes but it doesn’t work. I am still trying to figure out what I am missing as I see the traffic for 192.168.8.1 go out the Init7 Link when it should not.

Yallo Router

Swisscom Router

At some point…

The perfornamce is much better than with opnsense and also a reason why I switched over to VyOS.

This is from my proxy which is running under proxmox on my older server via trunk (SFP28) to the MS-01 routed through VyoS to the internet.