Skip to content

Instantly share code, notes, and snippets.

@joshenders

joshenders/openwrt_incus_multiwan.md

Last active August 2, 2025 08:58
Show Gist options
  • Save joshenders/4966c9c7da03bf57799cd07d8695c83f to your computer and use it in GitHub Desktop.
Save joshenders/4966c9c7da03bf57799cd07d8695c83f to your computer and use it in GitHub Desktop.
Download ZIP
OpenWrt 24.10 Setup Under Incus 6.0.x on Debian 12 with PCI Passthrough
Raw
openwrt_incus_multiwan.md

OpenWrt 24.10 Setup Under Incus 6.0.x on Debian 12 with PCI Passthrough

Install lxd via apt and configure

apt install incus

Configure lxd

The example config below uses the dir storage driver and no bridging since we'll be using PCI passthrough.

Tip

If you have different needs, just run lxd init (or incus admin init) interactively and adjust accordingly.

incus admin init --preseed << EOF
---
config: {}
networks: []
storage_pools:
- config: {}
  description: ""
  name: default
  driver: dir
profiles:
- config: {}
  description: ""
  devices:
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null
EOF

Set variables for convenience

Note

The current stable release of OpenWRT can be found here.

export SHORT_VERSION=24.10.2
export LONG_VERSION="openwrt-${SHORT_VERSION}-x86-64-generic-ext4-combined-efi"

Download EFI combined ext4 image and extract

Note

A decompression OK, trailing garbage ignored error is expected due to a harmless bug in the image builder.

wget "https://downloads.openwrt.org/releases/${SHORT_VERSION}/targets/x86/64/${LONG_VERSION}.img.gz"
gunzip "${LONG_VERSION}.img.gz"

Convert raw image to qcow2 image

Tip

qemu-img can be installed via the qemu-utils package.

qemu-img convert -f raw -O qcow2 "${LONG_VERSION}.img" "${LONG_VERSION}.qcow2"

Create metadata file for image and compress

cat << EOF > metadata.yaml
---
architecture: x86_64
creation_date: $(date +%s)
properties:
    description: ${LONG_VERSION}
    os: openwrt
    release: ${SHORT_VERSION}
EOF
tar -cvzf metadata.tar.gz metadata.yaml

Import metadata and image to local lxc repository

incus image import metadata.tar.gz "${LONG_VERSION}.qcow2" --alias "openwrt/${SHORT_VERSION}"

Create an new profile for the instance

This profile is configured for 4GiB of memory, 2 CPUs, 4GiB root disk, setting the instance to autostart on boot and disabling secureboot as the EFI image of OpenWrt we're using is unsigned. You will want to modify the values for parent devices to match your device names.

Tip

For a full list of instance options, see here. If you're unsure what to set here, you can always change this later.

incus profile create multi-wan
incus profile edit multi-wan << EOF
---
config:
  limits.memory: 4GiB
  limits.cpu: 2,3
  boot.autostart: true
  security.secureboot: false
description: 3x NIC Passthrough
devices:
  eth0:
    name: eth0
    nictype: physical
    parent: eno2
    type: nic
  eth1:
    name: eth1
    nictype: physical
    parent: eno3
    type: nic
  eth2:
    name: eth2
    nictype: physical
    parent: eno4
    type: nic
  root:
    path: /
    pool: default
    type: disk
    size: 4GiB
name: multi-wan
used_by:
EOF

Initialize new instance derived from the multi-wan profile

Tip

If you didn't define any limits.* above, you can pass --type and specify an AWS, GCE, or Azure instance type.

incus launch --profile multi-wan --vm local:"openwrt/${SHORT_VERSION}" router

At this point, you'll probably want to ensure and verify that a network cable is physically connected to the ports on your device as you'll need a working Internet connection for the following steps.

Connect to console

Tip

Ctrl-a, q to escape

incus console router

Verify network connectivity

ping 1.1.1.1

Expand rootfs

The disk image ${LONG_VERSION}.img is converted into the root disk during lxc launch. Usually this leaves a lot of free space on the disk (after the first partition) which can be expanded to occupy the full size of the disk. This is most easily accomplished with a short script from the OpenWrt Wiki.

Tip

Read the contents of expand-root.sh before runnig this example: https://openwrt.org/docs/guide-user/advanced/expand_root

opkg update
opkg install parted losetup resize2fs

wget -O expand-root.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/expand_root?codeblock=0"
source ./expand-root.sh
sh /etc/uci-defaults/70-rootpt-resize

Update host machine networking defaults

Exclude your passthrough interfaces from being configured by the host machine. Multiple interfaces can be specified by separating them with a space inside the quotes.

/etc/default/networking

# Don't configure these interfaces. Shell wildcards supported.
EXCLUDE_INTERFACES="eno[234]"

Optional modifications

Reduce dhclient timers

If the host machine receives it's DHCP lease from its VM guest, you may also want to change dhclient's timeout and retry values so that the host machine doesn't enter into backoff before the guest is fully booted.

/etc/dhcp/dhclient.conf

timeout 30;
retry 30;

Review boot-related options

Tip

You can edit your existing profile with lxc profile edit multi-wan

You may want to review the boot-related options in the LXD Documentation to control startup/shutdown and prioritization.

Troubleshooting

Broken Networking/Missing drivers

If you find that your network card is not natively supported by OpenWrt (as was the case with an Intel 7xx series I was working with), you may need to copy driver files into the virtual machine image manually. This may seem a little daunting but it's actually pretty simple.

Note

The filesystem image is located at: /var/lib/incus/storage-pools/default/virtual-machines/router/root.img. All of the commands below assume you are in this directory.

List the partitions in the filesystem image:

fdisk -l root.img

Here we see three partitions, the root filesystem is the second partition that is "4G" starting at a 33280 "sector" offset from the beginning of the file.

Disk root.img: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E9BAD6FA-7D1C-4DD0-C46B-4B84E666DF00

Device      Start     End Sectors  Size Type
root.img1     512   33279   32768   16M Linux filesystem
root.img2   33280 8388574 8355295    4G Linux filesystem
root.img128    34     511     478  239K BIOS boot

Partition table entries are not in disk order.

Mount the image on a loopback device

losetup --find --partscan root.img

Verify the image is mounted

losetup -l 

NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                                                             DIO LOG-SEC
/dev/loop0         0      0         0  0 /var/lib/incus/storage-pools/default/virtual-machines/router/root.img   0     512

Mount the image

Debian's udev subsystem will create a device file for the second partition automatically when you mount the image onto a loopback device using losetup. We will create a /mnt/image directory to use as the mount point.

mkdir /mnt/image
mount /dev/loop0p2 /mnt/image

Verify the image mounted correctly

You can now list the contents of the image and verify the root filesystem of the virtual machine image is mounted correctly.

ls /mnt/image

Download the driver

OpenWrt builds a lot of compatible drivers that aren't automatically included in every release but still available for download as .ipk packages.

Download the relevant .ipk for your network card to root's home directory in your image.

wget 'https://downloads.openwrt.org/releases/24.10.1/targets/x86/64/kmods/6.6.86-1-af351158cfb5febf5155a3aa53785982/kmod-i40e_6.6.86-r1_x86_64.ipk' -O /mnt/image/root

Unmount the disk

Now, safely unmount the disk and remove the loopback device.

umount /mnt/image
losetup -d /dev/loop0

Start the virtual machine and install the driver

Start the virtual machine and use the console to install the .ipk package. Reboot the virtual machine and verify your network card is now working.

incus start router
incus console router
opkg install /root/kmod-i40e_6.6.86-r1_x86_64.ipk
reboot
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment