Jump to content
ljm42

Guide: bind devices to vfio-pci for easy passthrough to VMs

2 posts in this topic Last Reply

Recommended Posts

At times you will want to "hide" devices from Unraid so that they can be passed through to a VM.

 

Unraid Prior to 6.7
In the past (pre Unraid 6.7) we would stub the device by adding a Vendor:Device code to the vfio-pci.ids parameter in Syslinux, something like this:

append vfio-pci.ids=8086:1533

This worked, but had several downsides:

  • If you have multiple devices with the same Vendor:Device code, all of them would be stubbed (hidden) from Unraid
  • It is a fairly technical process to find the right Vendor:Device code and modify the syslinux file. Make a mistake and your system won't boot!


As an alternative, you could add the <Domain:Bus:Device.Function> string to the xen-pciback.hide parameter in Syslinux:

append xen-pciback.hide=0000:03:00.0

This had downsides too:

  • Still a technical / risky process
  • If you add/remove hardware after modifying syslinux, the pci address could change and the wrong device could end up being stubbed. This would cause problems if a critical disk controller or NIC were suddenly hidden from Unraid
  • This broke in Unraid 6.7. More details

 

Unraid 6.7
Starting with Unraid 6.7 we could bind devices to the vfio-pci driver based on the <Domain:Bus:Device.Function> string (aka pci address). You needed to manually modify the config/vfio-pci.cfg file and specify the <Domain:Bus:Device.Function> string, like this:

BIND=03:00.0

This worked, but still had several downsides:

  • It was a fairly technical process to find the right string to place in the file. But at least if anything went wrong you could simply delete the config file off the flash drive and reboot.
  • We still had the problem where if you add/remove hardware after modifying the file, the pci addresses could change and the wrong device could end up being bound to vfio-pci


Unraid 6.9

For Unraid 6.9, Skittals has incorporated the excellent "VFIO-PCI Config" plugin directly into the Unraid webgui. So now from the Tools -> System Devices page you can easily see all of your hardware and which IOMMU groups they are in. Rather than editing the config file by hand, simply add a checkbox next to the devices that you want to bind to vfio-pci (aka hide from Unraid). If a device is being used by Unraid (such as a USB controller, disk controller, etc) then the web interface will prevent you from selecting it.

Additionally, we have a new version of the underlying vfio-pci script which can prevent the wrong devices from being bound when hardware is added or removed. When you click to bind a device on the System Devices page, it will write both the <Domain:Bus:Device.Function> and the <Vendor:Device> code to the config file, like this:

BIND=0000:03:00.0|8086:1533

In this example, the updated script will bind the device at pci address 0000:03:00.0, but only if the <Vendor:Device> code is 8086:1533. If a different <Vendor:Device> code is found at that address, it will not bind. This means we will never inadvertently bind a device that is important to Unraid! (However, since the desired device is not available to be bound, the VM expecting that device may not function correctly.)

Devices bound in this way can be passed through to your VMs by going to the VM tab, editing the template, and then selecting the appropriate device from one of the hardware dropdowns. Can't find it? Check under "Other PCI Devices".

If the System Devices page shows that multiple devices are in the same IOMMU group, it will automatically bind all the devices in that group to vfio-pci.  You should then pass all devices in that IOMMU group to the same VM.

Note: If you make hardware changes after setting this up, it would be a good idea to disable autostart on your VMs first. Then shutdown, add/remove hardware as needed, and boot back into Unraid. Visit the Tools -> System Devices page and ensure the correct devices are still being bound to vfio-pci. Adjust as needed and reboot, then start your VMs.

Troubleshooting Tips

  • If you had the VFIO-PCI Config plugin installed, you should remove it as that functionality is now built-in to Unraid 6.9
  • The System Devices page writes the device details to config/vfio-pci.cfg file on the flash drive. If you ever want to "start fresh" simply delete this file and reboot.
  • The /var/log/vfio-pci log file details each of the devices that were (un)successfully bound during boot, along with any available error messages.
  • Be sure to upload your diagnostics when requesting help as both the config file and the log are included in it

 

 

Hopefully this is helpful :) Feel free to let me know in the comments if anything is unclear or wrong.

 

 

  • Like 3
  • Thanks 6

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.