Jump to content
L0rdRaiden

New method for passthrough devices in Unraid 6.7 (vfio/BIND)

8 posts in this topic Last Reply

Recommended Posts

With the new release, Unraid 6.7

There is a new method to passthrough devices

Quote

New vfio-bind method.  Since it appears that the xen-pciback/pciback kernel options no longer work, we introduced an alternate method of binding, by ID, selected PCI devices to the vfio-pci driver. This is accomplished by specifying the PCI ID(s) of devices to bind to vfio-pci in the file 'config/vfio-pci.cfg' on the USB flash boot device.  This file should contain a single line that defines the devices:

BIND=<device> <device> ...

Where <device> is a Domain:Bus:Device.Function string, for example,

BIND=02:00.0

Multiple device should be separated with spaces.

 

The script /usr/local/sbin/vfio-pci is called very early in system start-up, right after the USB flash boot device is
mounted but before any kernel modules (drivers) have been loaded.  The function of the script is to bind each specified device to the vfio-pci driver, which makes them available for assignment to a virtual machine, and also prevents the Linux kernel from automatically binding them to any present host driver. 

 

In addition, and importantly, this script will bind not only the specified device(s), but all other devices in the same IOMMU group as well.  For example, suppose there is an NVIDIA GPU which defines both a VGA device at 02:00.0 and an audio device at 02.00.1.  Specifying a single device (either one) on the BIND line is sufficient to bind both device to vfio-pci.  The implication is that either all devices of an IOMMU group are bound to vfio-pci or none of them are. 

 

Right now most of us I guess have something like this in the Syslinux configuration since is the dicussed method in forum and some youtube videos

imagen.thumb.png.9ef5ab519a10b12c3f2ad96e083ae29f.png

And with this options enable

imagen.png.086919a31575e0f534f5a117a41d8cf0.png

 

So considering the new method and my current configuration to passthourgh a pci network card, what should I add, remove, keep of my current settings

@limetech since is a new feature maybe someone from the team could explain it in detail for the KVM noobs.

Share this post


Link to post

You don't have to do anything if using vfio-pci.ids. This is for those that use xen-pciback.hide with the PCI ID in case they have two (or more) cards with the same vendor:device ID and only want one to be hidden (USB controller).

Share this post


Link to post
3 hours ago, saarg said:

You don't have to do anything if using vfio-pci.ids. This is for those that use xen-pciback.hide with the PCI ID in case they have two (or more) cards with the same vendor:device ID and only want one to be hidden (USB controller).

So if someone was going to be setting up some devices for PT in a vanilla system, nothing PT'd yet. Should we use this new method, or the example above where it's appended in the syslinux config?

Share this post


Link to post

To expand on @saarg explanation: the "vfio-pci.ids" kernel parameter specifies devices that the Linux kernel should not try to initialize or assign to a driver, because doing so sometimes makes the device behave improperly when assigned to a VM (or makes it impossible to assign).  This parameter identifies devices using "Vendor:Model" strings where Vendor and Model are numeric values assigned by the device manufacturer.  This is easy because that string will never change.  The disadvantage is if you have two or more of the exact same device in your server, then all of them will be "invisible" to Linux.

 

The other kernel parameter available was "xen-pciback.hide" (and synonym "pciback") which accomplishes the same thing, but takes a string of the form "Domain:Bus:Device.Function".  Each of those values are also numbers that identify the device according to where it exists in your server PCI bus topology.  The advantage with this method is that an exact device can be identified independent of whether another of the same device exists in the server.  The disadvantage with this method is that if the physical h/w configuration changes, e.g., you move the device to a different PCI slot, then the PCI-ID of that device also changes.

 

The problem we ran across was that xen-pciback.hide/pciback wasn't working any more, and rather than wait for a kernel dev to fix it, we decided to use the above alternate method.  Note that in general, kernel evolution is moving away from kernel parameters and to more flexible methods.  For example, "isolcpus" is really deprecated and there is alternate method of isolation CPU's using config files which we will adopt in a future Unraid OS release.

 

As you can see, there is currently no "perfect" way of maintaining permanent assignment of devices to VM's.

Share this post


Link to post
3 minutes ago, limetech said:

To expand on @saarg explanation: the "vfio-pci.ids" kernel parameter specifies devices that the Linux kernel should not try to initialize or assign to a driver, because doing so sometimes makes the device behave improperly when assigned to a VM (or makes it impossible to assign).  This parameter identifies devices using "Vendor:Model" strings where Vendor and Model are numeric values assigned by the device manufacturer.  This is easy because that string will never change.  The disadvantage is if you have two or more of the exact same device in your server, then all of them will be "invisible" to Linux.

So, in my planning, I am going to be passing 2 x identical USB 3.1 PCIE x 4 expansion cards, and 2 x identical Zotac GTX 1080 mini GPU's through. The goal is to have one GPU and one USB card PT's to each VM, total of two VM's. Since I will have 2 x identical sets of matching hardware, I assume I should use this new method?

Share this post


Link to post
1 hour ago, cybrnook said:

So, in my planning, I am going to be passing 2 x identical USB 3.1 PCIE x 4 expansion cards, and 2 x identical Zotac GTX 1080 mini GPU's through. The goal is to have one GPU and one USB card PT's to each VM, total of two VM's. Since I will have 2 x identical sets of matching hardware, I assume I should use this new method?

Either method should work, note all it does is tell the kernel not to bind any driver to the device(s), though actually it does bind vfio stub driver (this is what prevents other drivers from binding).  Another consideration is the IOMMU grouping.  You can't pass two devices from the same IOMMU group to two different VM's unless you use the ACS override function.  Using ACS override is "ok" in most Unraid applications because usually you are in complete control over what VM's are running and what is running in each VM.  The cloud-server guys can't use ACS override because doing so (theoretically) allows one VM to gain access to another VM running on same hardware - this is why the ACS override patch will never be accepted into the official kernel source tree.  Personally, in this age we live in where everyone is extremely security-conscious, I would only use modern h/w with proper ACS support.

Share this post


Link to post

I have this in my syslinux.cfg file...

 

label KVM with GUI/unRAID OS
  menu default
  kernel /bzimage
  append vfio-pci.ids=1b21:1142 intel_iommu=on pcie_acs_override=downstream,multifunction initrd=/bzroot,/bzroot-gui

 

which worked great until about rc7, and now I cannot get my VM to recognize my USB keyboard, or any hard drives I connect to the USB port.

 

Do these changes have anything to do with my new issue, or is it just coincidence and I need to look elsewhere for a solution to this new problem?

 

if so, where might I look next?

 

Thanks

media-diagnostics-20190511-1957.zip

Share this post


Link to post
6 hours ago, JustinChase said:

I have this in my syslinux.cfg file...

 

label KVM with GUI/unRAID OS
  menu default
  kernel /bzimage
  append vfio-pci.ids=1b21:1142 intel_iommu=on pcie_acs_override=downstream,multifunction initrd=/bzroot,/bzroot-gui

 

which worked great until about rc7, and now I cannot get my VM to recognize my USB keyboard, or any hard drives I connect to the USB port.

 

Do these changes have anything to do with my new issue, or is it just coincidence and I need to look elsewhere for a solution to this new problem?

 

if so, where might I look next?

 

Thanks

media-diagnostics-20190511-1957.zip 94.63 kB · 0 downloads

That issue have nothing to do with the new method. Probably best to open a new thread about the issue.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.