Jump to content
Skitals

[Plugin] VFIO-PCI Config

53 posts in this topic Last Reply

Recommended Posts

1 hour ago, L0rdRaiden said:

I have this right now in my syslinux file

 

 

If I want to use this new method what do I need to remove from here? just this (vfio-pci.ids=8086:1521) or something else?

 

I'm doing passthrough of a pcie network card with 2 nics.

 

If I do the passthrough using this method (vfio-pci.cfg) all the other options of my syslinux config file still applies?

 

Thanks in advance.

If you have it working now, I wouldn't change anything. Unless you absolutely want to have a click and set way of doing it.

 

You only need to remove vfio-pci.ids=8086:1521.

The rest of the options you have in syslinux.cfg still applies.

Share this post


Link to post
8 minutes ago, saarg said:

If you have it working now, I wouldn't change anything. Unless you absolutely want to have a click and set way of doing it.

 

You only need to remove vfio-pci.ids=8086:1521.

The rest of the options you have in syslinux.cfg still applies.

But I understand that with this method I can bypass 1 of the 2 nics in the pcie card, and with the syslinux config I can only bypass both at the same time. Right?

Share this post


Link to post
10 minutes ago, L0rdRaiden said:

But I understand that with this method I can bypass 1 of the 2 nics in the pcie card, and with the syslinux config I can only bypass both at the same time. Right?

That depends on the network card.

 

Try it and see if it works.

Share this post


Link to post

This app worked great for my scenario.


-I have a Asus ROG Maximus XI Hero (wifi) with 2 NVMe M2 slots.

-One is my Array's Cache device, but i want the other passed-trough to a vm

-When I stub it manual on unraid  flash with "vfio-pci.ids=xxxx" i lose both.
-If i use the plugin, it stubs it on device address (which are unique for the 2 slots)

-Now i use that 2nd NVM-e to baremetal boot my windows PC or boot Unraid an use it on My VM Windows PC

Perfect for me because im having troubles finding a way to do it manual and now its just one ciick!

Thank you

Share this post


Link to post
Posted (edited)
On 4/10/2020 at 6:50 PM, saarg said:

That depends on the network card.

 

Try it and see if it works.

Hi,

I have a x520 pcie network card with 2 nics, The vfio-pci.ids method worked fine (I passed through those to xpenology). But when I try this plugin, the xpenology can no longer get an ip address. I checked lspci -v in unraid and it seems that unraid loaded the network card with vfio-pci, so I suppose they should be passed through.

07:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Netwo                                             rk Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2
        Flags: fast devsel, IRQ 11
        Memory at f2100000 (64-bit, non-prefetchable) [disabled] [size=512K]
        I/O ports at 9020 [disabled] [size=32]
        Memory at f2200000 (64-bit, non-prefetchable) [disabled] [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable- Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-ba-e9-ba
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: vfio-pci
        Kernel modules: ixgbe

07:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Netwo                                             rk Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter X520-2
        Flags: bus master, fast devsel, latency 0, IRQ 17
        Memory at f2180000 (64-bit, non-prefetchable) [size=512K]
        I/O ports at 9000 [size=32]
        Memory at f2404000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-ba-e9-ba
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: vfio-pci
        Kernel modules: ixgbe

And here is the output of lspci -v in xpenology(I select only one nic to xpenology):

0000:04:00.0 Class 0200: Device 8086:10fb (rev 01)
        Subsystem: Device 8086:000c
        Flags: bus master, fast devsel, latency 0, IRQ 23
        Memory at 98200000 (64-bit, non-prefetchable) [size=512K]
        I/O ports at 7000 [size=32]
        Memory at 98280000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-ba-e9-ba
        Kernel driver in use: ixgbe

I'm kind of confused. It seems both unraid and xpenology is showing the driver correctly, but no ip address for the nic. Everything worked just fine using vfio-pci.ids method but I just want to try if I could pass only one nic and leave the other one.

I will attach the diagnostics below.

 

I made some further effort trying the xen-pciback.hide method, which uses pci ids instead of vendor ids so that I could pass only one of the two nics. And it works. So i guess this might be some bug in vfio-pci config plugin.

tower-diagnostics-20200424-2248.zip

Edited by sarieri
adding diagnostic

Share this post


Link to post

Thank you for this excellent plugin.. wish I would've seen this before I started messing around with my VMs!

Share this post


Link to post

How do I undo changes made by the plugin? My unraid server will not boot anymore

Share this post


Link to post
13 hours ago, Cliff said:

How do I undo changes made by the plugin? My unraid server will not boot anymore

If it won't boot at all, then you'll have to pull the stick and edit /syslinux/syslinux.cfg on another computer and remove the vfio section the plugin added.

 

Or boot via GUI mode (or vice versa), and then via Main, Boot Device, click on flash then syslinux configuration and edit the file there.

Share this post


Link to post

I used the vfio-config plugin to get a 4 port NIC to pass through. 

 

I had to enable acs override to get the IOMMU groups to break up first.  Then I used the plugin to create the cfg file. 

These are the for ports I am trying to pass through

IOMMU group 25:	[8086:10bc] 08:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 26:	[8086:10bc] 08:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 27:	[8086:10bc] 09:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 28:	[8086:10bc] 09:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

 

The config file adds all 4 of them

 

BIND=08:00.0 08:00.1 09:00.0 09:00.1

 

But I only get 2 ports to show in my VM

 

image.png.9af9507205d622e73d2ecddfc296cf30.png

 

When I look in the plugin I only have a green orb next to the ones that are being passed through.

 

image.thumb.png.d0ca02839d62494f7fee243f132d96c4.png

 

Any idea how I can get all 4 ports to pass through?

Share this post


Link to post

Is there any specific thing to look for to see why a device chosen in the plugin doesn't bind to vfio, and as such, is not available to be selected in a VM?  So far, I have not been able to pass through anything other than my video card (and it's associated sound).  After trying differnet ACS settings, the card I'm wanting to pass through is grouped with just a PCI bridge (guessing the one it is routed through), so when they are all checked, they still don't bind.

Share this post


Link to post

This broke UnRAID for me.

 

Setup:

  • Ryzen 5 2600X
  • MSI B450m Mortar
  • 2x16Gb RAM
  • 3 Drives via onboard SATA
  • 4 Drives via Dell H310 IT-mode
  • 1 NVME drive

Tried to pass through one of the two onboard USB controllers that has a USB hub with Keyboard and mouse connected. It is in it's own IOMMU group and can be passed through without a problem using the "vfio-pci-ids=" method.

Unraid fails to load with text about the USB controller on the screen, no explicit error messages. (Didn't take a picture, sorry). Fixed by removing /config/vfio-pci.cfg on a Windows machine.

Share this post


Link to post

Hey, thx for this great plugin! I have usb ports on separate groups but they have same pci.id. With your plugin i could easily passthrough one! Now i went to 6.9 because of the core temps update. Any chance you will update to be compatible with new unraid version?

Share this post


Link to post
4 hours ago, Dargor said:

Hey, thx for this great plugin! I have usb ports on separate groups but they have same pci.id. With your plugin i could easily passthrough one! Now i went to 6.9 because of the core temps update. Any chance you will update to be compatible with new unraid version?

The plugin is included in 6.9 beta22+ and should not be installed in those versions

Share this post


Link to post
2 hours ago, Squid said:

The plugin is included in 6.9 beta22+ and should not be installed in those versions

Found it I would never have checked unless I changed hardware. Thx a lot!

Share this post


Link to post

I used this plugin to pass through my nvme controller/drive and created a new Windows VM (selected 'None' for vdisk).  The install finished, but on every VM startup, the "press any key to boot from DVD..." still shows then I have to type exit at the next prompt, then navigate the emulated BIOS to manually select the boot drive to boot into Windows.  How do I fix the VM config so that it boots into Windows normally?

Share this post


Link to post

First of all, thanks for creating this plugin. This is excellent!

 

I have a question how the binding works. I noticed that my GPU is bound (tick shown in the plugin with the GPU, no tick with the audio card):

 

Group 36 65:00.010de:1c03VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)

 65:00.110de:10f1Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)

 

I passthrough this card to a Windows VM. I don't remember that I had set this "tick". Is it needed / helpful?

Share this post


Link to post
On 7/5/2020 at 6:49 PM, steve1977 said:

First of all, thanks for creating this plugin. This is excellent!

 

I have a question how the binding works. I noticed that my GPU is bound (tick shown in the plugin with the GPU, no tick with the audio card):

 

Group 36 65:00.010de:1c03VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)

 65:00.110de:10f1Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)

 

I passthrough this card to a Windows VM. I don't remember that I had set this "tick". Is it needed / helpful?

 

The reason for "adding a tick" to a device is to ensure Unraid doesn't install any drivers for it, thus leaving the device available to be passed through to a VM. I'd guess that Unraid doesn't load audio drivers right now so it isn't strictly necessary to bind the audio device to vfio-pci.  But I'd probably add the tick anyhow, in case something changes in the future.

Share this post


Link to post
On 4/5/2020 at 12:38 PM, ljm42 said:

Sure, on the flash drive delete the file /config/vfio-pci.cfg

 

 

 

 

Than you for this tip, it just saved my bacon!

 

Share this post


Link to post
On 5/12/2020 at 5:33 PM, Armed Ferret said:

I used the vfio-config plugin to get a 4 port NIC to pass through. 

 

I had to enable acs override to get the IOMMU groups to break up first.  Then I used the plugin to create the cfg file. 

These are the for ports I am trying to pass through


IOMMU group 25:	[8086:10bc] 08:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 26:	[8086:10bc] 08:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 27:	[8086:10bc] 09:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)
IOMMU group 28:	[8086:10bc] 09:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06)

 

The config file adds all 4 of them

 


BIND=08:00.0 08:00.1 09:00.0 09:00.1

 

But I only get 2 ports to show in my VM

 

image.png.9af9507205d622e73d2ecddfc296cf30.png

 

When I look in the plugin I only have a green orb next to the ones that are being passed through.

 

image.thumb.png.d0ca02839d62494f7fee243f132d96c4.png

 

Any idea how I can get all 4 ports to pass through?

I don't know what would cause vfio-pci to skip devices, but as part of migrating this functionality to Unraid 6.9, everything vfio-pci does during bootup is now logged.  More info here:
  https://forums.unraid.net/topic/93781-guide-bind-devices-to-vfio-pci-for-easy-passthrough-to-vms/

 

Check it out when 6.9 is released, or try a beta if you are feeling adventurous:
  https://forums.unraid.net/bug-reports/prereleases/ 

Share this post


Link to post

screwed up and passed through my usb controller, with the unraid startup on it.  Any advice how to get out of this??

 

Share this post


Link to post
1 hour ago, 22350 said:

screwed up and passed through my usb controller, with the unraid startup on it.  Any advice how to get out of this??

Delete the file config/vfio-pci.cfg from the flashdrive and reboot. 

Share this post


Link to post
On 9/8/2020 at 8:21 PM, ljm42 said:

Delete the file config/vfio-pci.cfg from the flashdrive and reboot. 

that did the trick

 

Share this post


Link to post
On 9/8/2020 at 10:21 PM, ljm42 said:

Delete the file config/vfio-pci.cfg from the flashdrive and reboot. 

 

Will that remove the plugin also or just the settings? 

 

I'm sure this will be a stupid question I've only been with Unraid for about a month.........how do you delete it that file on the flashdrive? When I go to "main" then to the files on the flash drive it wont let me delete it. If I try to open it in Krusader it wont let me access the files on the flash drive. Should I just power down the server, put the usb drive in my pc, delete that file, then bring the server back up?

Edited by SPOautos

Share this post


Link to post

Hey guys,  I have a thread, where I am trying to solve a passthrough problem.  If you have a second, could you take a look??

 

thanks 

 

 

Share this post


Link to post
On 9/30/2020 at 10:56 AM, SPOautos said:

 

Will that remove the plugin also or just the settings? 

 

I'm sure this will be a stupid question I've only been with Unraid for about a month.........how do you delete it that file on the flashdrive? When I go to "main" then to the files on the flash drive it wont let me delete it. If I try to open it in Krusader it wont let me access the files on the flash drive. Should I just power down the server, put the usb drive in my pc, delete that file, then bring the server back up?

In Windows, go to the search bar and type \\tower (or whatever the name of your server is). Then navigate to the "flash" share, then the "config" folder, and finally delete the vfio-pci.cfg file. If you don't use Windows, hopefully this will be enough to point you in the right direction.

 

Deleting this file will just remove the settings. To remove the plugin, go to http://tower (or whatever the name of your server is) then go to the Plugins page and remove it from there.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.