Jump to content
archedraft

***GUIDE*** Passthrough Entire PCI USB Controller

480 posts in this topic Last Reply

Recommended Posts

Posted (edited)
On 1/2/2016 at 10:57 PM, saarg said:

For me it's much easier to use the hostdev tag instead of the qemu:arg. It's just too much trouble with it when you don't know what you are doing.

So for passing through a PCI(e) device with the hostdev tag this is the starting code:

 

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
 

 

 

You then have to modify some parts of it to get it to work. The good thing is that you do not have to care about which bus and address it's supposed to have in the VM. You only need to find out the host PCI address.

The part you change is bus, slot and function. In your case it's 00:14.0.

Let's brake it down.

00 is the bus. You simply exchange the two numbers after the 0x.

14 is the slot. Same method as above.

0 is the function. Her it's also the same method as above.

So in your case the full device tag would be like this:

 

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
      </source>
    </hostdev>
 

 

 

After you start the VM you will see that there are some lines added to the tag, but those you don't have to care about. They get created automatically. If you copy a host device tag to pass through a new device, be sure to remove the two lines created after the


</source>
 

tag, as they are specific to that VM.

 

 


      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
 

 

I've got my external USB hub on the controller at 43:00.0, the bus is 07 and the port is 0.

My XML looks like this: 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
    </hostdev>

 

When starting the VM, I get this error: 

Device 0000:07:00.0 not found: could not access /sys/bus/pci/devices/0000:07:00.0/config: No such file or directory

Also noted is the fact that it doesn't show up in /sys/bus/pci/devices in an ls (see attached)

 

I've also attached proof of the bus (the second screenshot).

 

Update: that device is a dummy: 

ot:   00:07.0
Class:  Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
NUMANode:       0

 

And the real device is 43:00.3, but unRAID doesn't let me type over 1F (43=2B)

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x2B' function='0x3'/>
      </source>
    </hostdev>

 

Note: upon changing the second slot (the automatically generated one after </source> to 0x00 errors out with an XML error saying it needs to be >=1.

Screen Shot 2019-08-05 at 12.56.30.png

Screen Shot 2019-08-05 at 12.57.55.png

Edited by dxfydd

Share this post


Link to post

Hey guys,

 

I have a Gigabyte Aorus Gaming K7 (AX370) Mainboard and Unraid shows only 12 IOMMU groups:

 

PCI Devices and IOMMU Groups

IOMMU group 0:	[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 1:	[1022:1483] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 2:	[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 3:	[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 5:	[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 6:	[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 7:	[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
[1022:148a] 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 8:	[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
[1022:1485] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
[1022:1486] 09:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
[1022:149c] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
[1022:1487] 09:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
IOMMU group 9:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 10:	[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
IOMMU group 11:	[1022:43b9] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller (rev 02)
[1022:43b5] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller (rev 02)
[1022:43b0] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset PCIe Upstream Port (rev 02)
[1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1b21:1343] 03:00.0 USB controller: ASMedia Technology Inc. ASM1143 USB 3.1 Host Controller
[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
[1969:e0b1] 05:00.0 Ethernet controller: Qualcomm Atheros Killer E2500 Gigabit Ethernet Controller (rev 10)
IOMMU group 12:	[10de:1c03] 07:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
[10de:10f1] 07:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)

My goal is to pass a USB Controller to a VM, but each of the 3 USB Controllers has other devices in its group.. In BIOS I enabled ACS (is this correct?). GPU Passes just fine.. Am I trapped with the USB Controllers? Are there any other possibilities (did I miss some BIOS option?), is my last resort some Kernel Patch?

 

Thanks for your advice!

Share this post


Link to post
On 10/18/2019 at 7:01 PM, glockmane said:

Hey guys,

 

I have a Gigabyte Aorus Gaming K7 (AX370) Mainboard and Unraid shows only 12 IOMMU groups:

 

My goal is to pass a USB Controller to a VM, but each of the 3 USB Controllers has other devices in its group.. In BIOS I enabled ACS (is this correct?). GPU Passes just fine.. Am I trapped with the USB Controllers? Are there any other possibilities (did I miss some BIOS option?), is my last resort some Kernel Patch?

 

Thanks for your advice!

For my x570 Aorus Elite Wifi in BIOS I had to enable ACS Enable and Enable AER Cap to get decent IOMMU grouping. After that I STILL had to set PCIe ACS override to "both" in Unraid. I was able to pass 2 out of 3 usb controllers on my board, including the front panel gen 2 USB Type C!

 

Initially I only had one controller in it's own IOMMU group. That controller would not/could not pass, it would lock up Unraid when I started the VM. After the above changes,  same story for that controller. I switched my Unraid usb to a port on that controller. The other two controllers (now that they are isolated) pass fine!

 

I'm using /boot/config/vfio-pci.cfg to bind those PCI IDs. The USB controllers now show up under "Other PCI Devices" so I can pass them to VMs without editing any XML.

 

Edited by Skitals

Share this post


Link to post
On 11/12/2019 at 4:52 PM, Skitals said:

For my x570 Aorus Elite Wifi in BIOS I had to enable ACS Enable and Enable AER Cap to get decent IOMMU grouping. After that I STILL had to set PCIe ACS override to "both" in Unraid. I was able to pass 2 out of 3 usb controllers on my board, including the front panel gen 2 USB Type C!

 

Initially I only had one controller in it's own IOMMU group. That controller would not/could not pass, it would lock up Unraid when I started the VM. After the above changes,  same story for that controller. I switched my Unraid usb to a port on that controller. The other two controllers (now that they are isolated) pass fine!

 

I'm using /boot/config/vfio-pci.cfg to bind those PCI IDs. The USB controllers now show up under "Other PCI Devices" so I can pass them to VMs without editing any XML.

 

I did the same (dunno if I enabnled AER Cap, have to look later), but I have ACS Enable and ACS Override set as you said and getting much better IOMMU Groups.. Binding like you said in your last sentence looks interesting, can you post the section from your .cfg? Thank you :)

Share this post


Link to post
On 11/13/2019 at 11:07 AM, glockmane said:

I did the same (dunno if I enabnled AER Cap, have to look later), but I have ACS Enable and ACS Override set as you said and getting much better IOMMU Groups.. Binding like you said in your last sentence looks interesting, can you post the section from your .cfg? Thank you :)

I created the file /boot/config/vfio-pci.cfg that simply contains:

 

Quote

BIND=06:00.3 06:00.1 06:00.1

Any device you list there is stubbed and available to select under "other pci devices" on the vm xml gui. My understanding is this method supercedes pci-stub and vfio-pci boot flags. Note that if you add or remove pci devices to your system the addresses can/will change. I had to update my vfio-pci.cfg and vm xmls when I pulled my second gpu.

Edited by Skitals

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.