Jump to content
archedraft

***GUIDE*** Passthrough Entire PCI USB Controller

497 posts in this topic Last Reply

Recommended Posts

On 1/2/2016 at 10:57 PM, saarg said:

For me it's much easier to use the hostdev tag instead of the qemu:arg. It's just too much trouble with it when you don't know what you are doing.

So for passing through a PCI(e) device with the hostdev tag this is the starting code:

 

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
 

 

 

You then have to modify some parts of it to get it to work. The good thing is that you do not have to care about which bus and address it's supposed to have in the VM. You only need to find out the host PCI address.

The part you change is bus, slot and function. In your case it's 00:14.0.

Let's brake it down.

00 is the bus. You simply exchange the two numbers after the 0x.

14 is the slot. Same method as above.

0 is the function. Her it's also the same method as above.

So in your case the full device tag would be like this:

 

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
      </source>
    </hostdev>
 

 

 

After you start the VM you will see that there are some lines added to the tag, but those you don't have to care about. They get created automatically. If you copy a host device tag to pass through a new device, be sure to remove the two lines created after the


</source>
 

tag, as they are specific to that VM.

 

 


      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
 

 

I've got my external USB hub on the controller at 43:00.0, the bus is 07 and the port is 0.

My XML looks like this: 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
    </hostdev>

 

When starting the VM, I get this error: 

Device 0000:07:00.0 not found: could not access /sys/bus/pci/devices/0000:07:00.0/config: No such file or directory

Also noted is the fact that it doesn't show up in /sys/bus/pci/devices in an ls (see attached)

 

I've also attached proof of the bus (the second screenshot).

 

Update: that device is a dummy: 

ot:   00:07.0
Class:  Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
NUMANode:       0

 

And the real device is 43:00.3, but unRAID doesn't let me type over 1F (43=2B)

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x2B' function='0x3'/>
      </source>
    </hostdev>

 

Note: upon changing the second slot (the automatically generated one after </source> to 0x00 errors out with an XML error saying it needs to be >=1.

Screen Shot 2019-08-05 at 12.56.30.png

Screen Shot 2019-08-05 at 12.57.55.png

Edited by dxfydd

Share this post


Link to post

Hey guys,

 

I have a Gigabyte Aorus Gaming K7 (AX370) Mainboard and Unraid shows only 12 IOMMU groups:

 

PCI Devices and IOMMU Groups

IOMMU group 0:	[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 1:	[1022:1483] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 2:	[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 3:	[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4:	[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 5:	[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 6:	[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 7:	[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
[1022:148a] 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 8:	[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
[1022:1485] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
[1022:1486] 09:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
[1022:149c] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
[1022:1487] 09:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
IOMMU group 9:	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 10:	[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
IOMMU group 11:	[1022:43b9] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller (rev 02)
[1022:43b5] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller (rev 02)
[1022:43b0] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset PCIe Upstream Port (rev 02)
[1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)
[1b21:1343] 03:00.0 USB controller: ASMedia Technology Inc. ASM1143 USB 3.1 Host Controller
[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
[1969:e0b1] 05:00.0 Ethernet controller: Qualcomm Atheros Killer E2500 Gigabit Ethernet Controller (rev 10)
IOMMU group 12:	[10de:1c03] 07:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
[10de:10f1] 07:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)

My goal is to pass a USB Controller to a VM, but each of the 3 USB Controllers has other devices in its group.. In BIOS I enabled ACS (is this correct?). GPU Passes just fine.. Am I trapped with the USB Controllers? Are there any other possibilities (did I miss some BIOS option?), is my last resort some Kernel Patch?

 

Thanks for your advice!

Share this post


Link to post
On 10/18/2019 at 7:01 PM, glockmane said:

Hey guys,

 

I have a Gigabyte Aorus Gaming K7 (AX370) Mainboard and Unraid shows only 12 IOMMU groups:

 

My goal is to pass a USB Controller to a VM, but each of the 3 USB Controllers has other devices in its group.. In BIOS I enabled ACS (is this correct?). GPU Passes just fine.. Am I trapped with the USB Controllers? Are there any other possibilities (did I miss some BIOS option?), is my last resort some Kernel Patch?

 

Thanks for your advice!

For my x570 Aorus Elite Wifi in BIOS I had to enable ACS Enable and Enable AER Cap to get decent IOMMU grouping. After that I STILL had to set PCIe ACS override to "both" in Unraid. I was able to pass 2 out of 3 usb controllers on my board, including the front panel gen 2 USB Type C!

 

Initially I only had one controller in it's own IOMMU group. That controller would not/could not pass, it would lock up Unraid when I started the VM. After the above changes,  same story for that controller. I switched my Unraid usb to a port on that controller. The other two controllers (now that they are isolated) pass fine!

 

I'm using /boot/config/vfio-pci.cfg to bind those PCI IDs. The USB controllers now show up under "Other PCI Devices" so I can pass them to VMs without editing any XML.

 

Edited by Skitals

Share this post


Link to post
On 11/12/2019 at 4:52 PM, Skitals said:

For my x570 Aorus Elite Wifi in BIOS I had to enable ACS Enable and Enable AER Cap to get decent IOMMU grouping. After that I STILL had to set PCIe ACS override to "both" in Unraid. I was able to pass 2 out of 3 usb controllers on my board, including the front panel gen 2 USB Type C!

 

Initially I only had one controller in it's own IOMMU group. That controller would not/could not pass, it would lock up Unraid when I started the VM. After the above changes,  same story for that controller. I switched my Unraid usb to a port on that controller. The other two controllers (now that they are isolated) pass fine!

 

I'm using /boot/config/vfio-pci.cfg to bind those PCI IDs. The USB controllers now show up under "Other PCI Devices" so I can pass them to VMs without editing any XML.

 

I did the same (dunno if I enabnled AER Cap, have to look later), but I have ACS Enable and ACS Override set as you said and getting much better IOMMU Groups.. Binding like you said in your last sentence looks interesting, can you post the section from your .cfg? Thank you :)

Share this post


Link to post
On 11/13/2019 at 11:07 AM, glockmane said:

I did the same (dunno if I enabnled AER Cap, have to look later), but I have ACS Enable and ACS Override set as you said and getting much better IOMMU Groups.. Binding like you said in your last sentence looks interesting, can you post the section from your .cfg? Thank you :)

I created the file /boot/config/vfio-pci.cfg that simply contains:

 

Quote

BIND=06:00.3 06:00.1 06:00.1

Any device you list there is stubbed and available to select under "other pci devices" on the vm xml gui. My understanding is this method supercedes pci-stub and vfio-pci boot flags. Note that if you add or remove pci devices to your system the addresses can/will change. I had to update my vfio-pci.cfg and vm xmls when I pulled my second gpu.

Edited by Skitals

Share this post


Link to post

Has someone experienced problem passing a USB mouse - it works in Windows, but when I start a game and move the mouse - FPS drops from 250 to 50 back and forth. I'm passing USB Wi-Fi keyboard/mouse in 1 port, and a wired USB mouse (Lenovo Y) in another. When moving the Wi-Fi mouse - no issues, when moving wired - fps drop.

 

Tried another wired mouse (Razer) in the same port - it didn't have any issues.

 

Weird.

Share this post


Link to post

Try not passing the USB wireless adapter and just passing the one device to eliminate variables. It might be just that mouse then or it might be interactions

Share this post


Link to post
On 1/2/2020 at 10:47 AM, methanoid said:

Try not passing the USB wireless adapter and just passing the one device to eliminate variables. It might be just that mouse then or it might be interactions

The new mouse started doing the same an hour ago...

Share this post


Link to post

If I change the cpufreq governor to "performance", the fps drop still exists, but from ~350 to ~250.

With "ondemand" - from 300 to 50.

 

I can't figure out why this occurs. Perhaps I'll try to pass one of the USB controllers directly into the VM later.

Share this post


Link to post
1 minute ago, ChewbaccaBG said:

The new mouse started doing the same an hour ago...

I have no idea what you mean, wired or wireless. You dont seem very keen to actually tell us what we need to be able to help you. Also you decided to post your query on an UNRELATED thread about passing the whole controller NOT about passing a USB device. Maybe have a rethink about your approach and post a new thread with a little more expansive info and you might get a solution. The community here is helpful but it isn't (yet) psychic

Share this post


Link to post
Just now, ChewbaccaBG said:

If I change the cpufreq governor to "performance", the fps drop still exists, but from ~350 to ~250.

With "ondemand" - from 300 to 50.

 

I can't figure out why this occurs. Perhaps I'll try to pass one of the USB controllers directly into the VM later.


Good idea.. then at least you WOULD be the right thread and wouldnt need to post again (PS edit would have sufficed rather than another post)

Share this post


Link to post
Posted (edited)

So, this is the USB controller I'd like to pass to the VM:

3rd Gen AMD Ryzen™ Threadripper™ Desktop Processors
    - 2 x USB 3.2 Gen 2 ports (1 x Type-C™, 1 x Type A at back panel)
    - 2 x USB 3.2 Gen 1 ports at back panel

 

03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
../../../devices/pci0000:00/0000:00:08.1/0000:03:00.3/usb1
../../../devices/pci0000:00/0000:00:08.1/0000:03:00.3/usb2

 

IOMMU group 9:     [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 15:    [1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller

 

Since the 1st post is 5 years old, vfio-bind doesn't exist in Unraid 6.8..

What do I need to do?

 

Here's my current config (1 GPU passthrough):

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>f0f6d8a1-ff05-65df-7cd5-69d6624a3f9b</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>24</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='24'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='25'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='26'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='7' cpuset='27'/>
    <vcpupin vcpu='8' cpuset='4'/>
    <vcpupin vcpu='9' cpuset='28'/>
    <vcpupin vcpu='10' cpuset='5'/>
    <vcpupin vcpu='11' cpuset='29'/>
    <vcpupin vcpu='12' cpuset='6'/>
    <vcpupin vcpu='13' cpuset='30'/>
    <vcpupin vcpu='14' cpuset='7'/>
    <vcpupin vcpu='15' cpuset='31'/>
    <vcpupin vcpu='16' cpuset='8'/>
    <vcpupin vcpu='17' cpuset='32'/>
    <vcpupin vcpu='18' cpuset='9'/>
    <vcpupin vcpu='19' cpuset='33'/>
    <vcpupin vcpu='20' cpuset='10'/>
    <vcpupin vcpu='21' cpuset='34'/>
    <vcpupin vcpu='22' cpuset='11'/>
    <vcpupin vcpu='23' cpuset='35'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/f0f6d8a1-ff05-65df-7cd5-69d6624a3f9b_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='24' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='yes'/>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk2.img'/>
      <target dev='hdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.173.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='10' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:08:d9:27'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/boot/vBIOS/ASUS-ROG-STRIX-nVidia-GTX1050Ti-GP107-KVM.rom'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

Edited by ChewbaccaBG

Share this post


Link to post
12 minutes ago, ChewbaccaBG said:

So, this is the USB controller I'd like to pass to the VM:

3rd Gen AMD Ryzen™ Threadripper™ Desktop Processors
    - 2 x USB 3.2 Gen 2 ports (1 x Type-C™, 1 x Type A at back panel)
    - 2 x USB 3.2 Gen 1 ports at back panel

 

03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
../../../devices/pci0000:00/0000:00:08.1/0000:03:00.3/usb1
../../../devices/pci0000:00/0000:00:08.1/0000:03:00.3/usb2

 

IOMMU group 9:     [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 15:    [1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller

 

Since the 1st post is 5 years old, vfio-bind doesn't exist in Unraid 6.8..

What do I need to do?

 

Please do make efforts to read the thread not just give up after 1st post is 5 years old :( 

 

You have (sorry) posted only an incomplete section of your groupings (I suspect you have more than 2 controllers!). The principles are laid out here clearly in thread. Make sure you have separate groupings, make sure your unRAID USB is NOT on a controller you decide to pass. Then stub ALL the items in that group (Sorry if this isn't a spoonfeed answer but I'm not running unRAID now, so not sure of latest method but it IS in the thread) and then reboot server and in the VM template you will have those stubbed items appear as passable PCI devices. Pass all the bits and then you should be golden. 

Share this post


Link to post

Yes, I've checked the other groups, the unraid USB stick is not in the group I'd like to pass.

That group only has 4 ports directly connected to the CPU.

Share this post


Link to post
1 minute ago, ChewbaccaBG said:

Thank you so much @methanoid you helped fuck up all my VMs.

Now I guess I'd have to reinstall unraid completely just to get libvirt working agian.

 

THANK YOU

 

Please dont blame me when you failed to ever provide the requested IOMMU groupings. I even PM'd you to explain how you failed to help yourself. Nothing I have suggested would have led to your VMs being deleted. 

 

I note you editted your brief previous message to add some detail but still you never provide the requested info. You wont help yourself but you want to blame me. Good show. Good luck getting anyone else to help with that attitude.  Community here is good but being rude and deliberately difficulty is not the way to get help.

Share this post


Link to post
1 minute ago, ChewbaccaBG said:

Well, right now I ain't got any VMs at all so.. you've heard of any such problem ?

You could try posting your IOMMU groupings (not your VM XML yet) and your unRAID startup options (visible at the Flash sub screen). Maybe someone else will pipe up to help. I am done wasting my time trying to help and getting pissy posts. 

Share this post


Link to post
Just now, methanoid said:

You could try posting your IOMMU groupings (not your VM XML yet) and your unRAID startup options (visible at the Flash sub screen). Maybe someone else will pipe up to help. I am done wasting my time trying to help and getting pissy posts. 

You aren't even running unraid, as you said. Why were you trying to "help" anyway?

Share this post


Link to post
2 hours ago, ChewbaccaBG said:

You aren't even running unraid, as you said. Why were you trying to "help" anyway?

Maybe because he tries to be nice?

Share this post


Link to post
Just now, saarg said:

Maybe because he tries to be nice?

That's good.. In any case I solved the missing VM problem, now scrolling through the whole topic to find a way to pass the USB controller w/o crashing the server again.

Share this post


Link to post
Posted (edited)

So, here is it again, full information as was requested above.

 

All USB controller(s) are integrated on the mobo:
ASUSTeK COMPUTER INC. PRIME TRX40-PRO, Version Rev 1.xx, BIOS Version 0702 (latest)

 

Current syslinux config (8086:10c9 is a GPU):
kernel /bzimage
append amd_iommu=on iommu=pt pci-stub.ids=8086:10c9 video=efifb:off isolcpus=0-11,24-35 vfio_iommu_type1.allow_unsafe_interrupts=1 rd.driver.pre=vfio-pci modprobe.blacklist=radeon,amdgpu,fglrx,nouveau,nvidiafb,nvidia,nvidia_drm,snd-pcsp initrd=/bzroot

 

Using the script from https://www.youtube.com/watch?v=UilWLtMYd-U

 

Bus 1 --> 0000:03:00.3 (IOMMU group 15)
Bus 001 Device 003: ID 062a:4c01 MosArt Semiconductor Corp. 2.4G INPUT DEVICE <- wifi mouse & keyboard (dongle)
Bus 001 Device 002: ID 1532:005c Razer USA, Ltd Razer DeathAdder Elite <- wired mouse
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 2 --> 0000:03:00.3 (IOMMU group 15)
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 3 --> 0000:22:00.3 (IOMMU group 28)
Bus 003 Device 002: ID 8564:1000 Transcend Information, Inc. JetFlash <- test USB stick
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 4 --> 0000:22:00.3 (IOMMU group 28)
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 5 --> 0000:46:00.1 (IOMMU group 44)
Bus 005 Device 003: ID 0b05:1918 ASUSTek Computer, Inc. USB Audio <- unknown
Bus 005 Device 004: ID 0b05:18f3 ASUSTek Computer, Inc.
Bus 005 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 6 --> 0000:46:00.1 (IOMMU group 44)
Bus 006 Device 002: ID 0bc2:2321 Seagate RSS LLC Expansion Portable <- "unassigned" disk
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 7 --> 0000:46:00.3 (IOMMU group 44)
Bus 007 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 8 --> 0000:46:00.3 (IOMMU group 44)
Bus 008 Device 003: ID 13fe:6400 Kingston Technology Company Inc. <- unRAID usb stick
Bus 008 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub


IOMMU group 15
[RESET] 03:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
IOMMU group 28
[RESET] 22:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
IOMMU group 44
[RESET] 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4]
[RESET] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
[RESET] 46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]


The controller I'd like to pass to the VM is in IOMMU groups 15 & 28 (Bus 1,2,3,4).
It's the "3rd Gen AMD Ryzen™ Threadripper™ Desktop Processors", according to the MOBO manual:
    - 2 x USB 3.2 Gen 2 ports (1 x Type-C™, 1 x Type A at back panel)
    - 2 x USB 3.2 Gen 1 ports at back panel

 

Last time I attempted to pass these to the VM by using: vfio-pc.ids=1022:148c (and stub as well) the VM did not start at all, ~5 minutes later - the server went w/o network (after responding with pings of 900~1900ms for about 2 minutes) and had to be rebooted manually (no pings response at all), which then resulted in libvirt failed to start.

 

vfio-pci.ids=1022:148c

AMD Starship USB 3.0 Host Controller | USB controller (03:00.3)
AMD Starship USB 3.0 Host Controller | USB controller (22:00.3)

Both 03:00.3 and 22:00.3 passed to the vm:
Jan  6 10:22:28 beast kernel: vfio-pci 0000:03:00.3: not ready 1023ms after FLR; waiting
Jan  6 10:22:30 beast kernel: vfio-pci 0000:03:00.3: not ready 2047ms after FLR; waiting
Jan  6 10:22:33 beast kernel: vfio-pci 0000:03:00.3: not ready 4095ms after FLR; waiting
Jan  6 10:22:39 beast kernel: vfio-pci 0000:03:00.3: not ready 8191ms after FLR; waiting
Jan  6 10:22:48 beast kernel: vfio-pci 0000:03:00.3: not ready 16383ms after FLR; waiting
Jan  6 10:23:05 beast kernel: vfio-pci 0000:03:00.3: not ready 32767ms after FLR; waiting
Jan  6 10:23:41 beast kernel: clocksource: timekeeping watchdog on CPU36: Marking clocksource 'tsc' as unstable because the skew is too large:
Jan  6 10:23:41 beast kernel: clocksource:                       'hpet' wd_now: b6ece015 wd_last: b53757aa mask: ffffffff
Jan  6 10:23:41 beast kernel: clocksource:                       'tsc' cs_now: d4be71c208 cs_last: d3dc1106e4 mask: ffffffffffffffff
Jan  6 10:23:41 beast kernel: tsc: Marking TSC unstable due to clocksource watchdog
Jan  6 10:23:41 beast kernel: TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
Jan  6 10:23:41 beast kernel: sched_clock: Marking unstable (212891471252, -8998045)<-(213040103613, -157630481)



Only 22:00.3 passed to the vm (tsc unstable again, just didn't copy it)
Jan  6 10:28:21 beast kernel: vfio-pci 0000:22:00.3: not ready 1023ms after FLR; waiting
Jan  6 10:28:23 beast kernel: vfio-pci 0000:22:00.3: not ready 2047ms after FLR; waiting
Jan  6 10:28:26 beast kernel: vfio-pci 0000:22:00.3: not ready 4095ms after FLR; waiting
Jan  6 10:28:31 beast kernel: vfio-pci 0000:22:00.3: not ready 8191ms after FLR; waiting

 

Tried passing vfio-pc.ids=1022:1485,1022:149c
IOMMU group 44:    [1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
    [1022:1485] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
    [1022:149c] 46:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
    [1022:149c] 46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
    
The device 1022:1485 appears also in several other IOMMU groups by itself:
IOMMU group 14:    [1022:1485] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 26:    [1022:1485] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 52:    [1022:1485] 4b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 63:    [1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

 

The USB controller (ASMedia) was passed successfully.

 

I'll try to pass the CPU-bound controller again, this time: vfio-pc.ids=1022:1485,1022:148c - didn't work.

 

So, if someone's got an idea on how to pass 1022:148c - I'm open to any ideas.

Edited by ChewbaccaBG

Share this post


Link to post

Thank you for this tutorial.  It really helped me to figure out which IOMMU group of USB controllers to passthrough to my VM without catching my UNRAID thumbdrive in the mess.  I was able to pass my entire frontpanel through as well as both USB2 headers internally and I even wrote the XML right the first time.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.