ChewbaccaBG

Also, I owe an apology to @methanoid I was on a few vodkas (and angry for not being able to pass that controller) last night, sorry mate.

So, here is it again, full information as was requested above. All USB controller(s) are integrated on the mobo: ASUSTeK COMPUTER INC. PRIME TRX40-PRO, Version Rev 1.xx, BIOS Version 0702 (latest) Current syslinux config (8086:10c9 is a GPU): kernel /bzimage append amd_iommu=on iommu=pt pci-stub.ids=8086:10c9 video=efifb:off isolcpus=0-11,24-35 vfio_iommu_type1.allow_unsafe_interrupts=1 rd.driver.pre=vfio-pci modprobe.blacklist=radeon,amdgpu,fglrx,nouveau,nvidiafb,nvidia,nvidia_drm,snd-pcsp initrd=/bzroot Using the script from https://www.youtube.com/watch?v=UilWLtMYd-U Bus 1 --> 0000:03:00.3 (IOMMU group 15) Bus 001 Device 003: ID 062a:4c01 MosArt Semiconductor Corp. 2.4G INPUT DEVICE <- wifi mouse & keyboard (dongle) Bus 001 Device 002: ID 1532:005c Razer USA, Ltd Razer DeathAdder Elite <- wired mouse Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 2 --> 0000:03:00.3 (IOMMU group 15) Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 3 --> 0000:22:00.3 (IOMMU group 28) Bus 003 Device 002: ID 8564:1000 Transcend Information, Inc. JetFlash <- test USB stick Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 4 --> 0000:22:00.3 (IOMMU group 28) Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 5 --> 0000:46:00.1 (IOMMU group 44) Bus 005 Device 003: ID 0b05:1918 ASUSTek Computer, Inc. USB Audio <- unknown Bus 005 Device 004: ID 0b05:18f3 ASUSTek Computer, Inc. Bus 005 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 6 --> 0000:46:00.1 (IOMMU group 44) Bus 006 Device 002: ID 0bc2:2321 Seagate RSS LLC Expansion Portable <- "unassigned" disk Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 7 --> 0000:46:00.3 (IOMMU group 44) Bus 007 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 8 --> 0000:46:00.3 (IOMMU group 44) Bus 008 Device 003: ID 13fe:6400 Kingston Technology Company Inc. <- unRAID usb stick Bus 008 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub IOMMU group 15 [RESET] 03:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] IOMMU group 28 [RESET] 22:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] IOMMU group 44 [RESET] 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4] [RESET] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] 46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] [RESET] 46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] The controller I'd like to pass to the VM is in IOMMU groups 15 & 28 (Bus 1,2,3,4). It's the "3rd Gen AMD Ryzen™ Threadripper™ Desktop Processors", according to the MOBO manual: - 2 x USB 3.2 Gen 2 ports (1 x Type-C™, 1 x Type A at back panel) - 2 x USB 3.2 Gen 1 ports at back panel Last time I attempted to pass these to the VM by using: vfio-pc.ids=1022:148c (and stub as well) the VM did not start at all, ~5 minutes later - the server went w/o network (after responding with pings of 900~1900ms for about 2 minutes) and had to be rebooted manually (no pings response at all), which then resulted in libvirt failed to start. vfio-pci.ids=1022:148c AMD Starship USB 3.0 Host Controller | USB controller (03:00.3) AMD Starship USB 3.0 Host Controller | USB controller (22:00.3) Both 03:00.3 and 22:00.3 passed to the vm: Jan 6 10:22:28 beast kernel: vfio-pci 0000:03:00.3: not ready 1023ms after FLR; waiting Jan 6 10:22:30 beast kernel: vfio-pci 0000:03:00.3: not ready 2047ms after FLR; waiting Jan 6 10:22:33 beast kernel: vfio-pci 0000:03:00.3: not ready 4095ms after FLR; waiting Jan 6 10:22:39 beast kernel: vfio-pci 0000:03:00.3: not ready 8191ms after FLR; waiting Jan 6 10:22:48 beast kernel: vfio-pci 0000:03:00.3: not ready 16383ms after FLR; waiting Jan 6 10:23:05 beast kernel: vfio-pci 0000:03:00.3: not ready 32767ms after FLR; waiting Jan 6 10:23:41 beast kernel: clocksource: timekeeping watchdog on CPU36: Marking clocksource 'tsc' as unstable because the skew is too large: Jan 6 10:23:41 beast kernel: clocksource: 'hpet' wd_now: b6ece015 wd_last: b53757aa mask: ffffffff Jan 6 10:23:41 beast kernel: clocksource: 'tsc' cs_now: d4be71c208 cs_last: d3dc1106e4 mask: ffffffffffffffff Jan 6 10:23:41 beast kernel: tsc: Marking TSC unstable due to clocksource watchdog Jan 6 10:23:41 beast kernel: TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. Jan 6 10:23:41 beast kernel: sched_clock: Marking unstable (212891471252, -8998045)<-(213040103613, -157630481) Only 22:00.3 passed to the vm (tsc unstable again, just didn't copy it) Jan 6 10:28:21 beast kernel: vfio-pci 0000:22:00.3: not ready 1023ms after FLR; waiting Jan 6 10:28:23 beast kernel: vfio-pci 0000:22:00.3: not ready 2047ms after FLR; waiting Jan 6 10:28:26 beast kernel: vfio-pci 0000:22:00.3: not ready 4095ms after FLR; waiting Jan 6 10:28:31 beast kernel: vfio-pci 0000:22:00.3: not ready 8191ms after FLR; waiting Tried passing vfio-pc.ids=1022:1485,1022:149c IOMMU group 44: [1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 [1022:1485] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:149c] 46:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller The device 1022:1485 appears also in several other IOMMU groups by itself: IOMMU group 14: [1022:1485] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 26: [1022:1485] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 52: [1022:1485] 4b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 63: [1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP The USB controller (ASMedia) was passed successfully. I'll try to pass the CPU-bound controller again, this time: vfio-pc.ids=1022:1485,1022:148c - didn't work. So, if someone's got an idea on how to pass 1022:148c - I'm open to any ideas.

That's good.. In any case I solved the missing VM problem, now scrolling through the whole topic to find a way to pass the USB controller w/o crashing the server again.

You aren't even running unraid, as you said. Why were you trying to "help" anyway?

Yes, I've checked the other groups, the unraid USB stick is not in the group I'd like to pass. That group only has 4 ports directly connected to the CPU.

So, this is the USB controller I'd like to pass to the VM: 3rd Gen AMD Ryzen™ Threadripper™ Desktop Processors - 2 x USB 3.2 Gen 2 ports (1 x Type-C™, 1 x Type A at back panel) - 2 x USB 3.2 Gen 1 ports at back panel 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller ../../../devices/pci0000:00/0000:00:08.1/0000:03:00.3/usb1 ../../../devices/pci0000:00/0000:00:08.1/0000:03:00.3/usb2 IOMMU group 9: [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 15: [1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller Since the 1st post is 5 years old, vfio-bind doesn't exist in Unraid 6.8.. What do I need to do? Here's my current config (1 GPU passthrough): <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10</name> <uuid>f0f6d8a1-ff05-65df-7cd5-69d6624a3f9b</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>20971520</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>24</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='24'/> <vcpupin vcpu='2' cpuset='1'/> <vcpupin vcpu='3' cpuset='25'/> <vcpupin vcpu='4' cpuset='2'/> <vcpupin vcpu='5' cpuset='26'/> <vcpupin vcpu='6' cpuset='3'/> <vcpupin vcpu='7' cpuset='27'/> <vcpupin vcpu='8' cpuset='4'/> <vcpupin vcpu='9' cpuset='28'/> <vcpupin vcpu='10' cpuset='5'/> <vcpupin vcpu='11' cpuset='29'/> <vcpupin vcpu='12' cpuset='6'/> <vcpupin vcpu='13' cpuset='30'/> <vcpupin vcpu='14' cpuset='7'/> <vcpupin vcpu='15' cpuset='31'/> <vcpupin vcpu='16' cpuset='8'/> <vcpupin vcpu='17' cpuset='32'/> <vcpupin vcpu='18' cpuset='9'/> <vcpupin vcpu='19' cpuset='33'/> <vcpupin vcpu='20' cpuset='10'/> <vcpupin vcpu='21' cpuset='34'/> <vcpupin vcpu='22' cpuset='11'/> <vcpupin vcpu='23' cpuset='35'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/f0f6d8a1-ff05-65df-7cd5-69d6624a3f9b_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='24' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='yes'/> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows 10/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows 10/vdisk2.img'/> <target dev='hdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.173.iso'/> <target dev='hdb' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='10' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:08:d9:27'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/boot/vBIOS/ASUS-ROG-STRIX-nVidia-GTX1050Ti-GP107-KVM.rom'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain>

If I change the cpufreq governor to "performance", the fps drop still exists, but from ~350 to ~250. With "ondemand" - from 300 to 50. I can't figure out why this occurs. Perhaps I'll try to pass one of the USB controllers directly into the VM later.

The new mouse started doing the same an hour ago...

Has someone experienced problem passing a USB mouse - it works in Windows, but when I start a game and move the mouse - FPS drops from 250 to 50 back and forth. I'm passing USB Wi-Fi keyboard/mouse in 1 port, and a wired USB mouse (Lenovo Y) in another. When moving the Wi-Fi mouse - no issues, when moving wired - fps drop. Tried another wired mouse (Razer) in the same port - it didn't have any issues. Weird.

If someone needs to do some testing, I've got a TRX4 up and running. Let me know.