ChewbaccaBG

A fix is probably coming from FreeBSD, but we won't see it in an installable ISO anytime soon, meaning installing FreeBSD would be kind of impossible. Any chance of getting an updated kernel @ the next unRAID-RC ?

Hmm, has this been tested against 12.1-release/stable? And still the other question - why wasn't proxmox affected by these changes? Weird..

I'm already passing the usb controller (ASMedia, still having trouble passing the other one). I'll check now the other stuff and reboot the vm. Here are pinned / isolated cores for the VM. New VM (xml) config edited @ 1st post. kernel /bzimage append amd_iommu=on iommu=pt pci-stub.ids=8086:10c9 vfio-pci.ids=1022:1485,1022:149c rd.driver.pre=vfio-pci video=efifb:off isolcpus=12-23,36-47 nohz_full=12-23,36-47 rcu_nocbs=12-23,36-47 modprobe.blacklist=radeon,amdgpu,fglrx,nouveau,nvidiafb,nvidia,nvidia_drm,snd-pcsp initrd=/bzroot

The VM CPUs are isolated, forgot to mention that.

Here's the topology

# lstopo /mnt/user/appdata/topoligy-tr3960x.png lstopo: error while loading shared libraries: libudev.so.0: cannot open shared object file: No such file or directory ** solved ln -s /lib64/libudev.so.1 /lib64/libudev.so.0

Well, it's a gaming VM. There's some stuttering, fps drops from time to time w/o any changes - at one boot it can work okay, at reboot - poor performance. So, I'm looking to optimize the VM as much as possible. From what I see in dmesg - there's 4 NUMA regions: [ 0.000000] No NUMA configuration found [ 0.412283] pci_bus 0000:00: on NUMA node 0 [ 0.414292] pci_bus 0000:20: on NUMA node 1 [ 0.424251] pci_bus 0000:40: on NUMA node 2 [ 0.425815] pci_bus 0000:60: on NUMA node 3

@jonp just tested Version: 6.8.1-rc1 - it's same - no network (including passed-through pcie nics), no disk. Tested on: Q35-4.2 & i440fx-4.2, FreeBSD 12.1-release I'll also test the most-recent 12.1-stable snapshot to see if there's any difference -- same.

Okay guys, I'm running a ASUS TRX40-PRO & Threadripper 3960x, 32Gigs 3200 memory. Right now I'm just passing 10c/10t "host" CPU & 20G ram, but I'm not impressed... Any tips/tricks of getting the best performance on a Windows VM? Here's the current VM config: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='2'> <name>Windows 10</name> <uuid>f0f6d8a1-ff05-65df-7cd5-69d6624a3f9b</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>20971520</memory> <currentMemory unit='KiB'>20971520</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>24</vcpu> <iothreads>2</iothreads> <cputune> <vcpupin vcpu='0' cpuset='12'/> <vcpupin vcpu='1' cpuset='36'/> <vcpupin vcpu='2' cpuset='13'/> <vcpupin vcpu='3' cpuset='37'/> <vcpupin vcpu='4' cpuset='14'/> <vcpupin vcpu='5' cpuset='38'/> <vcpupin vcpu='6' cpuset='15'/> <vcpupin vcpu='7' cpuset='39'/> <vcpupin vcpu='8' cpuset='16'/> <vcpupin vcpu='9' cpuset='40'/> <vcpupin vcpu='10' cpuset='17'/> <vcpupin vcpu='11' cpuset='41'/> <vcpupin vcpu='12' cpuset='18'/> <vcpupin vcpu='13' cpuset='42'/> <vcpupin vcpu='14' cpuset='19'/> <vcpupin vcpu='15' cpuset='43'/> <vcpupin vcpu='16' cpuset='20'/> <vcpupin vcpu='17' cpuset='44'/> <vcpupin vcpu='18' cpuset='21'/> <vcpupin vcpu='19' cpuset='45'/> <vcpupin vcpu='20' cpuset='22'/> <vcpupin vcpu='21' cpuset='46'/> <vcpupin vcpu='22' cpuset='23'/> <vcpupin vcpu='23' cpuset='47'/> <emulatorpin cpuset='0-1'/> <iothreadpin iothread='1' cpuset='2-3'/> <iothreadpin iothread='2' cpuset='4-5'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/f0f6d8a1-ff05-65df-7cd5-69d6624a3f9b_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> <reset state='on'/> <vendor_id state='on' value='none'/> <frequencies state='on'/> </hyperv> <kvm> <hidden state='on'/> <hint-dedicated state='on'/> </kvm> <vmport state='off'/> <smm state='on'> <tseg unit='MiB'>48</tseg> </smm> <ioapic driver='kvm'/> <vmcoreinfo state='on'/> </features> <cpu mode='host-passthrough' check='partial'> <topology sockets='1' cores='24' threads='1'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> <feature policy='require' name='svm'/> <feature policy='require' name='apic'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='invtsc'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='yes'/> <timer name='hypervclock' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback' io='threads' iothread='1' queues='2'/> <source file='/mnt/user/domains/Windows 10/vdisk1.img' index='3'/> <backingStore/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback' io='threads' iothread='2' queues='2'/> <source file='/mnt/user/domains/Windows 10/vdisk2.img' index='2'/> <backingStore/> <target dev='hdd' bus='virtio'/> <alias name='virtio-disk3'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.173.iso' index='1'/> <backingStore/> <target dev='hdb' bus='sata'/> <readonly/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0x10'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='10' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <alias name='pci.10'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:08:d9:27'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-2-Windows 10/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/boot/vBIOS/ASUS-ROG-STRIX-nVidia-GTX1050Ti-GP107-KVM.rom'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x46' slot='0x00' function='0x1'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x46' slot='0x00' function='0x3'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

Isn't this a dual-port 40gig (qsfp) nic?

That's obviously important ... we already know in the topic that versions prior to 11.3 can work with unRAID (they are already End of Life).

Also, I owe an apology to @methanoid I was on a few vodkas (and angry for not being able to pass that controller) last night, sorry mate.

Hopefully @jonp will find the way PVE is doing this and apply it to unRAID. That way no more issues.

Freezes here:

For some reason the installer freezes on "USB tablet" and never shows the install menu. FreeBSD 12.1-RELEASE 64 bit Opnsense is based on FreeBSD but could well be an older version that "runs" at the moment. Unfortunately, I need 12.1

@Ioannis what did you select from the i440fx? There are ~20 in the menu.

I'll try later today.

So, here is it again, full information as was requested above. All USB controller(s) are integrated on the mobo: ASUSTeK COMPUTER INC. PRIME TRX40-PRO, Version Rev 1.xx, BIOS Version 0702 (latest) Current syslinux config (8086:10c9 is a GPU): kernel /bzimage append amd_iommu=on iommu=pt pci-stub.ids=8086:10c9 video=efifb:off isolcpus=0-11,24-35 vfio_iommu_type1.allow_unsafe_interrupts=1 rd.driver.pre=vfio-pci modprobe.blacklist=radeon,amdgpu,fglrx,nouveau,nvidiafb,nvidia,nvidia_drm,snd-pcsp initrd=/bzroot Using the script from https://www.youtube.com/watch?v=UilWLtMYd-U Bus 1 --> 0000:03:00.3 (IOMMU group 15) Bus 001 Device 003: ID 062a:4c01 MosArt Semiconductor Corp. 2.4G INPUT DEVICE <- wifi mouse & keyboard (dongle) Bus 001 Device 002: ID 1532:005c Razer USA, Ltd Razer DeathAdder Elite <- wired mouse Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 2 --> 0000:03:00.3 (IOMMU group 15) Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 3 --> 0000:22:00.3 (IOMMU group 28) Bus 003 Device 002: ID 8564:1000 Transcend Information, Inc. JetFlash <- test USB stick Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 4 --> 0000:22:00.3 (IOMMU group 28) Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 5 --> 0000:46:00.1 (IOMMU group 44) Bus 005 Device 003: ID 0b05:1918 ASUSTek Computer, Inc. USB Audio <- unknown Bus 005 Device 004: ID 0b05:18f3 ASUSTek Computer, Inc. Bus 005 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 6 --> 0000:46:00.1 (IOMMU group 44) Bus 006 Device 002: ID 0bc2:2321 Seagate RSS LLC Expansion Portable <- "unassigned" disk Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 7 --> 0000:46:00.3 (IOMMU group 44) Bus 007 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 8 --> 0000:46:00.3 (IOMMU group 44) Bus 008 Device 003: ID 13fe:6400 Kingston Technology Company Inc. <- unRAID usb stick Bus 008 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub IOMMU group 15 [RESET] 03:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] IOMMU group 28 [RESET] 22:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] IOMMU group 44 [RESET] 42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:57a4] [RESET] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] 46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] [RESET] 46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] The controller I'd like to pass to the VM is in IOMMU groups 15 & 28 (Bus 1,2,3,4). It's the "3rd Gen AMD Ryzen™ Threadripper™ Desktop Processors", according to the MOBO manual: - 2 x USB 3.2 Gen 2 ports (1 x Type-C™, 1 x Type A at back panel) - 2 x USB 3.2 Gen 1 ports at back panel Last time I attempted to pass these to the VM by using: vfio-pc.ids=1022:148c (and stub as well) the VM did not start at all, ~5 minutes later - the server went w/o network (after responding with pings of 900~1900ms for about 2 minutes) and had to be rebooted manually (no pings response at all), which then resulted in libvirt failed to start. vfio-pci.ids=1022:148c AMD Starship USB 3.0 Host Controller | USB controller (03:00.3) AMD Starship USB 3.0 Host Controller | USB controller (22:00.3) Both 03:00.3 and 22:00.3 passed to the vm: Jan 6 10:22:28 beast kernel: vfio-pci 0000:03:00.3: not ready 1023ms after FLR; waiting Jan 6 10:22:30 beast kernel: vfio-pci 0000:03:00.3: not ready 2047ms after FLR; waiting Jan 6 10:22:33 beast kernel: vfio-pci 0000:03:00.3: not ready 4095ms after FLR; waiting Jan 6 10:22:39 beast kernel: vfio-pci 0000:03:00.3: not ready 8191ms after FLR; waiting Jan 6 10:22:48 beast kernel: vfio-pci 0000:03:00.3: not ready 16383ms after FLR; waiting Jan 6 10:23:05 beast kernel: vfio-pci 0000:03:00.3: not ready 32767ms after FLR; waiting Jan 6 10:23:41 beast kernel: clocksource: timekeeping watchdog on CPU36: Marking clocksource 'tsc' as unstable because the skew is too large: Jan 6 10:23:41 beast kernel: clocksource: 'hpet' wd_now: b6ece015 wd_last: b53757aa mask: ffffffff Jan 6 10:23:41 beast kernel: clocksource: 'tsc' cs_now: d4be71c208 cs_last: d3dc1106e4 mask: ffffffffffffffff Jan 6 10:23:41 beast kernel: tsc: Marking TSC unstable due to clocksource watchdog Jan 6 10:23:41 beast kernel: TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'. Jan 6 10:23:41 beast kernel: sched_clock: Marking unstable (212891471252, -8998045)<-(213040103613, -157630481) Only 22:00.3 passed to the vm (tsc unstable again, just didn't copy it) Jan 6 10:28:21 beast kernel: vfio-pci 0000:22:00.3: not ready 1023ms after FLR; waiting Jan 6 10:28:23 beast kernel: vfio-pci 0000:22:00.3: not ready 2047ms after FLR; waiting Jan 6 10:28:26 beast kernel: vfio-pci 0000:22:00.3: not ready 4095ms after FLR; waiting Jan 6 10:28:31 beast kernel: vfio-pci 0000:22:00.3: not ready 8191ms after FLR; waiting Tried passing vfio-pc.ids=1022:1485,1022:149c IOMMU group 44: [1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 [1022:1485] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:149c] 46:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller The device 1022:1485 appears also in several other IOMMU groups by itself: IOMMU group 14: [1022:1485] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 26: [1022:1485] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 52: [1022:1485] 4b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 63: [1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP The USB controller (ASMedia) was passed successfully. I'll try to pass the CPU-bound controller again, this time: vfio-pc.ids=1022:1485,1022:148c - didn't work. So, if someone's got an idea on how to pass 1022:148c - I'm open to any ideas.

That's good.. In any case I solved the missing VM problem, now scrolling through the whole topic to find a way to pass the USB controller w/o crashing the server again.

Solved.

Well, after trying to pass-through a USB controller by adding it to pci-stub.ids= the VM didn't start, the whole server lost network connectivity after ~5 minutes. I had to reboot it manually, and after reboot - the VMs menu disappeared. Accessing it manually resulted in "Libvirt Service failed to start." /etc/libvirt is empty, there is also /etc/libvirt- that has some configs & folders, however, copying that content over to the empty libvirt and rebooting - again resulted in an empty /etc/libvirt and the identical error message. I've since removed the USB controller ID from pci-stub.ids, however, that doesn't fix the issue. Has someone seen any such problem? Here's the syslinux configuration: kernel /bzimage append amd_iommu=on iommu=pt pci-stub.ids=8086:10c9 video=efifb:off isolcpus=0-11,24-35 vfio_iommu_type1.allow_unsafe_interrupts=1 rd.driver.pre=vfio-pci modprobe.blacklist=radeon,amdgpu,fglrx,nouveau,nvidiafb,nvidia,nvidia_drm,snd-pcsp initrd=/bzroot Here's the "System devices" information: System Devices PCI Devices and IOMMU Groups IOMMU group 0: [1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 1: [1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 2: [1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 3: [1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 4: [1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 5: [1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 6: [1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 7: [1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 8: [1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 9: [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 10: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 11: [1022:1490] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1491] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1492] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1493] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1494] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1495] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1496] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1497] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 IOMMU group 12: [10de:1c82] 01:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1) [10de:0fb9] 01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1) IOMMU group 13: [1022:148a] 02:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 14: [1022:1485] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 15: [1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller IOMMU group 16: [1022:1482] 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 17: [1022:1482] 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 18: [1022:1482] 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 19: [1022:1482] 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 20: [1022:1482] 20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 21: [1022:1482] 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 22: [1022:1484] 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 23: [1022:1482] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 24: [1022:1484] 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 25: [1022:148a] 21:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 26: [1022:1485] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 27: [1022:1486] 22:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP IOMMU group 28: [1022:148c] 22:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller IOMMU group 29: [1022:1487] 22:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller IOMMU group 30: [1022:1482] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 31: [1022:1483] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 32: [1022:1483] 40:01.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 33: [1022:1482] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 34: [1022:1482] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 35: [1022:1482] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 36: [1022:1482] 40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 37: [1022:1482] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 38: [1022:1484] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 39: [1022:1482] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 40: [1022:1484] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 41: [1022:57ad] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad IOMMU group 42: [1022:57a3] 42:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3 IOMMU group 43: [1022:57a3] 42:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3 IOMMU group 44: [1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 [1022:1485] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:149c] 46:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 45: [1022:57a4] 42:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 [1022:7901] 47:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 46: [1022:57a4] 42:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 [1022:7901] 48:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 47: [8086:10c9] 43:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) IOMMU group 48: [8086:10c9] 43:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) IOMMU group 49: [8086:1539] 45:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 50: [10ec:5763] 49:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. Device 5763 (rev 01) IOMMU group 51: [1022:148a] 4a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 52: [1022:1485] 4b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 53: [1022:1482] 60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 54: [1022:1482] 60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 55: [1022:1482] 60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 56: [1022:1482] 60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 57: [1022:1482] 60:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 58: [1022:1482] 60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 59: [1022:1484] 60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 60: [1022:1482] 60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 61: [1022:1484] 60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 62: [1022:148a] 61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 63: [1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP CPU Thread Pairings Pair 1: cpu 0 / cpu 24 Pair 2: cpu 1 / cpu 25 Pair 3: cpu 2 / cpu 26 Pair 4: cpu 3 / cpu 27 Pair 5: cpu 4 / cpu 28 Pair 6: cpu 5 / cpu 29 Pair 7: cpu 6 / cpu 30 Pair 8: cpu 7 / cpu 31 Pair 9: cpu 8 / cpu 32 Pair 10: cpu 9 / cpu 33 Pair 11: cpu 10 / cpu 34 Pair 12: cpu 11 / cpu 35 Pair 13: cpu 12 / cpu 36 Pair 14: cpu 13 / cpu 37 Pair 15: cpu 14 / cpu 38 Pair 16: cpu 15 / cpu 39 Pair 17: cpu 16 / cpu 40 Pair 18: cpu 17 / cpu 41 Pair 19: cpu 18 / cpu 42 Pair 20: cpu 19 / cpu 43 Pair 21: cpu 20 / cpu 44 Pair 22: cpu 21 / cpu 45 Pair 23: cpu 22 / cpu 46 Pair 24: cpu 23 / cpu 47 USB Devices Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 003 Device 002: ID 062a:4c01 MosArt Semiconductor Corp. 2.4G INPUT DEVICE Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 005 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 005 Device 003: ID 0b05:1918 ASUSTek Computer, Inc. USB Audio Bus 005 Device 004: ID 0b05:18f3 ASUSTek Computer, Inc. Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 006 Device 002: ID 0bc2:2321 Seagate RSS LLC Expansion Portable Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 007 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub Bus 007 Device 003: ID 1532:005c Razer USA, Ltd Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 008 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub Bus 008 Device 003: ID 13fe:6400 Kingston Technology Company Inc. ASM107x SCSI Devices [0:0:0:0] disk Seagate Expansion 0637 /dev/sda 1.00TB [1:0:0:0] disk USB DISK 3.0 PMAP /dev/sdb 63.1GB [N:0:1:1] disk XPG GAMMIX S5__1 /dev/nvme0n1 1.02TB Also attached to the topic is the diagnostics report. Hopefully someone's gonna figure out whats wrong. beast-diagnostics-20200105-2206.zip

You aren't even running unraid, as you said. Why were you trying to "help" anyway?

Yes, I've checked the other groups, the unraid USB stick is not in the group I'd like to pass. That group only has 4 ports directly connected to the CPU.

So, this is the USB controller I'd like to pass to the VM: 3rd Gen AMD Ryzen™ Threadripper™ Desktop Processors - 2 x USB 3.2 Gen 2 ports (1 x Type-C™, 1 x Type A at back panel) - 2 x USB 3.2 Gen 1 ports at back panel 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller ../../../devices/pci0000:00/0000:00:08.1/0000:03:00.3/usb1 ../../../devices/pci0000:00/0000:00:08.1/0000:03:00.3/usb2 IOMMU group 9: [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 15: [1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller Since the 1st post is 5 years old, vfio-bind doesn't exist in Unraid 6.8.. What do I need to do? Here's my current config (1 GPU passthrough): <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows 10</name> <uuid>f0f6d8a1-ff05-65df-7cd5-69d6624a3f9b</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>20971520</memory> <currentMemory unit='KiB'>2097152</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>24</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='24'/> <vcpupin vcpu='2' cpuset='1'/> <vcpupin vcpu='3' cpuset='25'/> <vcpupin vcpu='4' cpuset='2'/> <vcpupin vcpu='5' cpuset='26'/> <vcpupin vcpu='6' cpuset='3'/> <vcpupin vcpu='7' cpuset='27'/> <vcpupin vcpu='8' cpuset='4'/> <vcpupin vcpu='9' cpuset='28'/> <vcpupin vcpu='10' cpuset='5'/> <vcpupin vcpu='11' cpuset='29'/> <vcpupin vcpu='12' cpuset='6'/> <vcpupin vcpu='13' cpuset='30'/> <vcpupin vcpu='14' cpuset='7'/> <vcpupin vcpu='15' cpuset='31'/> <vcpupin vcpu='16' cpuset='8'/> <vcpupin vcpu='17' cpuset='32'/> <vcpupin vcpu='18' cpuset='9'/> <vcpupin vcpu='19' cpuset='33'/> <vcpupin vcpu='20' cpuset='10'/> <vcpupin vcpu='21' cpuset='34'/> <vcpupin vcpu='22' cpuset='11'/> <vcpupin vcpu='23' cpuset='35'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-4.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/f0f6d8a1-ff05-65df-7cd5-69d6624a3f9b_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='24' threads='1'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='yes'/> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows 10/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/user/domains/Windows 10/vdisk2.img'/> <target dev='hdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.173.iso'/> <target dev='hdb' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='10' model='pcie-to-pci-bridge'> <model name='pcie-pci-bridge'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:08:d9:27'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/boot/vBIOS/ASUS-ROG-STRIX-nVidia-GTX1050Ti-GP107-KVM.rom'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain>

If I change the cpufreq governor to "performance", the fps drop still exists, but from ~350 to ~250. With "ondemand" - from 300 to 50. I can't figure out why this occurs. Perhaps I'll try to pass one of the USB controllers directly into the VM later.