harperhendee

This should work. I have two Fresco 1100 USB cards in my rig. I run two gaming VMs in paralell running Oculus and Vive. My setup isn't perfect. I get occasional hangs, especially when playing "switchboard operator" on the USB ports. I think this is due to my own problems with PCI initialization. My cards reside at 4e and 54. Here's where they are discovered in syslog.txt: Similar stuff on 54. Later, it is revisited at 4c (which is the PCI bridge chip on MOBO) And then I get a non-fatal error: Then another non-fatal error: And then this message: And finally this message: USB devices seem to work fine in Unraid after this point. I can even boot from the card (better than my mobo usb, actually). However, when I launch my VM, I get one more non-fatal error: There's a warning in the VM log file as well: After this point, USB mostly works. But I get flaky behavior that I think is related to the non-fatal errors. Some of the flaky behavior results in hard hangs, which is not nice at all! I've been doing the research on how PCI enumeration works so that I can steer things into a more predictable and stable configuration. This thread is pure gold! --Harper

BTW, I was able to confirm a few things about CPU numbering and physical layout from Anandtech and a few other sources: 1) The CPUs are numbered in roughly consecutive order around the rings. The numbering is determined by a correlated variable, so they might not be exactly as you'd expect, but there's no functional difference between 0,1,2,3,4,5 and 1,0,3,2,5,4 ordering. 2) Base layouts give you basically 1, 1.5, or 2 iterations of the ring structure. If there are 12 cores in a ring, there will be 3 versions with 12/18/24 physical cores. 3) The CPUs are always fused off in equal numbers from each ring. This includes half-rings. So the 22 core Xeon has two rings of 11 cores each. There's never a 10 and 12 core ring. 4) There is a small latency price to pay when crossing rings. Try to minimize cross-ring traffic. From a topological point of view, if you cross a ring, you generate 2x the bandwidth.

I took an educated guess on the cpu numbering, then mapped all 8 VMs + 1 core for unraid. It looks like the diagram. I mainly mapped things based on what was easy in Visio. The final mapping does the following: Unraid: Gets first 2 threads Vanaheim and Muspelheim: Get the middle of each ring. Consume some of the other VM mappings if I want to run just these two. Alfheim and Svartalfheim: My next two VMs getting prepped for a new FOVE headset Niflheim, Helheim, Asgard, Midgard: Minimum size VMs are placeholders for future expansion Utgard: I'd like to have a dual-boot option into windows (machine name Jotunheim). Utgard is a VM that can read and execute from Jotunheim's unmanaged disk. ,

Gigabyte X99 Designare EX Usage: multi-user VR experiences Plus points: Plenty of data lanes for multiple GPUs and PCIe devices Good power distribution Mechanically sound solution for PCIe slots Alpine ridge with Thunderbolt and USB3.1 performs well when passed through to VM Plenty of connectors and USB3 ports Minus points: Renesys USB hubs don't do passthru very well PCIe endpoint 02 is overloaded Requires a monitor connected to boot. Doesn't like to let go of primary GPU (still WIP for me) GPU: MSI 1070 Quicksilver Usage: VR gaming GPU Plus points: Easy installation No SW problems Can use LED SW to indicate which VMs are located and active on each GPU Great cooling and silent operation under load Minus points: Only 1 HDMI out, no VGA (but who does nowadays?) No way to get at the covered PCIe slot (even with thin x1 ribbon cable)

I'm using a higher core count Xeon (22 CPUs/44 threads). I'm wondering if I should take into account the routing of these cores. In this generation of Xeon, there are multiple rings (see attached). It seems like you would want to keep VM cores near each other in the ring, so that they would use the fabric more efficiently. In the simplest case, I would create two VMs out of this topology using ring 1 and ring 2. When we go to 4 VMs, maybe align to the 4 columns. Of course, I can just as easily convince myself that the CPUs should be spread out so that each one has more distributed bandwidth. I think this works out if there are multiple VMs that are not necessarily running full tilt at the same time. Spreading out the CPUs ensures that each CPU lives in a "quiet neighborhood." Therefore, the isolated VM with spread out CPUs has more paths to memory and fewer rivals. But when all VMs are enabled and under load, they will end up stepping all over each other.

I have also found that passing through my Alpine Ridge based USB controller (on the mobo), I can avoid this mmap error and pass through the USB devices. However, I still get drops, just not as many. I think the issue is that there are occasionally stutters in latency. It is worse if the MMIO map isn't there. Any latency longer that .5ms will cause timeouts that lead to dropping the link. Here's an example from the vrserver.txt log from SteamVR: Mon Mar 20 2017 20:05:43.621 - lighthouse: LHR-FFA49F41 C: SendOnPoseChange took 0.657625ms Mon Mar 20 2017 20:05:43.621 - lighthouse: LHR-0D3CA9A9 H: SendOnPoseChange took 0.666565ms Mon Mar 20 2017 20:05:47.883 - lighthouse: LHR-FFA49F41 C: IMU went off scale. Mon Mar 20 2017 20:05:54.578 - lighthouse: LHR-F766BF45 C: SendOnPoseChange took 0.578845ms Mon Mar 20 2017 20:05:54.578 - lighthouse: LHR-0D3CA9A9 H: SendOnPoseChange took 0.641143ms Mon Mar 20 2017 20:05:54.578 - lighthouse: LHR-FFA49F41 C: SendOnPoseChange took 0.551746ms Mon Mar 20 2017 20:06:31.147 - lighthouse: LHR-FFA49F41 C: SendOnPoseChange took 0.503695ms Mon Mar 20 2017 20:07:20.464 - Capturing input focus for vrcompositor (8376). Old focus was 0. Focus stack size 0 Mon Mar 20 2017 20:07:20.464 - Sending InputFocusCaptured old=0 new=8376 Mon Mar 20 2017 20:07:21.832 - Releasing input focus for vrcompositor (8376) because of message. new focus is 0. Mon Mar 20 2017 20:07:21.832 - Sending VREvent_InputFocusReleased old=8376 new=0 Mon Mar 20 2017 20:07:30.209 - 2 - entering standby Mon Mar 20 2017 20:07:30.209 - lighthouse: Device LHR-FFA49F41 powering off upon entering standby. Mon Mar 20 2017 20:07:30.890 - lighthouse: 8B14732891: Wireless controller disconnected Mon Mar 20 2017 20:07:30.890 - lighthouse: LHR-FFA49F41: Disconnected from receiver 8B14732891 Mon Mar 20 2017 20:07:30.890 - lighthouse: LHR-FFA49F41 C: Dropped 1294 back-facing hits during the previous tracking session Link to the rest of the VR logs: https://drive.google.com/file/d/0B7ky_0GX-JUBUjBnTTAzUDBJUjg/view?usp=sharing

Here's the real message: 2017-03-21 02:59:32.014+0000: starting up libvirt version: 2.4.0, qemu version: 2.7.1, hostname: Yggdrasil LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name guest=Vanaheim,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-Vanaheim/master-key.aes -machine pc-i440fx-2.7,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/b3afaf7c-3828-85e2-eb36-d5043a7b6175_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 16384 -realtime mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid b3afaf7c-3828-85e2-eb36-d5043a7b6175 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-Vanaheim/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device nec-usb-xhci,id=usb,bus=pci.0,ad54:00.0,id=hostdev2,bus=pci.0,addr=0x8 -device vfio-pci,host=06:00.0,id=hostdev3,bus=pci.0,addr=0xa -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -msg timestamp=on Domain id=1 is tainted: high-privileges Domain id=1 is tainted: host-cpu char device redirected to /dev/pts/0 (label charserial0) 2017-03-21T02:59:34.425641Z qemu-system-x86_64: -device vfio-pci,host=54:00.0,id=hostdev2,bus=pci.0,addr=0x8: Failed to mmap 0000:54:00.0 BAR 2. Performance may be slow

I have two Innatech USB cards that I pass through to my two gaming VMs, used for Virtual Reality. The two cards are: Inateck PCI-E to USB 3.0 5-Port PCI Express Card and 15-Pin Power Connector https://www.amazon.com/Inateck-Express-Connector-Controller-Internal/dp/B00FPIMJEW and Inateck Superspeed 7 Ports PCI-E to USB 3.0 Expansion Card https://www.amazon.com/dp/B00FPIMICA/ref=psdc_284717_t2_B00FPIMJEW These two cards are mapped at 0000:4e:00.0 and 0000:54:00.0. The 4e card maps to PCI root at 0000:00:02.0 and the 54 card maps to 0000:00:1c.6. The 4e goes through a PEX8747 at 0000:4b:00.0 I followed the instructions here and successfully pass the cards to the VMs. They seem to work mostly ok--with occasional flakiness. The main flakiness is associated with my Vive touch controllers, that are attached to the 4e card. The Vive touch controllers communicate to the Vive headset via a wireless USB chip. There is a hub in the Vive headset. My symptoms on the Vive are loss of one touch controller at a time, seemingly at random, though it feels like always when I need it most! It happens on average about once every 2-5 minutes. When I launch the two VMs (Muspelheim with Oculus and 54 USB card, Vanaheim with Vive and 4e USB card), I get a similar warning in the virtio logs: qemu-system-x86_64: -device vfio-pci,host=4e:00.0,addr=00.0: Failed to mmap 0000:4e:00.0 BAR 2. Performance may be slow I am writing that warning by memory--sorry, I didn't copy the log when I went to work today. My theory about this failure is that there the KVM remap via IOMMU didn't work because Unraid has claimed the devices somehow. This gets some non-IOMMU or host assisted lookup that only occurs when unraid services the interrupt. This leads to occasional timeouts to the controllers, which are already at a pretty long chain of USB/PCIE hops: Xeon -> PCIe port -> PCIe bridge -> USB3.0 Hub (card) -> USB 3.0 hub (Vive)->Wireless USB (touch controller) What can I do to gather more information to debug the issue here?