rootd00d

I could be wrong, but my understanding is that only Windows 11 for Workstations (and relevant server SKUs) support RDMA. That’s a pretty hefty overhead in terms of licensing cost. They don’t even make it even remotely easy to find for purchase.

Using Robocopy, I maxed out at just over 1.3GB/s, so definitely having a multi-threaded copier is helping quite a bit over Windows File Explorer or TeraCopy. So it's an inch faster than 10Gbe -- maybe more copying needs to be done. Enabling the performance CPU governor didn't have any effect on the transfer speed, but I can see it's operational and hitting 3.5GHz turbo speeds.

Ah, good question -- it's a 4TB SN850X and clocks around 6.5GB/s sequential read and write, in CrystalDiskMark inside the VM, as it's mapped through directly.

I'm in a similar boat. I just upgraded the whole system yesterday -- * AsRock Rack ROMED8-2T * Epyc 7B13 64-core CPU * 256GB 3200MHz DDR4 EEC RAM * 500GB WD SN750 Gen3 * Mellanox MCX354A-FCCT The client machine (also running Unraid) is a Windows 11 Pro VM -- * MSI MEG z690 ACE * Intel i9-12900k * 32GB 5200Mhz DDR5 RAM * 2TB WD SN850 Gen4 * 4TB WD SN850X Gen4 * Mellanox MCX354A-FCCT The Windows VM is using the `virtio` driver instead of `virtio-net` because the latter would hard-cap the bandwidth to 10Gbps. Testing using iPerf3, I'm easily getting an upwards of 35Gbps across the link. Copying files over SMB or iSCSI cap out at around 550MB/s, to either the SSD or NVMe drive in the server, and it's basically the same speed both read and write. But 550MB/s is only 4Gbps -- could have just used the 10Gbe adapters! I was really expecting at least greater than 1GB/s (8000Gbps) to the NVMe drive since it's benched at 3.6GB/s sequential write -- was targeting at least several GB/s. If I can't get this going faster, then I don't see a point in getting a HyperX M.2 drive going. The next test would be to run the client VM bare-metal (which I can easily do) but I'm surprised there'd be that much overhead, and the issue doesn't seem to be the client since iPerf3 can easily saturate the link. Below @ 35Gbps peak -- Connecting to host 172.16.0.1, port 7777 [ 5] local 172.16.0.3 port 17297 connected to 172.16.0.1 port 7777 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 3.30 GBytes 28.3 Gbits/sec [ 5] 1.00-2.00 sec 3.77 GBytes 32.4 Gbits/sec [ 5] 2.00-3.00 sec 4.07 GBytes 35.0 Gbits/sec [ 5] 3.00-4.00 sec 3.85 GBytes 33.1 Gbits/sec [ 5] 4.00-5.00 sec 4.09 GBytes 35.1 Gbits/sec [ 5] 5.00-6.00 sec 3.81 GBytes 32.7 Gbits/sec [ 5] 6.00-7.00 sec 3.57 GBytes 30.6 Gbits/sec [ 5] 7.00-8.00 sec 3.97 GBytes 34.1 Gbits/sec [ 5] 8.00-9.00 sec 4.06 GBytes 34.9 Gbits/sec [ 5] 9.00-10.00 sec 4.04 GBytes 34.7 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 38.5 GBytes 33.1 Gbits/sec sender [ 5] 0.00-10.00 sec 38.5 GBytes 33.1 Gbits/sec receiver

Maybe need the MCX354A-FCCT to support RDMA, because it’s a ConnectX-3 Pro card, instead of non-Pro? I’ve just been specing this out recently. Seems like this is a major distinction between the two, at least according to Bing Chat.

Regarding #2, do you get an explicit option to map through the dock? I only seem to be able to map through the USB4 and Thunderbolt controllers, but dock and USB hub devices are listed if I don't have VFIO enabled. With my current configuration, the dock seems to get passed through normally and is detectable in Windows (Thunderbolt Control Center), but "regular" USB devices are not detected by the dock, such as wireless device receivers, USB audio devices, (like your #4 scenario) but *will* be available to pass-through via libvirt USB -- however *storage* devices (flash drives, SD cards, external HDDs) do actually pass through completely fine using any of the USB-A, USB-C data, or Thunderbolt ports. Currently looks like this -- I have no idea why I can't individually select these IOMMU groups. A quick albeit limited test of the video passthrough didn't seem to be working either -- might need to be plugged in at boot as per some other comments I've seen. saturn-diagnostics-20230312-1828.zip

Some progress, perhaps. I'm not sure if it changed anything, but I removed and re-added the Thunderbolt controllers in the VFIO configuration, after enabling "Both" for "ACS Override". As a correction for the above, it appears that USB storage devices *do* get detected, including both of the SD card reader slots. However, things like Bluetooth dongles, Razer wireless mouse & keyboard receivers, etc. do not pass through to the VM automatically -- but *can* be mapped via libvirt USB. I have an external USB audio interface (MOTU Ultralite mk3) that did not get detected inside the VM automatically, and also could not be passed through libvirt. I have no idea why storage devices would "just work" and other classes of devices do not. Is there a way to "manually" map in these IOMMU groups that don't have checkboxes? The "Goshen Ridge 2020:" device is the Caldigit TS4. I tried a quick DisplayPort passthrough test which didn't seem to work. GTX1080 -> DP-In -> Thunderbolt-Out -> Caldigit TS4 -> Thunderbolt-Out -> USB-C to HDMI active adapter -> TV. saturn-diagnostics-20230312-1828.zip

Just picked up a Caldigit TS4 Thunderbolt dock. Has anyone had any luck getting this passed through to a Windows VM? I believe I have the Thunderbolt and USB4 controllers passed through correctly, and the Thunderbolt Control Center shows the TS4 is connected. No USB devices are detected when plugged into the dock, and the built-in Ethernet controller is not detected.

Just a quick follow-up -- I can report the same issue with this card (YEELIYA 7-port USB 3.0 PCIe card): https://www.amazon.ca/dp/B08PF8XR73?psc=1&ref=ppx_yo2ov_dt_b_product_details Next one on the way is this (Inateck 8-port USB 3.2 Gen 2 PCIe card): https://www.amazon.com/Inateck-Power-Supply-Bandwidth-KU8212/dp/B0BCFN6HV2/ref=sr_1_9?crid=2BENALT0VVN4R&keywords=inateck+usb+3.2+gen+2+pcie&qid=1678171053&sprefix=inateck+usb+3.2+gen+2+pci%2Caps%2C124&sr=8-9 Not really expecting it will even resolve anything at this point...

I actually got the keyboard working. I’m not sure if it was disabling all the HID consumer devices (that obviously weren’t associated with other devices, e.g. Razer mouse, keyboard, etc.), or uninstalling and reinstalling the Microsoft Mouse and Keyboard Center in the VM, or both that and disabling the USB power saving options in the power profile; but it works now, just using USB pass through. Pretty sure I won’t be able to fix the USB audio issues connected to my PCIe card that uses 4 Renesas USB chips that’s passed through - I thought that would have made an improvement. The Fresco Logic FL1100 $30 card I have works better - have a couple different ones on order that I’ll be able to report status on. Hopefully even a Thunderbolt dock… Thank you! Maybe this will help someone out.

What did you do to resolve the keyboard and audio issues?

Well, you can also use ASIO for just audio play back in Windows using donation ware tools like Voicemeeter - https://vb-audio.com/Voicemeeter/index.htm This is a really awesome Swiss Army knife for audio routing on Windows. I’ve actually got everything *almost* perfect. And by that I mean no detectable issues (for at least hours at a time) even at 64 sample buffer sizes for 1.3ms of latency. I switched to this USB controller and the clicks and pops are *basically* gone - Sedna - PCI Express USB 3.1 Gen I (5Gbps) 4 Port Adapter with Low Profile Bracket - Supports Windows, Linux and Mac Pro (2008 to 2012 Late Version), no Need Power Connector https://a.co/d/1J07Owp Stupidly I assumed it was a configuration issue I must have resolved, and switched back to this - Vantec Quad Chip 4-Port Dedicated 5Gbps USB 3.0 PCIe Host Card (UGT-PCE430-4C) https://a.co/d/eLRm7La I must be of the mistaken impression that having a dedicated chip per port would make latency with the DAC more consistent somehow, but that does not prove to be the case. Perhaps it would relieve the bandwidth pressure you cited, but at the cost of clicks and pops in the audio. It has 4 Renesas controllers, and the Sedna has one Fresco FL1100 controller which has actually been cited as resolving this specific kind of issue in a customer review - Review on Amazon: Easy to assign to VMs in Unraid (Linux) https://www.amazon.ca/gp/aw/review/B06XKDTYMM/R10HYDI8YK4IRQ?ref_=cm_sw_r_apin_dprv_9Z9SMS2B5CFG29R04Y6G I’m going to switch back to the Sedna card, and if it’s still actually bulletproof I’d be interested in finding another card that had more ports or even multiple Fresco controllers if that’s really what matters. I’m really curious to find out how it ends up working if connected to a Thunderbolt dock.

Curious about your setup here — what were using the audio interface for? Were you using ASIO drivers for recording and monitoring? If so, what sample rate and buffer size did you use? Did you have any problem with intermittent pops and clicks? What USB controller did you use? I’ve just tried 2 separate USB cards and they both have the same result which is intermittent pops and clicks. Seemed almost perfect at 512 samples, but only after a reset of the whole system.

Using a dedicated PCIe USB card for a MOTI UltraLite-mk3 got me close, but not perfect. I rant about it here a bit - The MOTU work perfectly when using the ASIO drivers on bare metal, but no amount of tweaking allowed me to eliminate clicks and pops occurring intermittently, despite no issues being detected by LatencyMon. Would love to hear about someone pulling this off and what kind of hardware was used. I wanted to get a Thunderbolt Apollo Twin X but if it can’t even handle a USB 2.0 device, I’m not sure I want to spend that much to find out that’s not going to work either. Seems like the Thunderbolt pass through has some challenges itself, at least without a hub, and even then no hot-plug. Oh and FWIW, trying to do USB pass through of the device did not work at all - needed to pass through a discrete USB controller and let the VM own the MOTU entirely.

Ok, I got it to boot. Had no trouble getting everything set up and a stable VM running with seemingly excellent performance. The trick was to set a static setting for “Max TOLUD” in the BIOS. Any value appeared to work - just not “Dynamic”. A major drawback however was that ALL of the USB controllers or this motherboard (MSI MEG z690 ACE) are in the same IOMMU group. Crap! So, despite only having only 2 additional PCIe slots on the board, I opted to pick up a USB controller and 10gbe NIC. https://vantecusa.com/products_detail.php?p_id=119 https://www.asus.com/networking-iot-servers/wired-networking/all-series/xg-c100c/ Since the USB controller has 4 dedicated chips for controlling each port, I’d be able to assign each port to the VM however necessary, but in my case I just assigned them all to the same VM. That let me have my primary keyboard, wireless mouse, monitor USB hub, and USB audio interface all connected to the PCIe USB controller - and that’s when I was defeated. Absolutely no amount of tweaking let me get the audio interface to work without the krispies. It’s a MOTU UltraLite-mk3. https://motu.com/products/motuaudio/ultralite-mk3 This interface has terrible WDM drivers evidently, and only operates even on bare metal without clicks and pops when using the ASIO drivers, either in a DAW, or using Voicemeeter. But, assuming that was the case it would work flawlessly. After endless tweaking (message-signaled interrupts, CPU pinning/reservation) I could not get the card to operate without it clicking at least every few minutes. I could tell the tweaking was actually helping because it got to a point where LatencyMon wasn’t complaining at all, even when running 3D benchmarks. It *seems* like it might have been tied to something pertaining to the storage, strangely enough. The VM is running off a dedicated NVMe drive, and there’d just be moments where I knew there was some disk or network activity and the clicks would start to occur more often. Can someone recommend a better USB controller that’s actually worked with an audio interface (DAC)? This would clearly be unworkable when trying to use it as a recording workstation, and defeat the whole point of having the computer at all… so back to bare metal. In summary - the MEG is not really setup for IOMMU very well, and USB audio issues killed my UNRAID NAS+game system dreams… so I installed it on the computer my new system replaced - an Intel Q9650 on a Gigabyte GA-EP45-UD3P with 8GB of DDR2 and it’s running absolutely fine. It has zero IOMMU groups however, and an ancient MSI Radeon HD6850 Twin Frozr that’s eating itself alive. Validated it could detect the 10TB WD Red Plus (3gbps SATA 2.0 lulz), and off to the races. If it’s stable, I’ll probably pick up a few more drives, possibly an HBA, and another 10Gbe NIC. Meh. Fscking IOMMU and USB. The real pissoff was that it didn’t seem like I was going to be able to have a VM driven off a single Thunderbolt cable passed through the wall, but maybe I should try that after I buy a Thunderbolt 4 dock that doesn’t look like a piece of trash. Perhaps I could even get the audio interface to work properly without the Vantec. /rant