jbartlett

2950 - I planned on building two of these, it'll go into the 2nd build.

I just picked up a 2990wx to switch into this board. Exciting times! Needed the extra juice.

Interesting. I haven't heard of that.

With Downstream set, I was able to pass a video card and an onboard LAN to a VM once. The VM would start after shutting down but there'd be no response from it. Haven't tried combinations of one or the other yet with Downstream set though I suspect the issue is related to the above. I plan on investigation further but won't be able to until tomorrow.

I used pci-stub.ids to stub the card. I'm not familiar with "pci-id.vfio"

OP updated with additional onboard passthrough information. Also learned that the 10GB passthrough issue only happens if a video card is also passed through. Dropping the video card after the 10GB isn't responding and starting up the VM, the 10GB works - and doesn't if I put the video card back on. The two Quadro's don't need their BIOS dumped and included to work but I wonder if that's somehow affecting things by not doing it.

Oh man, I wish!

Slot 4 can be passed through to a VM with PCIe ACS override set to Multifunction. In progress of testing what else I can pass through in this state.

My memory might have been faulty.

Okay - whittled it down a bit. Seems like sometimes the 10GB controller survives a reboot/reset, sometimes it doesn't, when passed through to a VM.

Twice now, the 10GB onboard LAN wouldn't work and would disable/enable itself every 10 seconds or so. It happened using it as unraid's port and when bound to a VM. Rebooting the box fixes. Not sure yet what causes it but when bound to a VM, it started when I rebooted the VM after a day of heavy usage. I've rebooted the VM several times since then with no issues with the 10GB port.

Testing the intended use of this motherboard, I've been ingesting four 4K Brio's via an USB3 add-on card passed through to a Win10VM and outputting a 4K stream to YouTube as well as outputting those 4K Brio's via NDI. Stable as a rock after 12 hours. I also had 4 additional USB2 web cams (Logitech 922/930, 2 each) passed into it via the motherboard's USB3 ports who's controller was passed through to the VM. The USB2 cams were able to be passed through as a USB passthrough (no controller) though that was a shits-n-grins test.

Updated OP: The 10GB LAN port can be passed through

Yes, a GeForce GT 1030 single-slot card using it's fan to also bring air over the USB3 card which I have in the 2nd slot.

Added onboard USB3 controllers that could be routed to a VM

Oh neat, I never noticed the CPU Isolation part on the CPU Pinning page (never scrolled down far enough)

Other than I didn't know about being able to hyphenate the range, no.

ASUS ROG Zenith Extreme Alpha X399 HEDT Gaming Motherboard AMD Threadripper 2 (TR4) EATX DDR4 M.2 10G LAN USB 3.1 Gen2 https://www.amazon.com/gp/product/B07M6SD5GP/ https://www.newegg.com/Product/Product.aspx?Item=N82E16813119186 A video card in PCIe slot 1 will pass through to a VM if you dump the card's BIOS and pass it in on the VM configuration, however, a nVidia drivers will detect KVM and will not initialize (Code 43). Not passing through the BIOS results in no video for the VM in slot 1. The PCIe 4 (x4) slot can't be passed through (it's grouped with a SATA controller) without PCIe ACS override. Slots 2, 3, & 5 were able to be passed through to a VM with PCIe ACS override set to "Disabled". nVidia drivers do not detect KVM and report a Code 43 if Slot 1 has a graphics card dedicated to unraid and the nVidia card is in a different slot passed through to a VM. To enable full virtualization for unraid in the BIOS: Advanced > CPU Configuration > SVM Mode: Enabled Advanced > AMD PBS > Enumerate all IOMMU in IVRS: Enabled NUMA Node info for the 2990WX Node 0: PCIe 3 & 5, CPUs 0-7/32-39 Node 1: CPUs 16-23/48-55 Node 2: PCIe 1 & 2, CPUs 8-15/40-47 Node 3: CPUs 24-31/56-63 Notes: The onboard video can take a bit to display on power up, be patient. After a BIOS update, the system will automatically reboot and your screen will be black/off for awhile. Be patient. 6-7 watts at power off 86-88 watts at OS/idle, one spinner powered down, 3 SSD's No watt measurement under load taken yet Primary slot-1 video card does not init if not attached to a monitor. No option in the BIOS to change which slot is the primary video. If you have the BIOS set to remember the last power on state in the event of power loss and it keeps powering back up after shutting down, change the BIOS setting to disabled, save, and then change it back to remember. The PCIe clamps tend to re-engage after unlocking when you start to pull the graphics card out. If VM's are extreamly slow to start after flashing a BIOS, load the defaults and reset the Enable virtualization. Onboard Controllers passthrough A [1022:145f] 48:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller B [1022:145f] 0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller C [1b21:2142] 0d:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller *1 D [1b21:2142] 0f:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller *1 E [1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02) *1 F [1d6a:07b1] 09:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02) *2 G [8086:1539] 05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) *1 *1 Requires PCIe ACS override=Multifunction *2 Does not always survive a reboot with a Graphics Card also passed through It looks like the wifi & SATA controllers can be passed through if the PCIe ACS override is set to Multifunction but I did not test and don't plan on testing. I did not find a USB controller for the two USB2 ports located near the lower USB3 port on the motherboard itself. If passing through the 10GB LAN to Windows, install the LAN driver in the VM prior to doing so. The attached IOMMU groups shows the details with three video cards and one USB3 card. PCIe ACS override=Both.txt PCIe ACS override=Disabled.txt PCIe ACS override=Downstream.txt PCIe ACS override=Multifunction.txt

I had that same thought. I just built a 2950 system but it hasn't been on long enough at once to see if it shows up. It has not shown up on my Intel build.

https://www.amazon.com/gp/product/B015CQ8DCS/ IOMMU Group [10b5:8609] 41:00.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) [10b5:8609] 41:00.1 System peripheral: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) [10b5:8609] 42:01.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) [10b5:8609] 42:05.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) [10b5:8609] 42:07.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) [10b5:8609] 42:09.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) [1b21:1142] 43:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 44:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 45:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 46:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller Using pci-stub.ids=1b21:1142 in the syslinix.cfg lets one or more of the 4 controllers to bind to a VM (but not different VM's without the PCIe override being set). I tested thumb drives & had three Brio web cams ingesting at 4K simultaneously without issue. No need to stub 10b5:8609. The card is not compatible with UNRAID's "BIND" via vfio-pci.cfg

On your HTOP, the matching CPU from the Dashboard graph is also maxed which isn't the scenario I reported here. I'd recommend excluding the VM CPU's in the sys config to keep the OS away from them. IE: append isolcpus=12,13,14,15,28,29,30,31 initrd=/bzroot

In regards to the "Unknown PCI header" error, try creating a new VM with the same settings, pointing to the same drives, and see if it starts. I've had such issues in the past and creating a new VM shell resolved. I'm not knowledgeable enough on the underlaying XML to debug it.

Success! Updated the OP.

Happening in RC7 after about 5 days uptime.

Wow, that is an impressive dip! I know I have logic in place to prevent some wonky benchmarks from being submitted to the online database. I need to find time to review this to see if it would catch this test. Though now that I think of it, I should add a "Wonky" flag to my database to exclude them from the averages so you would still be able to purge the benchmark and restore it later (future DiskSpeed update). If you're willing to consider this a junk drive (I would), I would reimburse you for shipping if you mail it to me so I can use it for app logic testing. PM me if you are interested.