Prograde

I believe users with this specific card (Startech PEXUSB3S44V) have no issues passing them to the machine, but have issues once passed to the machine inside the VM. Mine throws a Code 10 error on the USB hub that the device "cannot start" when passed to a Windows 10 VM using either provided drivers or Microsoft drivers. It also does not function with an Ubuntu 16.04 VM, in my experience, which has native UAS/UASP support if I'm not mistaken. Either we need some sort of kernel patch to get it to work, or Startech's implementation on this card is subject, which may be the case either way. The odd thing about this card specifically is that it uses the same Renesas µPD720202 controllers that I know work passed to VM in unRAID and ESXI, at least without a PCI Express Switch (PLX) in-between it and the host. This points to a potential botched implementation on Startech's part, unless they use some sort of custom firmware on the controllers themselves. However, I have record of this specific card working as expected in ESXI 6.0 U3 and ESXI 6.5.0d. I believe VMware added support for UAS/UASP standards with a patch in ESXI 5.5 back in 2014, and the Code 10 error in a Windows VM was what you would get before that patch. I know this in no way means unRAID needs explicit UAS/UASP support to get this card to work, as ESXI =/= KVM, but it it worth trying out. Thank you for the help!

Hello, I have had great success with both ESXI and unRAID passing through a USB controller to VMs using the Buffalo IFC-PCIE2U3, which is a simple dual port card powered by a single Renesas µPD720202 surface mount. Now you can imagine physical PCI-e ports can disappear quickly if you want to pass-through such native USB controllers to multiple VMs. What I have been looking for is a quad-port four-controller USB (3.0 or 2.0, just want some I/O) PCI-e card that can be split into four different pass-through devices to use with four different VMs similar to how something like an Intel I350-T4 can be split up into four separate NICs each with a single port. I have found and tested a few devices that use a PLX chip to feed four separate USB controllers, and have had no success finding a reliable solution to work with unRAID. Here are the two models I have tried so-far: HighPoint RocketU 1144C with four Asmedia 1042A Controllers - Latest Firmware dated to 2014 IOMMU Splits up OK with ACS Override and VFIO blacklisting. Passes through to VMs OK, being seen in the OS. Windows: HighPoint drivers do work OK, and all is well on first boot. USB seems to work fine. Shutting down the VM and starting back up logs an error: DMAR: [DMA Read] Request device [0a:00.0] fault addr ed000 [fault reason 06] PTE Read access is not set. The port will not come back to functionality until the entire host is rebooted. Ubuntu: Works perfectly with native drivers on first boot. Shutting down the VM and starting back up logs an error: DMAR: [DMA Read] Request device [0a:00.0] fault addr ed000 [fault reason 06] PTE Read access is not set. The port will not come back to functionality until the entire host is rebooted. ESXI/Windows: Same issue after clean shut down and cold boot with not handing controller back to the host/VM properly. StarTech PEXUSB3S44V with four Renesas µPD720202 Controllers - Latest Firmware dated to 2012 IOMMU Splits up OK with ACS Override and VFIO blacklisting. Passes through to VMs OK, being seen in the OS. Windows: Throws a Code 10 error, device cannot start directly on hub. Startech drivers and Microsoft xHCI drivers both do not work. Ubuntu: Does not work at all, presumably due to same issue the windows VM was seeing. ESXI/Windows: Seems to work identically to the IFC-PCIE2U3, in that everything functions both in and outside the VMs and between shut-downs and reboots. This is where I think there should be hope for the StarTech card with unRAID/KVM with some discrete setting, or as seen in the last link below, a kernel supporting USB Attached SCSI. Are there any 4 or more port USB PCI-e controllers other users have found to work properly with unRAID? Or even two port cards? This would add a whole lot of value to the platform, having a reliable way to pass USB controllers en masse to VMs. I have found a few references looking about the web for both cards, and some recent information regarding the Startech PEXUSB3S44V and a possible custom kernel: HighPoint 1144C References: https://www.redhat.com/archives/vfio-users/2016-June/msg00102.html Startech PEXUSB3S44V References: The last link is the most promising, and something I am looking to try out. I am willing to try out just about anything, having two spare test unRaid servers currently going and hardware in hand. Any insight or experience is welcomed. Thank you for your help!

Wow, thank you for all of your research, that is quite useful, an amazing amount of information there you have come up with! I have saved that link for future reference. I'm honestly surprised expanders did so well in your tests. Very little overhead with them. Looks like my rebuild times WOULD go up using an H310 and a dual-link expander to something like (doing some math) 22 hours if all 24 drive bays are populated. This, being limited to 95 MB/s, would be 4-6 hours slower than normal, which is just barely bearable, 24 hours being my cut-off. I'm almost willing to just say SAS-2 with an expander is the fastest I would need until the physical SATA interface is replaced altogether. Anyway, I like the rest of your points in that link. Fortunately, the rest of the system (-mostly eBay deals-) I built is still up to speed, even being almost three(!) years old: Supermicro X10SRL-F, Xeon E5-1660 V3, 64GB Samsung M393A2G40DB0-CPB0, 1.4TB P420M PCI-e Cache (works on newer versions), Intel X520-DA2 I've loved the simplicity of Unraid, and the community support due to users like you has been awesome. Thank you again!

Hello, I am looking to move my current Unraid server to a larger Supermicro 4U case with more drive bays, specifically something in the CSE 846 lineup that sport 24 front-port bays and support full height PCI-e cards. I've been using an H310 flashed to IT mode with an eight drive CSE 745 case-imposed limit, and the move to dual parity pushed my need for more bays, even with 6+2 8TB drives ("48TB accessible"). Now, with the "SAS-2 era" versions of the CSE 846, you can settle with either an expander backplane (BPN-SAS2-846EL1) limited to a maximum of 48 Gbps with two SFF-8087 uplinks or choose a "direct attach" "TQ" or "A" that either have 24 SATA ports (TQ) or 6 SFF-8087 ports (A), some of which are said to support even SAS-3 speeds, as there is no controller between your drives and your HBAs. I have a feeling that the 48 Gbps cap imposed by the expander would not be seen in my typical use, but I can see it being a severe detriment to rebuild and parity check speeds if all drives are being read from simultaneously and I have the entire 24-bay server populated. Although, theoretically, 48 Gbps gives you 250 MB/s to each of 24 disks, which is faster than the 7200 RPM HGST drives I use. I know theory is only good for just that, theorizing, so I remain concerned. Here is a great breakdown of SMCI backplane types by nephri at ServeTheHome: LINK What I am asking is if there is anyone with experience who can tell me that using an expander with 24 8TB 7200 RPM drives would affect rebuild and parity-check times significantly. I am currently sitting at 16-18 hours for a 7200 RPM 8TB rebuild/parity check using an 8 port TQ-style backplane and a single LSI 9211-8i equivalent, which is already not exactly timely. I am guessing that moving to an expander with more drives would push rebuild/parity check times over 24 hours, which would be my personal limit. Upping to 6 SFF-8087 connections would require another 4-port LSI card like a 9305-16i or 9201-16i, but that should not be a big deal if it saves time on rebuilds and parity checks. Thank you for your input!

Hello! Good news! Shortly after my last reply I bought an HDMI coupler to go between my (at the time unfinished) TV connection and the (at the time unfinished) keystone run I have for the Unraid server (my other troubleshooting systems used the same drop). No problems at all with that, being wired directly to the TV. I am limited to using HDMI as I am running through 20 ft and 100 ft in-wall repeaters run between keystones, and have no other video+audio HDMI gear to test with (DisplayPort is my preference). I comically did try to haul the 70" TV down to the basement rack where the Unraid server lives to eliminate some variables, but found a corner too tight to navigate. There are no network drops at the TV (working on that), otherwise, the Unraid server would have been de-racked and brought up. Long story short, this whole issue stemmed from passing HDMI through a receiver that ended up being faulty: a three month old Onkyo TX-RZ800. Every input on it eventually produced the same issue, strangely only with my Maxwell cards that have HDMI 2.0. The new unit is working just fine, and the keystones are finished. All is well. p.s. I hate leaving stuff unanswered on forums.

Yes, as buried in my post above (sorry about the number of words, I tend to be verbose), I have changed both the Audio and Video components of the GTX 980 to MSI mode: I made this a must-do on every card I use HDMI with, so every card I tested with so far has been in MSI mode. I am going to drop that GT 740 into the X10SRL-F server after work, and test HDMI. Latest BIOS on that board too (dated 12/2015). Could it be a problem with the motherboard BIOS? Really seems unlikely to me, as the GTX 960 had issues in all of the systems I tested in, but maybe the problem lies with the firmware on that specific GTX 960 (EVGA 4GB 04G-P4-1962-KR). I have not tested the GTX 980 (Asus STRIX-GTX980-DC2OC-4GD5) in another machine. I'll test that in another system then too. Thank you for your input though, if you have any other suggestions, I'm eager to hear them!

Hello, I've been trying to track down a hard-to-place and hard to replicate Video/Audio glitch I am having with Nvidia's Maxwell GPUs when passed to a Windows 10 virtual machine under unRAID with QEMU. Let me first say, I have been passing through AMD GPUs with ESXI for probably 3-4 years now since ESXI 5.0, with varying results, with most of my issues stemming from AMD's drivers. While I notice much better performance and general stability using Nvidia GPUs under unRAID, I am having one major issue that is stopping me from sticking with this solution. Problem Description: Anywhere from 15 minutes to 2 days after booting, there will be slowed audio and video playback that surfaces either at idle OR while being used. This affects not only video playback in VLC/MPC-HC/Chrome, but also system sounds, indicating it is something more than just a GPU acceleration problem with a program. This problem sometimes will fix itself just seconds after it appears, mostly after you close all running programs. Sometimes the problem persists, and requires a reboot, which does fix the issue until it reappears. What I have determined is that every time it happens, playback is slowed to exactly 33% (1/3rd) of normal speed. I do have a video of it happening, if needed. System noises also appear to be slowed to 33% as well. Very odd, but at least consistent. I have been troubleshooting this issue for about a month now by myself, and have come up with no fixes. I have three IOMMU/VT-D systems with which to test in, any many cards to test with: MAIN unRAID system (unRAID 6.0 Plus): Supermicro X10SRL-F with E5-1650 V3 / 32GB ECC DDR4 / Buffalo IFC-PCIE2U3S2 USB 3.0 Passthrough / H310 IT MODE / Tested GPUs: GTX 980 / GTX 960 PCI Devices: 00:00.0 Host bridge: Intel Corporation Xeon E5 v3/Core i7 DMI2 (rev 02) 00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:01.1 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) 00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) 00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) 00:03.2 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) 00:04.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 0 (rev 02) 00:04.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 1 (rev 02) 00:04.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 2 (rev 02) 00:04.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 3 (rev 02) 00:04.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 4 (rev 02) 00:04.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 5 (rev 02) 00:04.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 6 (rev 02) 00:04.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DMA Channel 7 (rev 02) 00:05.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02) 00:05.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Hot Plug (rev 02) 00:05.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02) 00:05.4 PIC: Intel Corporation Xeon E5 v3/Core i7 I/O APIC (rev 02) 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05) 00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05) 00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05) 00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05) 00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #2 (rev 05) 00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5) 00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5) 00:1c.5 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 (rev d5) 00:1c.6 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 (rev d5) 00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05) 00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05) 00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05) 02:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 03:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) 03:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1) 04:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1) 04:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1) 05:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) 06:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) 07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 09:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03) 0a:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30) ff:0b.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0b.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0b.2 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02) ff:0c.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0c.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02) ff:0f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02) ff:0f.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02) ff:0f.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:0f.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:0f.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02) ff:10.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02) ff:10.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02) ff:10.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:10.6 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:10.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02) ff:12.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02) ff:12.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02) ff:13.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02) ff:13.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02) ff:13.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02) ff:13.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02) ff:13.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02) ff:14.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02) ff:14.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02) ff:14.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02) ff:14.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02) ff:14.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02) ff:14.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02) ff:14.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02) ff:14.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02) ff:15.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control (rev 02) ff:15.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control (rev 02) ff:15.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers (rev 02) ff:15.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers (rev 02) ff:16.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02) ff:16.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02) ff:16.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02) ff:17.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02) ff:17.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:17.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02) ff:1e.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1e.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02) ff:1f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02) ff:1f.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02) IOMMU Groups: /sys/kernel/iommu_groups/0/devices/0000:ff:0b.0 /sys/kernel/iommu_groups/0/devices/0000:ff:0b.1 /sys/kernel/iommu_groups/0/devices/0000:ff:0b.2 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.0 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.1 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.2 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.3 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.4 /sys/kernel/iommu_groups/1/devices/0000:ff:0c.5 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.0 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.1 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.4 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.5 /sys/kernel/iommu_groups/2/devices/0000:ff:0f.6 /sys/kernel/iommu_groups/3/devices/0000:ff:10.0 /sys/kernel/iommu_groups/3/devices/0000:ff:10.1 /sys/kernel/iommu_groups/3/devices/0000:ff:10.5 /sys/kernel/iommu_groups/3/devices/0000:ff:10.6 /sys/kernel/iommu_groups/3/devices/0000:ff:10.7 /sys/kernel/iommu_groups/4/devices/0000:ff:12.0 /sys/kernel/iommu_groups/4/devices/0000:ff:12.1 /sys/kernel/iommu_groups/5/devices/0000:ff:13.0 /sys/kernel/iommu_groups/5/devices/0000:ff:13.1 /sys/kernel/iommu_groups/5/devices/0000:ff:13.2 /sys/kernel/iommu_groups/5/devices/0000:ff:13.3 /sys/kernel/iommu_groups/5/devices/0000:ff:13.4 /sys/kernel/iommu_groups/5/devices/0000:ff:13.5 /sys/kernel/iommu_groups/5/devices/0000:ff:13.6 /sys/kernel/iommu_groups/5/devices/0000:ff:13.7 /sys/kernel/iommu_groups/6/devices/0000:ff:14.0 /sys/kernel/iommu_groups/6/devices/0000:ff:14.1 /sys/kernel/iommu_groups/6/devices/0000:ff:14.2 /sys/kernel/iommu_groups/6/devices/0000:ff:14.3 /sys/kernel/iommu_groups/6/devices/0000:ff:14.4 /sys/kernel/iommu_groups/6/devices/0000:ff:14.5 /sys/kernel/iommu_groups/6/devices/0000:ff:14.6 /sys/kernel/iommu_groups/6/devices/0000:ff:14.7 /sys/kernel/iommu_groups/7/devices/0000:ff:15.0 /sys/kernel/iommu_groups/7/devices/0000:ff:15.1 /sys/kernel/iommu_groups/7/devices/0000:ff:15.2 /sys/kernel/iommu_groups/7/devices/0000:ff:15.3 /sys/kernel/iommu_groups/8/devices/0000:ff:16.0 /sys/kernel/iommu_groups/8/devices/0000:ff:16.6 /sys/kernel/iommu_groups/8/devices/0000:ff:16.7 /sys/kernel/iommu_groups/9/devices/0000:ff:17.0 /sys/kernel/iommu_groups/9/devices/0000:ff:17.4 /sys/kernel/iommu_groups/9/devices/0000:ff:17.5 /sys/kernel/iommu_groups/9/devices/0000:ff:17.6 /sys/kernel/iommu_groups/9/devices/0000:ff:17.7 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.0 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.1 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.2 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.3 /sys/kernel/iommu_groups/10/devices/0000:ff:1e.4 /sys/kernel/iommu_groups/11/devices/0000:ff:1f.0 /sys/kernel/iommu_groups/11/devices/0000:ff:1f.2 /sys/kernel/iommu_groups/12/devices/0000:00:00.0 /sys/kernel/iommu_groups/13/devices/0000:00:01.0 /sys/kernel/iommu_groups/14/devices/0000:00:01.1 /sys/kernel/iommu_groups/15/devices/0000:00:02.0 /sys/kernel/iommu_groups/16/devices/0000:00:03.0 /sys/kernel/iommu_groups/17/devices/0000:00:03.2 /sys/kernel/iommu_groups/18/devices/0000:00:04.0 /sys/kernel/iommu_groups/18/devices/0000:00:04.1 /sys/kernel/iommu_groups/18/devices/0000:00:04.2 /sys/kernel/iommu_groups/18/devices/0000:00:04.3 /sys/kernel/iommu_groups/18/devices/0000:00:04.4 /sys/kernel/iommu_groups/18/devices/0000:00:04.5 /sys/kernel/iommu_groups/18/devices/0000:00:04.6 /sys/kernel/iommu_groups/18/devices/0000:00:04.7 /sys/kernel/iommu_groups/19/devices/0000:00:05.0 /sys/kernel/iommu_groups/19/devices/0000:00:05.1 /sys/kernel/iommu_groups/19/devices/0000:00:05.2 /sys/kernel/iommu_groups/19/devices/0000:00:05.4 /sys/kernel/iommu_groups/20/devices/0000:00:11.0 /sys/kernel/iommu_groups/20/devices/0000:00:11.4 /sys/kernel/iommu_groups/21/devices/0000:00:14.0 /sys/kernel/iommu_groups/22/devices/0000:00:16.0 /sys/kernel/iommu_groups/22/devices/0000:00:16.1 /sys/kernel/iommu_groups/23/devices/0000:00:1c.0 /sys/kernel/iommu_groups/24/devices/0000:00:1c.4 /sys/kernel/iommu_groups/25/devices/0000:00:1c.5 /sys/kernel/iommu_groups/26/devices/0000:00:1c.6 /sys/kernel/iommu_groups/27/devices/0000:00:1f.0 /sys/kernel/iommu_groups/27/devices/0000:00:1f.2 /sys/kernel/iommu_groups/27/devices/0000:00:1f.3 /sys/kernel/iommu_groups/28/devices/0000:02:00.0 /sys/kernel/iommu_groups/29/devices/0000:03:00.0 /sys/kernel/iommu_groups/29/devices/0000:03:00.1 /sys/kernel/iommu_groups/30/devices/0000:04:00.0 /sys/kernel/iommu_groups/30/devices/0000:04:00.1 /sys/kernel/iommu_groups/31/devices/0000:05:00.0 /sys/kernel/iommu_groups/32/devices/0000:06:00.0 /sys/kernel/iommu_groups/33/devices/0000:07:00.0 /sys/kernel/iommu_groups/34/devices/0000:08:00.0 /sys/kernel/iommu_groups/35/devices/0000:09:00.0 /sys/kernel/iommu_groups/35/devices/0000:0a:00.0 ESXI System (unRAID 6.0 Free): Supermicro X10DRL-i with dual E5-2630 V3 / 128GB ECC DDR4 / Tested GPUs: GTX 960 / GT 740 / QUADRO 600 Troubleshooting System: (unRAID 6.0 Free) Gigabyte X79-UP4 with E5-2670 / 24GB DDR3 / 1144C USB 3.0 Passthrough / Tested GPUs: GTX 960 / GT 740 / QUADRO 600 / HD 7970 / HD 6450 The GTX 960 (Maxwell) has had this issue with all three systems, and the GTX 980 (Maxwell) has had issues with the only system I tested it in. The GT 740 (Kepler) so far has not seen this issue so far. The Quadro 600 does not have HDMI, but works fine with DVI. The GTX 960 and GTX 980 when using DVI and a USB Audio DAC have not experienced this issue. It is strictly a HDMI issue. Both AMD cards I tested do not have this issue. I have installed and reinstalled Windows 10 many times, and have stuck to using SeaBIOS as it seems more stable. I am limited to using HDMI, so I am looking for a solution. I have tried changing the GPU from Line Interrupts to Message Signaled Interrupts, with no change. I am looking to test Windows 7 with the same setup, and also try a Display-port to HDMI adapter. If anyone has any experiences with this or anything they think I could try out, I'd be happy to test. Thank you for your help!