jwoolen

Finally got my hardware. Here's what lstopo looks like. I have a 12700K on an ASUS TUF Gaming Z690-Plus WiFi D4 board. Here's the IOMMU groupings with no overrides. IOMMU group 0: [8086:4668] 00:00.0 Host bridge: Intel Corporation Device 4668 (rev 02) IOMMU group 1: [8086:460d] 00:01.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x16 Controller #1 (rev 02) IOMMU group 2: [8086:464d] 00:06.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 (rev 02) IOMMU group 3: [8086:467d] 00:0a.0 Signal processing controller: Intel Corporation Platform Monitoring Technology (rev 01) IOMMU group 4: [8086:467f] 00:0e.0 RAID bus controller: Intel Corporation Volume Management Device NVMe RAID Controller IOMMU group 5: [8086:7ae0] 00:14.0 USB controller: Intel Corporation Device 7ae0 (rev 11) Bus 001 Device 001 Port 1-0 ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002 Port 1-2 ID 0b05:19af ASUSTek Computer, Inc. AURA LED Controller Bus 001 Device 003 Port 1-3 ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.1 Hub Bus 001 Device 004 Port 1-6 ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub Bus 001 Device 005 Port 1-3.1 ID 045e:02fe Microsoft Corp. XBOX ACC Bus 001 Device 008 Port 1-3.3 ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.1 Hub Bus 001 Device 009 Port 1-6.3 ID 0a12:1243 Cambridge Silicon Radio, Ltd Audioengine HD3 Bus 001 Device 010 Port 1-3.4 ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.1 Hub Bus 001 Device 012 Port 1-3.4.4 ID 046d:c52b Logitech, Inc. Unifying Receiver Bus 002 Device 001 Port 2-0 ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 002 Device 002 Port 2-2 ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.1 Hub Bus 002 Device 003 Port 2-9 ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub Bus 002 Device 004 Port 2-2.3 ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.1 Hub Bus 002 Device 005 Port 2-2.4 ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.1 Hub Bus 002 Device 007 Port 2-9.4 ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive [8086:7aa7] 00:14.2 RAM memory: Intel Corporation Device 7aa7 (rev 11) IOMMU group 6: [8086:7af0] 00:14.3 Network controller: Intel Corporation Device 7af0 (rev 11) IOMMU group 7: [8086:7acc] 00:15.0 Serial bus controller [0c80]: Intel Corporation Device 7acc (rev 11) [8086:7acd] 00:15.1 Serial bus controller [0c80]: Intel Corporation Device 7acd (rev 11) [8086:7ace] 00:15.2 Serial bus controller [0c80]: Intel Corporation Device 7ace (rev 11) IOMMU group 8: [8086:7ae8] 00:16.0 Communication controller: Intel Corporation Device 7ae8 (rev 11) IOMMU group 9: [8086:7ae2] 00:17.0 SATA controller: Intel Corporation Device 7ae2 (rev 11) IOMMU group 10: [8086:7ac8] 00:1a.0 PCI bridge: Intel Corporation Device 7ac8 (rev 11) IOMMU group 11: [8086:7ac0] 00:1b.0 PCI bridge: Intel Corporation Device 7ac0 (rev 11) IOMMU group 12: [8086:7ab8] 00:1c.0 PCI bridge: Intel Corporation Device 7ab8 (rev 11) IOMMU group 13: [8086:7aba] 00:1c.2 PCI bridge: Intel Corporation Device 7aba (rev 11) IOMMU group 14: [8086:7abc] 00:1c.4 PCI bridge: Intel Corporation Device 7abc (rev 11) IOMMU group 15: [8086:7ab0] 00:1d.0 PCI bridge: Intel Corporation Device 7ab0 (rev 11) IOMMU group 16: [8086:7a84] 00:1f.0 ISA bridge: Intel Corporation Device 7a84 (rev 11) [8086:7ad0] 00:1f.3 Audio device: Intel Corporation Device 7ad0 (rev 11) [8086:7aa3] 00:1f.4 SMBus: Intel Corporation Device 7aa3 (rev 11) [8086:7aa4] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Device 7aa4 (rev 11) IOMMU group 17: [10de:2204] 01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1) [10de:1aef] 01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1) IOMMU group 18: [144d:a80a] 02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO This controller is bound to vfio, connected drives are not visible. IOMMU group 19: [144d:a80a] 03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [N:1:6:1] disk Samsung SSD 980 PRO 2TB__1 /dev/nvme1n1 2.00TB IOMMU group 20: [8086:15f3] 06:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) IOMMU group 21: [1000:00e6] 07:00.0 Serial Attached SCSI controller: Broadcom / LSI Fusion-MPT 12GSAS/PCIe Secure SAS38xx [9:0:0:0] disk ATA SAMSUNG MZ7LH7T6 304Q /dev/sdb 7.68TB [9:0:1:0] disk ATA SAMSUNG MZ7LH7T6 404Q /dev/sdc 7.68TB [9:0:2:0] disk ATA SAMSUNG MZ7LH7T6 404Q /dev/sdd 7.68TB [9:0:3:0] disk ATA SAMSUNG MZ7LH7T6 404Q /dev/sde 7.68TB [9:0:4:0] disk ATA SAMSUNG MZ7LH7T6 404Q /dev/sdf 7.68TB [9:0:5:0] disk ATA SAMSUNG MZ7LH7T6 404Q /dev/sdg 7.68TB [9:0:6:0] disk ATA SAMSUNG MZ7LH7T6 404Q /dev/sdh 7.68TB [9:0:7:0] disk ATA SAMSUNG MZ7LH7T6 404Q /dev/sdi 7.68TB Here's what core assignment looks like.

I have an Aorus z690 board and a 12700K coming later this week. I will post IOMMU groupings, etc. after I've had a chance to play with it.

@jtmoore81 I was having the same problem with a blank screen for the nvidia driver plugin and the system locking up until the other GPU in my system was bound to vfio. This was with unraid version 6.9.2. With version 6.9.1, I was able to get it working without binding the GPU to vfio as long as the BIOS was configured to legacy boot.

@scorcho99 Yes. All of the IOMMU groupings I posted were without any ACS overiddes.

I got my hands on an ROG Maximus XIII Hero. It's a little more flexible with the CPU lanes. I have it running x8/x4/x4 (m.2). Both GPUs still have their own group and both m.2 drives still get their own group. I'm OK with the second GPU getting 4 lanes because it's only going to be used for Plex transcoding. IOMMU group 0: [8086:4c43] 00:00.0 Host bridge: Intel Corporation Device 4c43 (rev 01) IOMMU group 1: [8086:4c01] 00:01.0 PCI bridge: Intel Corporation Device 4c01 (rev 01) IOMMU group 2: [8086:4c05] 00:01.1 PCI bridge: Intel Corporation Device 4c05 (rev 01) IOMMU group 3: [8086:4c07] 00:01.2 PCI bridge: Intel Corporation Device 4c07 (rev 01) IOMMU group 4: [8086:4c09] 00:06.0 PCI bridge: Intel Corporation Device 4c09 (rev 01) IOMMU group 5: [8086:43ed] 00:14.0 USB controller: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller (rev 11) Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 0b05:19ac ASUSTek Computer, Inc. USB Audio Bus 001 Device 003: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub Bus 001 Device 004: ID 0b05:18f3 ASUSTek Computer, Inc. AURA LED Controller Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver Bus 001 Device 006: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 001 Device 007: ID 05e3:0610 Genesys Logic, Inc. 4-port hub Bus 001 Device 069: ID 8087:0032 Intel Corp. Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 002 Device 002: ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive Bus 002 Device 003: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub [8086:43ef] 00:14.2 RAM memory: Intel Corporation Tiger Lake-H Shared SRAM (rev 11) IOMMU group 6: [8086:43e8] 00:15.0 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H Serial IO I2C Controller #0 (rev 11) [8086:43e9] 00:15.1 Serial bus controller [0c80]: Intel Corporation Device 43e9 (rev 11) IOMMU group 7: [8086:43e0] 00:16.0 Communication controller: Intel Corporation Tiger Lake-H Management Engine Interface (rev 11) IOMMU group 8: [8086:43d2] 00:17.0 SATA controller: Intel Corporation Device 43d2 (rev 11) IOMMU group 9: [8086:43c0] 00:1b.0 PCI bridge: Intel Corporation Device 43c0 (rev 11) IOMMU group 10: [8086:43c4] 00:1b.4 PCI bridge: Intel Corporation Device 43c4 (rev 11) IOMMU group 11: [8086:43c7] 00:1b.7 PCI bridge: Intel Corporation Device 43c7 (rev 11) IOMMU group 12: [8086:43b8] 00:1c.0 PCI bridge: Intel Corporation Device 43b8 (rev 11) IOMMU group 13: [8086:43bb] 00:1c.3 PCI bridge: Intel Corporation Device 43bb (rev 11) IOMMU group 14: [8086:43bc] 00:1c.4 PCI bridge: Intel Corporation Tiger Lake-H PCI Express Root Port #5 (rev 11) IOMMU group 15: [8086:43b0] 00:1d.0 PCI bridge: Intel Corporation Tiger Lake-H PCI Express Root Port #9 (rev 11) IOMMU group 16: [8086:4385] 00:1f.0 ISA bridge: Intel Corporation Device 4385 (rev 11) [8086:43c8] 00:1f.3 Audio device: Intel Corporation Tiger Lake-H HD Audio Controller (rev 11) [8086:43a3] 00:1f.4 SMBus: Intel Corporation Tiger Lake-H SMBus Controller (rev 11) [8086:43a4] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller (rev 11) IOMMU group 17: [10de:1380] 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2) [10de:0fbc] 01:00.1 Audio device: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] (rev a1) IOMMU group 18: [10de:1c31] 02:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2200] (rev a1) [10de:10f1] 02:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) IOMMU group 19: [144d:a808] 03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [N:0:4:1] disk Samsung SSD 970 PRO 1TB__1 /dev/nvme0n1 1.02TB IOMMU group 20: [144d:a808] 04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [N:1:4:1] disk Samsung SSD 970 PRO 512GB__1 /dev/nvme1n1 512GB IOMMU group 21: [8086:15f3] 06:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) IOMMU group 22: [8086:2725] 07:00.0 Network controller: Intel Corporation Device 2725 (rev 1a) IOMMU group 23: [8086:15f3] 09:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) IOMMU group 24: [8086:1136] 0a:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 25: [8086:1136] 0b:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 26: [8086:1136] 0b:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 27: [8086:1136] 0b:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 28: [8086:1136] 0b:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Maple Ridge 4C 2020] (rev 02) IOMMU group 29: [8086:1137] 0c:00.0 USB controller: Intel Corporation Thunderbolt 4 NHI [Maple Ridge 4C 2020] IOMMU group 30: [8086:1138] 0e:00.0 USB controller: Intel Corporation Thunderbolt 4 USB Controller [Maple Ridge 4C 2020] Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Yep. That's how they're connected.

I actually ended up getting the Ultra and an 11600K. The IOMMU groupings are pretty good. I have two GPUs and two NVME drives and everything has it's own group. This was with no ACS overrides. IOMMU group 0: [8086:4c53] 00:00.0 Host bridge: Intel Corporation Device 4c53 (rev 01) IOMMU group 1: [8086:4c01] 00:01.0 PCI bridge: Intel Corporation Device 4c01 (rev 01) IOMMU group 2: [8086:4c05] 00:01.1 PCI bridge: Intel Corporation Device 4c05 (rev 01) IOMMU group 3: [8086:4c09] 00:06.0 PCI bridge: Intel Corporation Device 4c09 (rev 01) IOMMU group 4: [8086:43ed] 00:14.0 USB controller: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller (rev 11) Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 0414:a00d Giga-Byte Technology Co., Ltd USB Audio Bus 001 Device 003: ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.1 Hub Bus 001 Device 004: ID 05e3:0608 Genesys Logic, Inc. Hub Bus 001 Device 005: ID 046d:c52b Logitech, Inc. Unifying Receiver Bus 001 Device 006: ID 05e3:0608 Genesys Logic, Inc. Hub Bus 001 Device 008: ID 048d:5702 Integrated Technology Express, Inc. ITE Device Bus 001 Device 009: ID 8087:0029 Intel Corp. Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 002 Device 002: ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.1 Hub Bus 002 Device 003: ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive [8086:43ef] 00:14.2 RAM memory: Intel Corporation Tiger Lake-H Shared SRAM (rev 11) IOMMU group 5: [8086:43e0] 00:16.0 Communication controller: Intel Corporation Tiger Lake-H Management Engine Interface (rev 11) IOMMU group 6: [8086:43d2] 00:17.0 SATA controller: Intel Corporation Device 43d2 (rev 11) IOMMU group 7: [8086:43c0] 00:1b.0 PCI bridge: Intel Corporation Device 43c0 (rev 11) [8086:43c4] 00:1b.4 PCI bridge: Intel Corporation Device 43c4 (rev 11) IOMMU group 8: [8086:43c2] 00:1b.2 PCI bridge: Intel Corporation Device 43c2 (rev 11) IOMMU group 9: [8086:43c3] 00:1b.3 PCI bridge: Intel Corporation Device 43c3 (rev 11) IOMMU group 10: [8086:43b8] 00:1c.0 PCI bridge: Intel Corporation Device 43b8 (rev 11) [8086:43bc] 00:1c.4 PCI bridge: Intel Corporation Tiger Lake-H PCI Express Root Port #5 (rev 11) IOMMU group 11: [8086:43b0] 00:1d.0 PCI bridge: Intel Corporation Tiger Lake-H PCI Express Root Port #9 (rev 11) IOMMU group 12: [8086:43b4] 00:1d.4 PCI bridge: Intel Corporation Device 43b4 (rev 11) IOMMU group 13: [8086:4385] 00:1f.0 ISA bridge: Intel Corporation Device 4385 (rev 11) [8086:43c8] 00:1f.3 Audio device: Intel Corporation Tiger Lake-H HD Audio Controller (rev 11) [8086:43a3] 00:1f.4 SMBus: Intel Corporation Tiger Lake-H SMBus Controller (rev 11) [8086:43a4] 00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller (rev 11) IOMMU group 14: [10de:1c31] 01:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2200] (rev a1) [10de:10f1] 01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) IOMMU group 15: [10de:1380] 02:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2) [10de:0fbc] 02:00.1 Audio device: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] (rev a1) IOMMU group 16: [144d:a808] 03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [N:0:4:1] disk Samsung SSD 970 PRO 1TB__1 /dev/nvme0n1 1.02TB IOMMU group 17: [8086:15f3] 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03) IOMMU group 18: [8086:2723] 06:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a) IOMMU group 19: [144d:a808] 0a:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [N:1:4:1] disk Samsung SSD 970 PRO 512GB__1 /dev/nvme1n1 512GB

@giganode Thanks! I ended up getting one of those and selling it because I couldn’t find any SATA breakout cables.

You should go with the Ultra, at least based on the block diagram. That way both GPUs get an x8 link to the CPU. Ultimately that will depend on the IOMMU groupings. I’m about to do some testing with an 11700K and a Z590 Master soon. I’ll let you know how it goes. The Master and the Ultra are very similar.

I set up a few more VMs recently and I found out that the culprit was the virtio-net setting under Network Model in the VM configuration. When I used virtio I didn't have any problems.

What’s crazy is that I changed it back to 8 and it still works.

Thanks Vr2lo! That fixed it. Do you know why this is necessary?

I recently added a new Windows 10 VM to my UnRAID server using Q35-5.1 and virtio-win-0.1.190.1 and and I noticed when copying from the VM to the cache drive it was significantly slower than an older VM running Q35-4.2 and virtio-win-0.1.173-2. To verify, I cloned the old VM template and did a fresh Windows install and confirmed that it was much faster than using Q35-5.1 and virtio-win-0.1.190.1. I also tried i440fx-5.1 , but it was slower than Q35-5.1. Is this a known issue? With Q35-5.1 / virtio-win-0.1.190-1: With Q35-4.2 / virtio-win-0.1.173-2: tower-diagnostics-20210129-0347.zip

I pass both Matisse Controllers to my VM. I'm using multifunction to pass the whole IOMMU group.

This now works. Something must have changed with PIA recently. Thanks @tjb_altf4!

When I use those addresses for the DNS I get the error message below in my log, and it repeats until I stop the docker. 2020-11-16 15:45:18,639 DEBG 'start-script' stdout output: [info] Adding 10.0.0.242 to /etc/resolv.conf 2020-11-16 15:45:18,641 DEBG 'start-script' stdout output: [info] Adding 10.0.0.243 to /etc/resolv.conf 2020-11-16 15:45:48,677 DEBG 'start-script' stderr output: Error: error sending query: Could not send or receive, because of network error

Sorry. I should've been more specific. I'm not using proxy settings in my brower (Edge). I have the internet connection set up to run through a proxy. This was working fine until the recent changes with PIA. I even manually set the DNS in TCP/IP properties. The address was complete in the proxy server settings. I just removed some of it to take the snapshot. This is running on a VM.

Anyone else having issues with DNS leaks since the changover with PIA? My DNS settings are set to the PIA DNS servers: 209.222.18.218, 209.222.18.222. No matter what it defaults to the one shown below. Privoxy PIA App

Those are the correct devices. With the latest unRAID beta you can determine which drive is which using System Devices if you know the serial number, since it gives you the device name.

I did my testing with the Asus Zenith II Alpha. I used the two x16 slots. No overrides or unsafe interrupts. I dumped the bios files in a bare metal windows installation and removed the header from the ROM files with HxD.

I was able to pass a secondary GPU without the error 43.

I’ve only tried with a single GPU, using a dumped vbios and passing the GPU as a multifunction device. I’ll try adding a second GPU and see how it goes.

I've only tested the Aorus Xtreme and the Zenith II Extreme (as well as the Alpha) with a GPU. My experience with all of the boards was very similar. You shouldn't have any problems with the Master.

I use a VM as my daily driver with this motherboard and I don’t have any issues. I’m using a dumped vbios and I pass the GPU through as a multifunction device. I’m running the latest unRAID beta.

This is what was causing turbo write to be disabled for me. Switching to high water allowed turbo write to work as expected. Thanks JorgeB!