jbartlett Posted June 7, 2019 Author Share Posted June 7, 2019 2 hours ago, macmus said: So NUMA#0 or NUMA#1 goes to first PCIE slot ? PCIe 1 is on numa 2 for the 2990WX, CPUs 8-15/40-47 PCIe 3 & 5 are on numa 0, CPUs 0-7/32-39 Numa 1 has CPUs 16-23/48-55 Numa 3 has CPUs 24-31/56-63 Quote Link to comment
authorleon Posted June 7, 2019 Share Posted June 7, 2019 Hello all, First successful boot. Everything seems to be working. Just testing it with Windows 10.. Will keep you fully updated cannot wait to get unraid on it Quote Link to comment
jbartlett Posted June 7, 2019 Author Share Posted June 7, 2019 I watched a LTT video last night that said the Noctua NH-U12A out performs AIO water coolers in both cooling & sound - but custom loops can still out perform the Nactua given a good pump & large reservoir. I put a Noctua NH-U14S on mine. Don't you love that BIOS display? Quote Link to comment
jbartlett Posted June 7, 2019 Author Share Posted June 7, 2019 Here's my current build. This is a VM box so SSD's for hosting the VMs (one per VM) and a spinner for backups of those VMs. I just realized that I haven't had two Win 10 VM's running at the same time. Gotta test that soon! PCIe 1: GeForce GT 1030 (OS) PCIe 2: High Point 4-Port 4-Controller USB 3.0 PCI-Express 2.0 x 4 HBA RocketU 1144D PCIe 3: Quadro P2000 (Win 10) PCIe 4: Empty PCIe 5: Quadro P620 (Win 10) Quote Link to comment
macmus Posted June 8, 2019 Share Posted June 8, 2019 I don't get this output why i don't see my primary graphic card in IOMMU group? Which NUMA node it's using if it's not in IOMMU ? Is it using chipset lanes or what? root@ubuntu:~# find /sys/kernel/iommu_groups/ -type l /sys/kernel/iommu_groups/17/devices/0000:09:00.0 /sys/kernel/iommu_groups/7/devices/0000:00:03.1 /sys/kernel/iommu_groups/25/devices/0000:0e:00.2 /sys/kernel/iommu_groups/15/devices/0000:00:19.6 /sys/kernel/iommu_groups/15/devices/0000:00:19.4 /sys/kernel/iommu_groups/15/devices/0000:00:19.2 /sys/kernel/iommu_groups/15/devices/0000:00:19.0 /sys/kernel/iommu_groups/15/devices/0000:00:19.7 /sys/kernel/iommu_groups/15/devices/0000:00:19.5 /sys/kernel/iommu_groups/15/devices/0000:00:19.3 /sys/kernel/iommu_groups/15/devices/0000:00:19.1 /sys/kernel/iommu_groups/5/devices/0000:00:02.0 /sys/kernel/iommu_groups/23/devices/0000:0d:00.3 /sys/kernel/iommu_groups/13/devices/0000:00:14.3 /sys/kernel/iommu_groups/13/devices/0000:00:14.0 /sys/kernel/iommu_groups/3/devices/0000:00:01.3 /sys/kernel/iommu_groups/21/devices/0000:0d:00.0 /sys/kernel/iommu_groups/11/devices/0000:00:08.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.1 /sys/kernel/iommu_groups/18/devices/0000:0a:00.0 /sys/kernel/iommu_groups/8/devices/0000:00:04.0 /sys/kernel/iommu_groups/26/devices/0000:0e:00.3 /sys/kernel/iommu_groups/16/devices/0000:03:00.0 /sys/kernel/iommu_groups/16/devices/0000:02:00.0 /sys/kernel/iommu_groups/16/devices/0000:08:00.0 /sys/kernel/iommu_groups/16/devices/0000:01:00.2 /sys/kernel/iommu_groups/16/devices/0000:02:03.0 /sys/kernel/iommu_groups/16/devices/0000:01:00.0 /sys/kernel/iommu_groups/16/devices/0000:07:00.0 /sys/kernel/iommu_groups/16/devices/0000:02:09.0 /sys/kernel/iommu_groups/16/devices/0000:06:00.0 /sys/kernel/iommu_groups/16/devices/0000:02:02.0 /sys/kernel/iommu_groups/16/devices/0000:05:00.0 /sys/kernel/iommu_groups/16/devices/0000:01:00.1 /sys/kernel/iommu_groups/16/devices/0000:02:01.0 /sys/kernel/iommu_groups/16/devices/0000:02:04.0 /sys/kernel/iommu_groups/6/devices/0000:00:03.0 /sys/kernel/iommu_groups/24/devices/0000:0e:00.0 /sys/kernel/iommu_groups/14/devices/0000:00:18.3 /sys/kernel/iommu_groups/14/devices/0000:00:18.1 /sys/kernel/iommu_groups/14/devices/0000:00:18.6 /sys/kernel/iommu_groups/14/devices/0000:00:18.4 /sys/kernel/iommu_groups/14/devices/0000:00:18.2 /sys/kernel/iommu_groups/14/devices/0000:00:18.0 /sys/kernel/iommu_groups/14/devices/0000:00:18.7 /sys/kernel/iommu_groups/14/devices/0000:00:18.5 /sys/kernel/iommu_groups/4/devices/0000:00:01.5 /sys/kernel/iommu_groups/22/devices/0000:0d:00.2 /sys/kernel/iommu_groups/12/devices/0000:00:08.1 /sys/kernel/iommu_groups/2/devices/0000:00:01.2 /sys/kernel/iommu_groups/20/devices/0000:0c:00.0 /sys/kernel/iommu_groups/20/devices/0000:0c:00.1 /sys/kernel/iommu_groups/10/devices/0000:00:07.1 /sys/kernel/iommu_groups/0/devices/0000:00:01.0 /sys/kernel/iommu_groups/19/devices/0000:0b:00.0 /sys/kernel/iommu_groups/9/devices/0000:00:07.0 root@ubuntu:~# lspci 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 00:01.5 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 00:19.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 00:19.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02) 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02) 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02) 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) 02:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) 03:00.0 Network controller: Intel Corporation Wireless-AC 9260 (rev 29) 05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) 06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 07:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) 08:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller 09:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02) 0a:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller 0b:00.0 Non-Volatile memory controller: Sandisk Corp WD Black 2018/PC SN720 NVMe SSD 0c:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1) 0c:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1) 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a 0d:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor 0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller 0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455 0e:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) 0e:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller 40:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex 40:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B 41:00.0 VGA compatible controller: NVIDIA Corporation GV102 (rev a1) 41:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1) 41:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Controller (rev a1) 41:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 UCSI Controller (rev a1) 42:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a 42:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor 42:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455 43:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) root@ubuntu:~# lspci -tvv -+-[0000:40]-+-00.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex | +-00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit | +-01.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge | +-02.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge | +-03.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge | +-03.1-[41]--+-00.0 NVIDIA Corporation GV102 | | +-00.1 NVIDIA Corporation TU102 High Definition Audio Controller | | +-00.2 NVIDIA Corporation TU102 USB 3.1 Controller | | \-00.3 NVIDIA Corporation TU102 UCSI Controller | +-04.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge | +-07.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge | +-07.1-[42]--+-00.0 Advanced Micro Devices, Inc. [AMD] Device 145a | | +-00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor | | \-00.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller | +-08.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge | \-08.1-[43]--+-00.0 Advanced Micro Devices, Inc. [AMD] Device 1455 | \-00.2 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] \-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex +-00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit +-01.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge +-01.1-[01-08]--+-00.0 Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller | +-00.1 Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller | \-00.2-[02-08]--+-00.0-[03]----00.0 Intel Corporation Wireless-AC 9260 | +-01.0-[04]-- | +-02.0-[05]----00.0 Intel Corporation I211 Gigabit Network Connection | +-03.0-[06]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller | +-04.0-[07]----00.0 Renesas Technology Corp. uPD720201 USB 3.0 Host Controller | \-09.0-[08]----00.0 ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller +-01.2-[09]----00.0 Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] +-01.3-[0a]----00.0 ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller +-01.5-[0b]----00.0 Sandisk Corp WD Black 2018/PC SN720 NVMe SSD +-02.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge +-03.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge +-03.1-[0c]--+-00.0 NVIDIA Corporation GK208B [GeForce GT 710] | \-00.1 NVIDIA Corporation GK208 HDMI/DP Audio Controller +-04.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge +-07.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge +-07.1-[0d]--+-00.0 Advanced Micro Devices, Inc. [AMD] Device 145a | +-00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor | \-00.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller +-08.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge +-08.1-[0e]--+-00.0 Advanced Micro Devices, Inc. [AMD] Device 1455 | +-00.2 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] | \-00.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller +-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller +-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge +-18.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 +-18.1 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 +-18.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 +-18.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 +-18.4 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 +-18.5 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 +-18.6 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 +-18.7 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 +-19.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 +-19.1 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 +-19.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 +-19.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 +-19.4 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 +-19.5 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 +-19.6 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 \-19.7 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 Quote Link to comment
macmus Posted June 8, 2019 Share Posted June 8, 2019 I also see that graphic card is not detected in BIOS nor in mapped to IOMMU groups at all Where is it ? which Numa ? ;/ Quote Link to comment
GreenEyedMonster Posted June 8, 2019 Share Posted June 8, 2019 10 hours ago, jbartlett said: Here's my current build. This is a VM box so SSD's for hosting the VMs (one per VM) and a spinner for backups of those VMs. I just realized that I haven't had two Win 10 VM's running at the same time. Gotta test that soon! PCIe 1: GeForce GT 1030 (OS) PCIe 2: High Point 4-Port 4-Controller USB 3.0 PCI-Express 2.0 x 4 HBA RocketU 1144D PCIe 3: Quadro P2000 (Win 10) PCIe 4: Empty PCIe 5: Quadro P620 (Win 10) Any way to passthrough the primary GPU as well? I read some where that you can? Quote Link to comment
authorleon Posted June 8, 2019 Share Posted June 8, 2019 14 hours ago, jbartlett said: Here's my current build. This is a VM box so SSD's for hosting the VMs (one per VM) and a spinner for backups of those VMs. I just realized that I haven't had two Win 10 VM's running at the same time. Gotta test that soon! PCIe 1: GeForce GT 1030 (OS) PCIe 2: High Point 4-Port 4-Controller USB 3.0 PCI-Express 2.0 x 4 HBA RocketU 1144D PCIe 3: Quadro P2000 (Win 10) PCIe 4: Empty PCIe 5: Quadro P620 (Win 10) Hello John and all, Right, I am going to start now. Please can you tell me which is the best USB port to use for the UNRAID OS. Thank you Quote Link to comment
jbartlett Posted June 9, 2019 Author Share Posted June 9, 2019 16 hours ago, authorleon said: Please can you tell me which is the best USB port to use for the UNRAID OS. Pick a port. Any port. Except ones you pass through to a VM, don't pick those. Quote Link to comment
jbartlett Posted June 9, 2019 Author Share Posted June 9, 2019 21 hours ago, GreenEyedMonster said: Any way to passthrough the primary GPU as well? I read some where that you can? Might be possible based on a Spaceinvader One video with passing through the card's ROM to it. I'm going to be looking into that because I'd rather have the two Quadro cards on separate nodes. Quote Link to comment
authorleon Posted June 9, 2019 Share Posted June 9, 2019 14 hours ago, jbartlett said: Pick a port. Any port. Except ones you pass through to a VM, don't pick those. Thank you for the information. I'm using the default profile to over clock it to 4 GHz. But sometimes it seems to be a bit on stable by Windows 10 locking up. What is your profile regarding overclocking? Thank you very much Quote Link to comment
jbartlett Posted June 9, 2019 Author Share Posted June 9, 2019 1 hour ago, authorleon said: What is your profile regarding overclocking? Haven't messed with the OC yet. Quote Link to comment
jbartlett Posted June 10, 2019 Author Share Posted June 10, 2019 A video card in PCIe 1 (primary video) can be passed through to a VM if you dump the cards BIOS and pass it in. Quote Link to comment
jbartlett Posted June 10, 2019 Author Share Posted June 10, 2019 Might have spoke too soon. The video card does indeed utilize but the P2000 is reporting that it found problems. Still mucking with it. Quote Link to comment
authorleon Posted June 12, 2019 Share Posted June 12, 2019 FYI. Just to let you know I am getting stability issues at 4 GHz. Really high temperatures. VRM Get a little bit too hot. I will put everything back. And install UNRAID. Quote Link to comment
testdasi Posted June 12, 2019 Share Posted June 12, 2019 (edited) On 6/8/2019 at 12:27 PM, authorleon said: Please can you tell me which is the best USB port to use for the UNRAID OS. Best ports, in decreasing order of "best-ness": Internal USB 2.0 ports. You can buy a cheapo USB2.0 internal to external adapter (basically a small little cable with internal USB 2.0 plug on one end and external USB2.0 socket on the other) and plug the Unraid USB stick on there. Alternatively, you can use your case USB2.0 header or some all-in-one USB2.0 internal hub. Reason for "best-ness": (1) USB2.0 has the least compatibility issues when booting (almost none); (2) few people have any other use for internal USB2.0 headers and (3) the USB2.0 motherboard controller can't be passed through to a VM even with ACS multifunction (many have tried and many have failed). Internal USB 3.0 ports. Similar to USB2.0 above, just that USB3.0 ports sometimes have boot issues. External USB 3.1 port (the red one at the back). This port can't be passed through to a VM similar to the internal USB2.0 port but give it's externally available, you might want to use it for some other purposes instead of booting Unraid (about 10s saving in boot time and that's it). External USB 3.0 ports (the 8 blue ones at the back). These are on their own controllers (2 controllers, 4 ports each) and in their own IOMMU group so they can be passed through to your VM with ease (need vfio stubbing). Note that they have identical IDs so if you stub 1 controller, you will end up stubbing both (I vaguely remember there's a way around that so you might want to google if required). USB 3.1 type C - because why would want to do that? 2 hours ago, authorleon said: I am getting stability issues at 4 GHz Highly NOT recommended to overclock Unraid. Instability on a server is at best a nuisance (e.g. needing to run Parity check whenever things crash) and at worst causing crippling issues (e.g. data corruption). You are better off to use AMD Precision Boost. It's an effort-free per-core on-demand overclock. Sure all-core boost can only reach 3.8GHz but that's only 5% lower than your best-effort overclock that more likely than not is less stable and definitely uses a lot more power. On a side note, don't use the latest BIOS (AGESA 1.1.0.2), use the one right before that (AGESA 1.1.0.1a). The current BIOS has AMD PB cut down a little bit on the voltage which runs 1-2 degrees cooler but has caused me some unexpected stability issue with certain workloads. Edited June 12, 2019 by testdasi 1 Quote Link to comment
jbartlett Posted June 12, 2019 Author Share Posted June 12, 2019 The 1st & 2nd PCIe slots both go through Numa node 2. Updated OP. Not sure where the 4th shared PCIe port routs too yet, I should get a PCIe 1x extension cable tomorrow that I can plug something into (just need it to detect). Quote Link to comment
jbartlett Posted June 12, 2019 Author Share Posted June 12, 2019 (edited) Using a nVidia card in slot 1 as UNRAID's video but then passed through to a VM will cause the nVidia drivers to throw a Code 43 (KVM detected) and not initialize. Nothing I've tried fixed it including patching the driver & adding stuff to the VM XML worked. The only thing that works is to dedicate a video card to UNRAID in slot 1 and put the nVidia card in a different slot. Edited June 12, 2019 by jbartlett Quote Link to comment
jordanmw Posted June 12, 2019 Share Posted June 12, 2019 Obviously not the same setup, but I have the x399 taichi and passing through a 960 to slot 1. I have all 4 slots populated with GPUs passed through- slot 1 displays the unraid boot until I start the VM in slot 1- then it takes over the video. The only way I could make that work is to dump and edit the rom- then pass that through. I was getting code 43 until I did that. I'm sure you probably already tried, but thought I would share my experience in case any of that helps. Quote Link to comment
jbartlett Posted June 12, 2019 Author Share Posted June 12, 2019 1 hour ago, jordanmw said: Obviously not the same setup, but I have the x399 taichi and passing through a 960 to slot 1. I have all 4 slots populated with GPUs passed through- slot 1 displays the unraid boot until I start the VM in slot 1- then it takes over the video. The only way I could make that work is to dump and edit the rom- then pass that through. I was getting code 43 until I did that. I'm sure you probably already tried, but thought I would share my experience in case any of that helps. Editing the ROM. That's something I didn't do. Hrm. Quote Link to comment
jbartlett Posted June 13, 2019 Author Share Posted June 13, 2019 I tested the Quadro P2000's performance on numa core & off using the infinity mesh, difference was negligible - 0.3% less off node. 3D Mark did keep hanging on hardware scanning after 1 or 2 tests which required a reboot of the VM. Toughest gaming card not in use is a 980ti that I may try for shits & grins. Quote Link to comment
itimpi Posted June 13, 2019 Share Posted June 13, 2019 (edited) @testdasiIn your list of options you left out 1a) USB 2 external ports? They always seem to work better than any type of USB3 ports. You also did not mention that USB3 drives seem to have a much higher failure rate. My suspicion is that they tend to run hot which is not good if they are going to run 24x7. Edited June 13, 2019 by itimpi Quote Link to comment
jordanmw Posted June 13, 2019 Share Posted June 13, 2019 15 hours ago, jbartlett said: Editing the ROM. That's something I didn't do. Hrm. Yeah, I had to use a hex editor to remove the nvidia headers before I could pass the rom properly. Quote Link to comment
jbartlett Posted June 13, 2019 Author Share Posted June 13, 2019 2 hours ago, jordanmw said: Yeah, I had to use a hex editor to remove the nvidia headers before I could pass the rom properly. Did you use GPU-Z to extract the ROM or download it? If so, that sounds like you removed the GPU-Z stuff that it adds to the start of the ROM. Quote Link to comment
jordanmw Posted June 13, 2019 Share Posted June 13, 2019 (edited) Yeah- gpu-z for extraction- then deleted the header as spaceinvader's video talked about. Shown at 2:20 here: Edited June 14, 2019 by jordanmw Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.