commotion Posted July 3, 2020 Posted July 3, 2020 (edited) I would like to say thanks to the unraid community and a special shout out to Space Invader One. This build is driving me nuts. Everything seems to be working on a hardware level .. i haven't tested my dockers other then plex works (no hardware trans coding yet though) VM's when attempting to pass my Video cards the vm is super slow. like would take 3 hours to click on everything to get through a windows 10 setup. Thanks in advance and i owe some beers to the person that helps me get to the bottom of this... Back Story: 3 years ago I repurposed my old gaming rig as a Unraid server Specs : Asus Sabertooth with I7 Extreme. 4x10tb WD white labels from shucked drives. 1 Samsung 860 Evo Asus GTX 1070 Nvidia p2000 passed through to Plex for trans coding Dockers: Plex Shinobi Duplicati VMs: Windows 2012 r2 Domain Controller Windows 2012 r2 application server Windows 10 daily driver GPU pass through to GTX 1070 Passed through onboard USB controller (mouse and Keyboard) Everything Ran fine Zero issues Fast forward to COVID-19 - Since im trapped in my house ill build a crazy unraid build Specs: https://pcpartpicker.com/list/RDCKtp I brought over the p2000 my WD reds and my Unraid USB drive. Fire it up had some issues trying to figure out which videcards which in the system Dockers fired right up (except plex) VM’s fired up except my daily driver (Videocard passthough needed configured) I could not get my daily driver to hook up to the GPUs either one of the RTX2080’s What I’ve done so far: Updated the ASRock x399m Bios to 3.80 (https://www.asrock.com/mb/AMD/X399M%20Taichi/index.asp#BIOS) I've disabled C-states in BIOS which seemed to fix a initial random freezing that was getting. Checked and IOMMU Groups were grouped up to much PCIe ACS override to Enabled I have the following plugins installed · Unraid Nvidia – For some reason the 2 RTX2080s will show up but not he p2000 (All 3 show up in the device listing) 1. Nvidia 6.9.0 beta22 -Nvidia Driver Version:440.82 – Error NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. 2. Nvidia 6.8.3 – same thing as above · VFIO-PCI Config Ive added another Samsung EVO and passed it directly to a new vm Windows 10 super slow like 25 mins to even boot up the ISO for windows 10 Created another VM for windows 10 made it create a image file directly on the SSD. Same issue. Tried another SSD same issue. I've looked at just about every Spaceinvader One's videos regarding GPU or PCIe pass through and his AMD build and issues and ways he has over came them. I'm also not able to get my p2000 to pass though to plex like i did before its not even showing up in the nvidia plugin. I looked at the Numa Nodes and the ones im using for the VM is the same ones my RTX 2080s are on. RTX2080super seems to have a bunch of extra stuff with it. So i passed it to the vm also NVIDIA TU104 USB 3.1 Host Controller | USB controller (41:00.2) NVIDIA TU104 USB Type-C UCSI Controller | Serial bus controller (41:00.3) I've also pinned my cpus 8/40,9/41,10/42,11/43,12/44,13/45,14/46,15/47 and assigned them to the VM i tried with half and doubled it but it didn't make any difference Unraid System Devices IOMMU group 0: [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 1: [1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 2: [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 3: [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 4: [1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 5: [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 6: [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 7: [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 8: [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 9: [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 10: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 11: [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 12: [1022:1460] 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:19.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:19.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 13: [1022:1460] 00:1a.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:1a.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:1a.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:1a.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:1a.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:1a.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:1a.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:1a.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 14: [1022:1460] 00:1b.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:1b.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:1b.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:1b.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:1b.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:1b.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:1b.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:1b.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 15: [1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02) IOMMU group 16: [1022:43b6] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02) IOMMU group 17: [1022:43b1] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02) IOMMU group 18: [1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) IOMMU group 19: [1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) IOMMU group 20: [1022:43b4] 02:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) IOMMU group 21: [1022:43b4] 02:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) IOMMU group 22: [1022:43b4] 02:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) IOMMU group 23: [8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 24: [8086:24fb] 05:00.0 Network controller: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10) IOMMU group 25: [8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 26: [10de:1c30] 08:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2000] (rev a1) IOMMU group 27: [10de:10f1] 08:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) IOMMU group 28: [1022:145a] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 29: [1022:1456] 09:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 30: [1022:145f] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller IOMMU group 31: [1022:1455] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 32: [1022:7901] 0a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 33: [1022:1457] 0a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller IOMMU group 34: [1022:1452] 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 35: [1022:1452] 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 36: [1022:1452] 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 37: [1022:1452] 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 38: [1022:1452] 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 39: [1022:1454] 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 40: [1022:1452] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 41: [1022:1454] 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 42: [1022:145a] 21:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 43: [1022:1456] 21:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 44: [1022:1455] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 45: [1022:1452] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 46: [1022:1453] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 47: [1022:1452] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 48: [1022:1452] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 49: [1022:1453] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 50: [1022:1452] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 51: [1022:1452] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 52: [1022:1454] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 53: [1022:1452] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 54: [1022:1454] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 55: [10de:1e81] 41:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 SUPER] (rev a1) IOMMU group 56: [10de:10f8] 41:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1) IOMMU group 57: [10de:1ad8] 41:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) IOMMU group 58: [10de:1ad9] 41:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1) IOMMU group 59: [10de:1e81] 42:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 SUPER] (rev a1) IOMMU group 60: [10de:10f8] 42:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1) IOMMU group 61: [10de:1ad8] 42:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) IOMMU group 62: [10de:1ad9] 42:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1) IOMMU group 63: [1022:145a] 43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 64: [1022:1456] 43:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 65: [1022:145f] 43:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller IOMMU group 66: [1022:1455] 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 67: [1022:7901] 44:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 68: [1022:1452] 60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 69: [1022:1452] 60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 70: [1022:1452] 60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 71: [1022:1452] 60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 72: [1022:1452] 60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 73: [1022:1454] 60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 74: [1022:1452] 60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 75: [1022:1454] 60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 76: [1022:145a] 61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 77: [1022:1456] 61:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 78: [1022:1455] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function CPU Thread Pairings Pair 1: cpu 0 / cpu 32 Pair 2: cpu 1 / cpu 33 Pair 3: cpu 2 / cpu 34 Pair 4: cpu 3 / cpu 35 Pair 5: cpu 4 / cpu 36 Pair 6: cpu 5 / cpu 37 Pair 7: cpu 6 / cpu 38 Pair 8: cpu 7 / cpu 39 Pair 9: cpu 8 / cpu 40 Pair 10: cpu 9 / cpu 41 Pair 11: cpu 10 / cpu 42 Pair 12: cpu 11 / cpu 43 Pair 13: cpu 12 / cpu 44 Pair 14: cpu 13 / cpu 45 Pair 15: cpu 14 / cpu 46 Pair 16: cpu 15 / cpu 47 Pair 17: cpu 16 / cpu 48 Pair 18: cpu 17 / cpu 49 Pair 19: cpu 18 / cpu 50 Pair 20: cpu 19 / cpu 51 Pair 21: cpu 20 / cpu 52 Pair 22: cpu 21 / cpu 53 Pair 23: cpu 22 / cpu 54 Pair 24: cpu 23 / cpu 55 Pair 25: cpu 24 / cpu 56 Pair 26: cpu 25 / cpu 57 Pair 27: cpu 26 / cpu 58 Pair 28: cpu 27 / cpu 59 Pair 29: cpu 28 / cpu 60 Pair 30: cpu 29 / cpu 61 Pair 31: cpu 30 / cpu 62 Pair 32: cpu 31 / cpu 63 USB Devices Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive Bus 001 Device 003: ID 8087:0aa7 Intel Corp. Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub SCSI Devices [0:0:0:0] disk Samsung Flash Drive FIT 1100 /dev/sda 32.0GB [1:0:0:0] disk ATA WDC WD100EMAZ-00 0A83 /dev/sdb 10.0TB [2:0:0:0] disk ATA WDC WD100EMAZ-00 0A83 /dev/sdc 10.0TB [3:0:0:0] disk ATA WDC WD100EMAZ-00 0A83 /dev/sdd 10.0TB [4:0:0:0] disk ATA WDC WD100EMAZ-00 0A83 /dev/sde 10.0TB [5:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdf 1.00TB [6:0:0:0] disk ATA Samsung SSD 860 3B6Q /dev/sdg 1.00TB [7:0:0:0] disk ATA KINGSTON SH103S3 BBF0 /dev/sdh 240GB SYSLINUX Configuration: default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS menu default kernel /bzimage append rcu_nocbs=0-63 vfio-pci.ids=1022:145f pcie_acs_override=downstream,multifunction isolcpus=8-15,40-47 initrd=/bzroot label Unraid OS GUI Mode kernel /bzimage append rcu_nocbs=0-63 pcie_acs_override=downstream,multifunction isolcpus=8-15,40-47 initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest Windows 10 VM XML <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>Windows10</name> <uuid>dfd9762f-b76b-19b9-7deb-5fbc4d715e3b</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='8'/> <vcpupin vcpu='1' cpuset='40'/> <vcpupin vcpu='2' cpuset='9'/> <vcpupin vcpu='3' cpuset='41'/> <vcpupin vcpu='4' cpuset='10'/> <vcpupin vcpu='5' cpuset='42'/> <vcpupin vcpu='6' cpuset='11'/> <vcpupin vcpu='7' cpuset='43'/> <vcpupin vcpu='8' cpuset='12'/> <vcpupin vcpu='9' cpuset='44'/> <vcpupin vcpu='10' cpuset='13'/> <vcpupin vcpu='11' cpuset='45'/> <vcpupin vcpu='12' cpuset='14'/> <vcpupin vcpu='13' cpuset='46'/> <vcpupin vcpu='14' cpuset='15'/> <vcpupin vcpu='15' cpuset='47'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/dfd9762f-b76b-19b9-7deb-5fbc4d715e3b_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/disk/by-id/ata-KINGSTON_SH103S3240G_50026B732C01BBE2'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/Windows10.iso'/> <target dev='hda' bus='sata'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.171.iso'/> <target dev='hdb' bus='sata'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0xf'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='9' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='10' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='10' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:ea:4a:f4'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/domains/vBIOSGPUROM/EVGA.RTX2080Super.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x09' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x43' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain> Edited July 10, 2020 by commotion Quote
commotion Posted July 7, 2020 Author Posted July 7, 2020 Ya im at a total loss here would love for this to work well on this cpu. I might have to switch back to Intel.. Quote
Jerky_san Posted July 7, 2020 Posted July 7, 2020 (edited) 51 minutes ago, commotion said: Ya im at a total loss here would love for this to work well on this cpu. I might have to switch back to Intel.. I'm on my phone right now but I'll look over this next time I get to my desk. Looking at the picture you posted it appears your using a 2990wx correct? It can be slightly difficult to say the least. Edit saw the parts picker thing. So yeah 2990wx. It only has two numa so it will make it all fun. Edited July 7, 2020 by Jerky_san Quote
commotion Posted July 7, 2020 Author Posted July 7, 2020 Ugh I don't like the sound of this.. lol I was running ubraid on a 8+ year old i7 extreme and I was able to actually use the vm as a daily driver. This is unusable takes 6 minutes to boot up (using unassigned disk -samsung sata ssd evo 860) once I type my password to login to windows 10 it takes ~30 seconds for it to print thr password in the box. Then another 6 minutes for windows to open. If I manage to get the task manager open everything is at under 5 % mostly sitting at 0% but when I move the mouse its super lagged and delayed. Any help would be great 👍 Quote
Jerky_san Posted July 7, 2020 Posted July 7, 2020 Just now, commotion said: Ugh I don't like the sound of this.. lol I was running ubraid on a 8+ year old i7 extreme and I was able to actually use the vm as a daily driver. This is unusable takes 6 minutes to boot up (using unassigned disk -samsung sata ssd evo 860) once I type my password to login to windows 10 it takes ~30 seconds for it to print thr password in the box. Then another 6 minutes for windows to open. If I manage to get the task manager open everything is at under 5 % mostly sitting at 0% but when I move the mouse its super lagged and delayed. Any help would be great 👍 I looked through your XML and it appears "normal/good". Before the newer versions you had to do a lot more customizations. The C-State fix shouldn't be required btw. If your getting freezing you'll need to do the option where it says "power supply idle" - normal instead of "low power". I'm unsure what this option is called in your particular board. Do you have "Tips and Tweaks" plugin installed? If you do make sure the CPU Scaling Governor is on "On Demand" and the Enable Intel Turbo/AMD Performance Boost setting is enabled as well. (Just working through the beginning things btw since I saw you watched space invader's very helpful vids. Just want to cover all the basis). On the windows install/vm. Have you tried just passing the ssd and nothing else to see where performance is landing you? Quote
testdasi Posted July 7, 2020 Posted July 7, 2020 On 7/3/2020 at 6:56 AM, commotion said: What I’ve done so far: Updated the ASRock x399m Bios to 3.80 (https://www.asrock.com/mb/AMD/X399M%20Taichi/index.asp#BIOS) I've disabled C-states in BIOS which seemed to fix a initial random freezing that was getting. First thing, turn C State back on. Global C States Control is no longer a fix for freezes while causing severe lags on Threadripper. Beside idle current to normal as jerky_san said, you should also turn off Precision Boost Overdrive. Normal PB is fine but PBO in recent BIOS has been unstable due to AMD cutting the voltage too low. That could be the cause of your random freezing. And please use the forum code functionality (the </> button next to the smiley button) when copy-paste text from Unraid e.g. config / xml etc. That will format and section the text correctly to make it easier to track things in your post. Otherwise everything is just the same blob of text of the same format, making it very hard to read. You are also better off having each problem on its own topic with only the relevant information. Unraid Nvidia issue is best asked in the LSIO dedicated support topic for the plugin. Quote
Jerky_san Posted July 8, 2020 Posted July 8, 2020 @commotion So some other things you need to do.. In the bios under I believe at least in the zenith was "CBS" there is a place to set how the memory channels are set. You need to set it to die I believe. Look for IOMMU and make sure its set to "enable" instead of "auto". Also I don't remember the IOMMU groupings being bad on my tachi so I'm surprised they'd be bad on yours but I only had the tachi for a short time. There is some numatune settings to your xml file you can do to optimize your settings once we get performance slightly better. The reason you don't want to do them right now is because it makes the gui editor not work as well. Quote
Jerky_san Posted July 8, 2020 Posted July 8, 2020 (edited) @commotion I should also mention that there is "another way" that kind of bypasses a lot of problems with QEMU but it involves virtualization of Unraid and running the free version of ESXi. I did that and it's basically the best of both worlds to me. It solved all my "AMD" problems since QEMU hasn't caught up yet, took away the hassle of customizing the XML heavily to fit threadripper, and also gives me the ability to better control everything. Basically unraid is now storage/dockers. ESXi is everything else like gaming machine, pfsense, and other things I've been toying with. The ONLY major problem I had was sorting out which USB controller would pass through. You could easily bypass that problem since you have 2080ti's that have onboard USB3.0's I believe. Anyways it solved all my hitching and stuff I'd have with my gaming VM in the old days. The only problem is that I sometimes get high ready time if I'm doing a heavy amount of downloading(writing/reading) and utilizing my machine at the same time. Slight stutter here and there. Also I never was able to pass the audio card so I just use USB audio that I have from my old "all in one zfs" box. Edit: ESXi 7.0 that is.. it also allows you to boot from USB which solves a lot of the old problems and slow boot times. Edited July 8, 2020 by Jerky_san Quote
commotion Posted July 8, 2020 Author Posted July 8, 2020 (edited) Wow thanks so much to the both of you. This is running 2000% better. 14 seconds boot time now... amazing Now that I have the bios settings correct in going to go back through the tweaks to make sure I have everything that suppose to.. Follow up questions/items - I was unable the memory setting your referring to. Unless its this advanced/amd CBS/Zen Common Options/RedirectForReturnDis? I can set to auto 1 and 2 - I've ran each for years.. but id rather just run unraid. - what are the xml tweaks your talking about. -i also have a unknown device with the hardware id of APCI\VEN_APP&DEV_0005. (I've tried search this online and alot of people didn't have a answer. I tried searching on the virIO disk) Thanks again.. Edited July 8, 2020 by commotion Quote
Jerky_san Posted July 8, 2020 Posted July 8, 2020 (edited) Quote 45 minutes ago, commotion said: Wow thanks so much to the both of you. This is running 2000% better. 14 seconds boot time now... amazing Now that I have the bios settings correct in going to go back through the tweaks to make sure I have everything that suppose to.. Follow up questions/items - I was unable the memory setting your referring to. Unless its this advanced/amd CBS/Zen Common Options/RedirectForReturnDis? I can set to auto 1 and 2 - I've ran each for years.. but id rather just run unraid. - what are the xml tweaks your talking about. -i also have a unknown device with the hardware id of APCI\VEN_APP&DEV_0005. (I've tried search this online and alot of people didn't have a answer. I tried searching on the virIO disk) Thanks again.. Advanced -> CBS -> DF there is a Memory model item that has choices like auto | distribute | channel. Channel should be selected. Looking at the numa stuff but I'll give you some updated stuff tmrw. Edited July 8, 2020 by Jerky_san Quote
Jerky_san Posted July 8, 2020 Posted July 8, 2020 @commotion Below you'll notice a few things <emulator pin <numatune> <- right below emulator pin <numa> Honestly I'd suggest just copy/pasting what I have for the vcpu allocation/emulatorpin/numatune part. Then on <numa> adjust it for how much ram you have in each "side" of the double memory. I had 128gb in this host so I had 16gb from each channel allocated. You CURRENTLY have 16gb allocated total so just take my numbers and divide them by 2 and boom done on that part. Under <features> are also some nifty things to use. DO NOT copy EVERYTHING such as <os> cause that will break your machine. Anyways the reason you probably want to copy most of it is because my config spans two numa nodes that have memory controllers thus you'll get the fastest you can get(unless you want to allocate 4 cpus from each numa). You'll not get a huge performance boost from doing that though unless it is VERY SPECIFIC WORK LOADS!!! Quote -i also have a unknown device with the hardware id of APCI\VEN_APP&DEV_0005. (I've tried search this online and alot of people didn't have a answer. I tried searching on the virIO disk) Where is this device? Inside the VM? Install the agent from the virtio iso if so. Quote I've ran each for years.. but id rather just run unraid. Consider trying it if you don't get the performance your looking for. The only problem I have right now is it doesn't properly do the ram as much as I'd like but I don't have to "allocate" cpus or anything. Just "works" Lastly just to ask.. is there a reason you choose the 2990wx over say a 3950x? <- this is just my curiosity and doesn't require any explanation if you don't want to. <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='36'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='37'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='38'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='39'/> <vcpupin vcpu='8' cpuset='8'/> <vcpupin vcpu='9' cpuset='40'/> <vcpupin vcpu='10' cpuset='9'/> <vcpupin vcpu='11' cpuset='41'/> <vcpupin vcpu='12' cpuset='10'/> <vcpupin vcpu='13' cpuset='42'/> <vcpupin vcpu='14' cpuset='11'/> <vcpupin vcpu='15' cpuset='43'/> <emulatorpin cpuset='4-11,36-43'/> </cputune> <numatune> <memory mode='strict' nodeset='0,2'/> <memnode cellid='0' mode='strict' nodeset='0'/> <memnode cellid='1' mode='strict' nodeset='2'/> </numatune> <os> <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/99642e81-2f13-a916-682c-90191636d75f_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <hyperv> <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> <reset state='on'/> <vendor_id state='on' value='KVM Hv'/> <frequencies state='on'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> <numa> <cell id='0' cpus='0-7' memory='16777216' unit='KiB'/> <cell id='1' cpus='8-15' memory='16777216' unit='KiB'/> </numa> </cpu> Quote
commotion Posted July 10, 2020 Author Posted July 10, 2020 On 7/8/2020 at 9:08 AM, Jerky_san said: Where is this device? Inside the VM? Install the agent from the virtio iso if so. It is located in a VM and ive already installed the agent off of the virtio disk On 7/8/2020 at 9:08 AM, Jerky_san said: Consider trying it if you don't get the performance your looking for. The only problem I have right now is it doesn't properly do the ram as much as I'd like but I don't have to "allocate" cpus or anything. Just "works" maybe we will see this is running alot better now that ive used your xml file items.. Thanks so much for that. What program do you use to bench mark within the VM to see what improvements are working ect. On 7/8/2020 at 9:08 AM, Jerky_san said: Lastly just to ask.. is there a reason you choose the 2990wx over say a 3950x? <- this is just my curiosity and doesn't require any explanation if you don't want to. I wanted lots of cores to play with. But it was shipped to us by accident at work and it sat in our server room for a year. so i based the entire build on it. since it was free... Other notes. I have noticed when I'm in the unraid interface moving between areas in unraid. sometimes the webpage will go unresponsive. I'm unable to ping it on the network. I have to power cycle and when it comes back up i have to manually start the VM manager. This started after the bios changes. so i think my freezing is back. Anyway to prove that with logs? I think this has happened maybe once a day? Thanks So much! Quote
Jerky_san Posted July 10, 2020 Posted July 10, 2020 It is located in a VM and ive already installed the agent off of the virtio disk. Hmm I always just mount the virtio iso and go to the device manager and tell it to just scan the top level like E:\ and click the "include subfolders" and let it find whatever it wants. maybe we will see this is running alot better now that ive used your xml file items.. Thanks so much for that. What program do you use to bench mark within the VM to see what improvements are working ect. Welcome o-o glad it works for you. Honestly I mostly use CPUZ since it's fast/easy. I also use AIDA64 to get memory times. I have noticed when I'm in the unraid interface moving between areas in unraid. sometimes the webpage will go unresponsive. I'm unable to ping it on the network. I have to power cycle and when it comes back up i have to manually start the VM manager. This started after the bios changes. so i think my freezing is back. Anyway to prove that with logs? I think this has happened maybe once a day? Hmm now this is interesting. I know I used to see this when the USB controller I was passing had an issue. It would affect other USB controllers that weren't passed. You can usually see it in the syslog about the USB thumbdrive becoming dismounted. Were you able to find the power supply thing I had talked about? I believe on my old zenith it was in the AMD CBS menu. If the thumbdrive disconnects it sadly won't write anything to the local logs but if you have a syslog you can get it out of there. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.