IamSpartacus Posted March 18, 2019 Share Posted March 18, 2019 I'm looking into (and will soon be testing) converting my main everyday PC (Threadripper 1950x + ASRock Taichi x399) into an Unraid server + Windows VM with just about all devices passed through to it to be used for my daily driver. Thus I want to be able to use all my current USB ports (I need almost all of them for things like VR, etc.) for the Windows VM. Thus I'll need to boot my Unraid USB off a non-passed through USB controller. Can I simply add a PCIe USB controller, and connect my Unraid USB that way for booting? Any issues with this? Quote Link to comment
Warrentheo Posted March 18, 2019 Share Posted March 18, 2019 It can be possible, but the short answer is "don't"... Get a separate card instead... Some motherboards have several separate "cards" mine has 3, one 3.1 Motherboard header, one 3.1 on the rear, and one 3.0/2.0 running most of the remaining ports on the back... One controller needs to remain to be able to run the UnRaid USB thumb drive, and on my motherboard the remaining 2 other controllers have issues with IOMMU groups and don't passthrough very well... I can still pass the individual devices, but that basically requires that they be permanently connected and never change... Ok for some devices, but not all... The best way to get them to act like a normal controller is to pass a controller of its own to the VM... You may get lucky and your motherboard may have more than one controller, and one of those controllers my be passable to the VM... Quote Link to comment
IamSpartacus Posted March 18, 2019 Author Share Posted March 18, 2019 25 minutes ago, Warrentheo said: It can be possible, but the short answer is "don't"... Get a separate card instead... Some motherboards have several separate "cards" mine has 3, one 3.1 Motherboard header, one 3.1 on the rear, and one 3.0/2.0 running most of the remaining ports on the back... One controller needs to remain to be able to run the UnRaid USB thumb drive, and on my motherboard the remaining 2 other controllers have issues with IOMMU groups and don't passthrough very well... I can still pass the individual devices, but that basically requires that they be permanently connected and never change... Ok for some devices, but not all... The best way to get them to act like a normal controller is to pass a controller of its own to the VM... You may get lucky and your motherboard may have more than one controller, and one of those controllers my be passable to the VM... So you're saying get a separate controller and only use that one to pass through to the VM? I'd really love to be able to use my front USB ports and thus basically treat the system like a baremetal PC even though it's running my "PC" as a VM. Quote Link to comment
JonathanM Posted March 18, 2019 Share Posted March 18, 2019 (edited) 9 minutes ago, IamSpartacus said: I'd really love to be able to use my front USB ports and thus basically treat the system like a baremetal PC even though it's running my "PC" as a VM. So get a card with the appropriate header and plug your front ports into that instead of the motherboard. Quick google gets this, no clue if it's pass through friendly. http://amzn.com/B01G85LFEK Product search terms pcie usb 20 pin Edited March 18, 2019 by jonathanm Added search phrase since it may not be obvious Quote Link to comment
IamSpartacus Posted March 18, 2019 Author Share Posted March 18, 2019 (edited) 11 minutes ago, jonathanm said: So get a card with the appropriate header and plug your front ports into that instead of the motherboard. Quick google gets this, no clue if it's pass through friendly. http://amzn.com/B01G85LFEK I would need one with a USB 3.0 & 3.1 type C header. This is my front panel. I guess this would work if I could find a PCIe card with dual USB 3.0 headers and at least external USB ports. http://www.lian-li.com/product/pw-inc1tr/ Edited March 18, 2019 by IamSpartacus Quote Link to comment
Warrentheo Posted March 18, 2019 Share Posted March 18, 2019 Some of those use the ones that are part of the motherboard, which usually has the same issues as using the ports that come with the motherboard... This is the one that I use: Inateck KT5001 PCI-E to USB 3.0 5-Port PCI Express Card and 15-Pin Power Connector, Red https://www.amazon.com/gp/product/B00FPIMJEW/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1 Works very well, and supplies power which is something you need to consider, the power coming from your motherboard is not going to be able to light up all the stuff you plug in, so you will either need to get a powered USB hub and plug that in, or get one that taps power off from the power supply directly... This one also has a second internal header, so you can connect up some of your front panel case ports to it to pass them also (on mine this is 2 ports on the front, 5 in the back, you will not find a card that does better than 7) Quote Link to comment
IamSpartacus Posted March 19, 2019 Author Share Posted March 19, 2019 13 hours ago, Warrentheo said: Some of those use the ones that are part of the motherboard, which usually has the same issues as using the ports that come with the motherboard... This is the one that I use: Inateck KT5001 PCI-E to USB 3.0 5-Port PCI Express Card and 15-Pin Power Connector, Red https://www.amazon.com/gp/product/B00FPIMJEW/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1 Works very well, and supplies power which is something you need to consider, the power coming from your motherboard is not going to be able to light up all the stuff you plug in, so you will either need to get a powered USB hub and plug that in, or get one that taps power off from the power supply directly... This one also has a second internal header, so you can connect up some of your front panel case ports to it to pass them also (on mine this is 2 ports on the front, 5 in the back, you will not find a card that does better than 7) Thanks for the link, that looks like a solid option I will definitely consider. I'm wondering if @bastl can comment since he has the same board as me and is passing through hardware to both Linux and Windows VMs. Have you successfully passed through some/all USB controllers on the ASRock x399 Taichi board to VMs? Quote Link to comment
bastl Posted March 19, 2019 Share Posted March 19, 2019 @IamSpartacus I'am using the ASRock X399 Fatal1ty Gaming and in the following you can see the groupings of the devices to maybe get an idea what you can passthrough and what not. This should be similar looking on the Taichi, except of the 10gig nic missing for you. As you can see on default on group 13 all the networking devices on that board and the USB 3.1 controller is grouped together. No big deal for me because I don't pass any of them through a VM. I use the "Libvirt Hotplug USB" if I wanna have access to an USB device like a phone or an external HDD inside a VM. Plugin the device, go to the VMs tab, underneath the VMs list you can choose the VM and the device you wanna "plugin", select attach, done. Works fine for me. In case you want to split the groupings further, the ACS Override Patch in the VM manager settings splits all the networking devices and the USB 3.1 controller in it's own group in my case. I guess passthrough should work than. Never tried it, still on my list. The easiest solution is an pcie addin card mentioned earlier for each VM. Explanation of my setup for 2 VMs I use. I have 2 mice and one keyboard to control both VMs. Each VM gets it's one mouse and the keyboard is shared between them. It's an Corsair K63 Wireless which has an build in BT controller and can be connected via USB. It's connected to my main VM via USB (selected in the VM setup) and can be switched via hotkey to BT which than connects to a cheap BT dongle connected to the second VM (dongle selected in the VM setup). My monitor has 2 inputs, display port connected to the 1080ti (VM1 Win10) and HDMI connected to a 1050ti (VM2 Linux). Switching between the VMs is only pressing a hotkey on the keyboard to switch to bluetooth, press a button on the monitor to switch the input and grab the second mouse. Done. Accessing a randomly plugged in USB device like a external HDD or a stick or a phone as mentioned earlier the solution with the plugin works fine for me. Other option is to set the device in UD to automount and autoshare and you will find it as a networkshare. PCI Devices and IOMMU Groups IOMMU group 0: [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 1: [1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 2: [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 3: [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 4: [1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 5: [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 6: [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 7: [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 8: [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 9: [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 10: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 11: [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 12: [1022:1460] 00:19.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1461] 00:19.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1462] 00:19.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1463] 00:19.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1464] 00:19.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1465] 00:19.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1466] 00:19.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1467] 00:19.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 IOMMU group 13: [1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02) [1022:43b6] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02) [1022:43b1] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02) [1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1022:43b4] 02:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) [1d6a:d107] 03:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02) [8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) [8086:24fb] 05:00.0 Network controller: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] (rev 10) [8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 14: [10de:1c82] 08:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1) [10de:0fb9] 08:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1) IOMMU group 15: [1022:145a] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 16: [1022:1456] 09:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 17: [1022:145c] 09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller IOMMU group 18: [1022:1455] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 19: [1022:7901] 0a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 20: [1022:1457] 0a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller IOMMU group 21: [1022:1452] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 22: [1022:1453] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 23: [1022:1453] 40:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 24: [1022:1452] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 25: [1022:1452] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 26: [1022:1453] 40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge IOMMU group 27: [1022:1452] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 28: [1022:1452] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 29: [1022:1454] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 30: [1022:1452] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge IOMMU group 31: [1022:1454] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B IOMMU group 32: [144d:a804] 41:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 IOMMU group 33: [144d:a804] 42:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 IOMMU group 34: [10de:1b06] 43:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) [10de:10ef] 43:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1) IOMMU group 35: [1022:145a] 44:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function IOMMU group 36: [1022:1456] 44:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor IOMMU group 37: [1022:145c] 44:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller IOMMU group 38: [1022:1455] 45:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function IOMMU group 39: [1022:7901] 45:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) Quote Link to comment
IamSpartacus Posted March 20, 2019 Author Share Posted March 20, 2019 (edited) @bastl How does one determine which device is in witch PCIe port? I have two identical 1080Ti's. Also, how did you determine which NVMe controller to pass through? I like you show 2 NVMe controllers. Attached is the output I got from lstopo, Edited March 20, 2019 by IamSpartacus Quote Link to comment
bastl Posted March 20, 2019 Share Posted March 20, 2019 (edited) @IamSpartacus I'am not exactly sure how you better can check which card is connected in which PCIe port in your case. Unplug one card and check LSTOPO again and you will see which port the one card is connected to. I have 2 different cards. "10de:1b06" should be a 1080ti. The controllers of my 2 NVMEs are the same, right. One is a 960 EVO (cache nvme1n1) and the other is a 960 PRO (passthrough nvme0n1). The main tab on the dashboard shows me the cache device as nvme1n1. That's how i figured it out Also mentioned you're running in UMA mode. You should find an option to change the memory interleaving config from "auto" to "channel" somewhere under "Advanced/CBS/DF something" in your BIOS. Edited March 20, 2019 by bastl Quote Link to comment
IamSpartacus Posted March 20, 2019 Author Share Posted March 20, 2019 4 hours ago, bastl said: @IamSpartacus I'am not exactly sure how you better can check which card is connected in which PCIe port in your case. Unplug one card and check LSTOPO again and you will see which port the one card is connected to. I have 2 different cards. "10de:1b06" should be a 1080ti. The controllers of my 2 NVMEs are the same, right. One is a 960 EVO (cache nvme1n1) and the other is a 960 PRO (passthrough nvme0n1). The main tab on the dashboard shows me the cache device as nvme1n1. That's how i figured it out Also mentioned you're running in UMA mode. You should find an option to change the memory interleaving config from "auto" to "channel" somewhere under "Advanced/CBS/DF something" in your BIOS. The NVMe's I got figured out just like you said, based on the UD plugin. As for the GPU's I can't take them out of their slots because it's all together in a loop so I'd have to pull my loop apart. I guess I'll have to do trial and error with those. Thanks for the time on the UMA mode. For this project I set my BIOS settings back to default to upgrade the BIOS to 2.30 and then the latest (I was on 2.00). Once I get all my VM pass through config worked out I'll start testing OC's again. Previously I was at 3.8Ghz with 1.24vcore which I'm fine with if that's the best my chip can do. Quote Link to comment
bastl Posted March 20, 2019 Share Posted March 20, 2019 Wait what??? ONLY 3.8GHz on water? I easy get my chip to 4GHz on air even 4.1 is possible without really throttling. I more and more think I got a golden chip. 4GHz @1.275 😂 Quote Link to comment
IamSpartacus Posted March 20, 2019 Author Share Posted March 20, 2019 5 minutes ago, bastl said: Wait what??? ONLY 3.8GHz on water? I easy get my chip to 4GHz on air even 4.1 is possible without really throttling. I more and more think I got a golden chip. 4GHz @1.275 😂 What are your temps like at 4GHz @1.275 and what stress are you doing while seeing those temps? From what I've seen, that is quite a great chip you've got there. That's a very low vcore to be able to hit 4GHz. Quote Link to comment
bastl Posted March 20, 2019 Share Posted March 20, 2019 With all dockers and 2 VMs running, only browsing the web so kinda idling it sits around 38-40 °C. Some short spikes to 48-50, but thats it. Having a docker running backing up all my VMs and compressing them it sits around 50C and while gaming depends on the game. It can go up to 60C. Keep in mind this all are tdie temps. The Ryzen chips all have an offset on top of it showing as tctl. For the 1950x the offset is 27C. Some tools reporting the tctl temps only and people freak out seeing this high temperatures. The 1950x starts to throttle at 68C tdie. So I'am kinda maxed out on cooling capacity when gaming. With a stress test like Prime95 inside a VM with half the cores I can get it up to 70C still stable and without really throttling. Maybe because not the complete chip is utilised inside the VM and the throttling also depends on the power draw besides high temps. I had my old AIO Liqtech TR4 360 failling last year because of an manufacturing issue, where corrosion gunked up the rad. I saw idle temps between 40-50 and a small load and it jumped up to 70C and instant throttled down below 2GHz on all 16cores on a full windows install. So if you think your OC is limited by the temperatures you're seeing you might check which one you're reading @IamSpartacus Quote Link to comment
IamSpartacus Posted March 20, 2019 Author Share Posted March 20, 2019 (edited) 18 minutes ago, bastl said: With all dockers and 2 VMs running, only browsing the web so kinda idling it sits around 38-40 °C. Some short spikes to 48-50, but thats it. Having a docker running backing up all my VMs and compressing them it sits around 50C and while gaming depends on the game. It can go up to 60C. Keep in mind this all are tdie temps. The Ryzen chips all have an offset on top of it showing as tctl. For the 1950x the offset is 27C. Some tools reporting the tctl temps only and people freak out seeing this high temperatures. The 1950x starts to throttle at 68C tdie. So I'am kinda maxed out on cooling capacity when gaming. With a stress test like Prime95 inside a VM with half the cores I can get it up to 70C still stable and without really throttling. Maybe because not the complete chip is utilised inside the VM and the throttling also depends on the power draw besides high temps. I had my old AIO Liqtech TR4 360 failling last year because of an manufacturing issue, where corrosion gunked up the rad. I saw idle temps between 40-50 and a small load and it jumped up to 70C and instant throttled down below 2GHz on all 16cores on a full windows install. So if you think your OC is limited by the temperatures you're seeing you might check which one you're reading @IamSpartacus Temps weren't really my issue in testing at 3.8GHz. I was idling around 27C and loading around 58-60C (I use HWMonitor by CPUID). But I had to bump the vcore up quite a bit in order to be stable in Prime95 using all 16 cores. I didn't do a ton of testing because i got preoccupied with other things like having my first child (this was a few months ago) so I will revisit this once I get all my VM stuff setup and working as I want it to. Edited March 20, 2019 by IamSpartacus Quote Link to comment
bastl Posted March 20, 2019 Share Posted March 20, 2019 congrats 👍 At least you're on the safe side with that 3.8 for stability. Quote Link to comment
Warrentheo Posted March 20, 2019 Share Posted March 20, 2019 6 hours ago, IamSpartacus said: Temps weren't really my issue in testing at 3.8GHz. I was idling around 27C and loading around 58-60C (I use HWMonitor by CPUID). But I had to bump the vcore up quite a bit in order to be stable in Prime95 using all 16 cores. I didn't do a ton of testing because i got preoccupied with other things like having my first child (this was a few months ago) so I will revisit this once I get all my VM stuff setup and working as I want it to. Prime95 uses a part of your CPU called AVX or AVX2... This puts so much stress on the CPU that most motherboards have a setting in their BIOS to UnderClock the CPU when it detects an AVX workload... Prime95 is an unrealistic CPU stress test for this reason... By default my motherboard BIOS drops 200Mhz off the clocks, and I have my overclock set to drop 300Mhz off the clock intentionally just for testing with Prime95... On my motherboard this is called "AVX instruction core negative offset [Auto] [1-31]"... Something like 3DMark is a much better CPU test for this reason... Overclock for normal load stability first, then downclock from there for AVX load stability... Quote Link to comment
IamSpartacus Posted March 21, 2019 Author Share Posted March 21, 2019 (edited) 16 hours ago, Warrentheo said: Prime95 uses a part of your CPU called AVX or AVX2... This puts so much stress on the CPU that most motherboards have a setting in their BIOS to UnderClock the CPU when it detects an AVX workload... Prime95 is an unrealistic CPU stress test for this reason... By default my motherboard BIOS drops 200Mhz off the clocks, and I have my overclock set to drop 300Mhz off the clock intentionally just for testing with Prime95... On my motherboard this is called "AVX instruction core negative offset [Auto] [1-31]"... Something like 3DMark is a much better CPU test for this reason... Overclock for normal load stability first, then downclock from there for AVX load stability... Good point. I always did think Prime95 was unreleastic. @bastl Here's my new lstopo output (you'll notice one NVMe gone as it's passed through) to the VM. I'm not having great luck with passing through my 1080Ti's though. When I try to pass through the GPU in slot 1 I get Error Code 43. When I try to passthrough the GPU in slot 3, it works but the screen goes black a lot. Sometimes every few minutes, sometimes every few seconds. I'm using the latest Nvidia drivers. Edited March 21, 2019 by IamSpartacus Quote Link to comment
bastl Posted March 21, 2019 Share Posted March 21, 2019 @IamSpartacus To passthrough a card in slot 1 I have to specify the VBIOS for this card in the XML otherwise I have the same issue. 3 minutes ago, IamSpartacus said: the screen goes black a lot What did you mean by that? Is it turning of your monitor or does it flicker? Did you applied the MS_fix tool inside the VM? Quote Link to comment
IamSpartacus Posted March 21, 2019 Author Share Posted March 21, 2019 1 minute ago, bastl said: @IamSpartacus To passthrough a card in slot 1 I have to specify the VBIOS for this card in the XML otherwise I have the same issue. What did you mean by that? Is it turning of your monitor or does it flicker? Did you applied the MS_fix tool inside the VM? The monitor is not turning off power wise but the screen flickers completely black. What MS_fix are you talking about? Quote Link to comment
bastl Posted March 21, 2019 Share Posted March 21, 2019 (edited) 9 minutes ago, IamSpartacus said: What MS_fix are you talking about? https://wiki.unraid.net/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support http://forums.guru3d.com/showthread.php?t=378044 Usually it's an sound related issue passing through the audio portion of a GPU to a VM causing crackling sound. This tool helps fixing this issue. Has to be reapplied after a driver update and sometimes also after an OS update. I think I for some people it also fixes flickering video output. Edit: Btw it's advised to always pass through the audio part of the GPU Edited March 21, 2019 by bastl Quote Link to comment
IamSpartacus Posted March 21, 2019 Author Share Posted March 21, 2019 (edited) 14 minutes ago, bastl said: https://wiki.unraid.net/UnRAID_6/VM_Guest_Support#Enable_MSI_for_Interrupts_to_Fix_HDMI_Audio_Support http://forums.guru3d.com/showthread.php?t=378044 Usually it's an sound related issue passing through the audio portion of a GPU to a VM causing crackling sound. This tool helps fixing this issue. Has to be reapplied after a driver update and sometimes also after an OS update. I think I for some people it also fixes flickering video output. Edit: Btw it's advised to always pass through the audio part of the GPU I did pass through the audio part of the GPU but does this fix apply if I'm only outputting via DP? EDIT: Ok i applied that fix, unfortunately I can't test it till I get home from work. I'm RDP'd into the VM right now but that is worthless for these testing purposes. Edited March 21, 2019 by IamSpartacus Quote Link to comment
bastl Posted March 21, 2019 Share Posted March 21, 2019 This fix changes the way the OS is comunicating with the card, so yes it will change something. My main card the 1080ti is connected via display port to my monitor and the audio portion is also passed through. I don't use the monitors sound output, but without I will have sound issues if I would use them. If you don't notice any sound issues leave that tweak by side for now but keep that in mind. Did you assigned cores from the node the card is attached to? Does it only flicker in games or also on desktop? Here as a example my main VM xml if you wanna play around a bit. notes: Unraid version 6.7.0_RC5 all cores on the second die are isolated emulatorpin and iothreadpin limited to the second die Q35 because of newly introduced PCIE-root-port-patch in RC5 (qemu part at the end needed) to have the OS and applications correctly notice the link speed of the slot the card is plugged in, otherwise Nvidia driver reports it as 1x speed only. For me feels smoother this way. not working for i440fx or on older Unraid versions numatune set to use only memory directly connected to the second node CPU model tweaks to get the right CPU cache reported to the VM except of the qemu part everything also works on i440fx type VMs for me VBIOS of the card dumped and modified, removed some Nvidia header (SpaceInvaderVideo) MSI_fix applied for GPU and audio devices CPU Scaling Governor set to "On Demand" (Tipps and Tweaks Plugin) some Windows settings tweaked, the usual stuff, power plan to high performance, a couple services disabled etc. <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>WIN10_NVME_UEFI_Q35</name> <uuid>de625f4a-3dfd-9751-a28a-68449d424909</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>14</vcpu> <iothreads>1</iothreads> <cputune> <vcpupin vcpu='0' cpuset='9'/> <vcpupin vcpu='1' cpuset='25'/> <vcpupin vcpu='2' cpuset='10'/> <vcpupin vcpu='3' cpuset='26'/> <vcpupin vcpu='4' cpuset='11'/> <vcpupin vcpu='5' cpuset='27'/> <vcpupin vcpu='6' cpuset='12'/> <vcpupin vcpu='7' cpuset='28'/> <vcpupin vcpu='8' cpuset='13'/> <vcpupin vcpu='9' cpuset='29'/> <vcpupin vcpu='10' cpuset='14'/> <vcpupin vcpu='11' cpuset='30'/> <vcpupin vcpu='12' cpuset='15'/> <vcpupin vcpu='13' cpuset='31'/> <emulatorpin cpuset='8,24'/> <iothreadpin iothread='1' cpuset='8,24'/> </cputune> <numatune> <memory mode='strict' nodeset='1'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/de625f4a-3dfd-9751-a28a-68449d424909_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> <reset state='on'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>EPYC</model> <topology sockets='1' cores='7' threads='2'/> <cache level='3' mode='emulate'/> <feature policy='require' name='topoext'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='disable' name='x2apic'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/> <backingStore/> <target dev='hdb' bus='sata'/> <readonly/> <alias name='sata0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='none'/> <source file='/mnt/user/isos/clover/spaces_win_clover.img'/> <backingStore/> <target dev='hdc' bus='sata'/> <boot order='1'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='threads' discard='unmap'/> <source dev='/dev/disk/by-id/ata-Samsung_SSD_850_EVO_1TB_S2RFNX0J606029L'/> <backingStore/> <target dev='hdd' bus='scsi'/> <alias name='scsi0-0-0-3'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <controller type='usb' index='0' model='nec-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='scsi' index='0' model='virtio-scsi'> <driver queues='4' iothread='1'/> <alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:65:2d:ab'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-WIN10_NVME_UEFI_Q35/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/Backup/vbios/Strix1080ti/AsusStrix1080TI_dump_edit.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x3'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc246'/> <address bus='5' device='2'/> </source> <alias name='hostdev4'/> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1b1c'/> <product id='0x1b50'/> <address bus='5' device='3'/> </source> <alias name='hostdev5'/> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> <qemu:commandline> <qemu:arg value='-global'/> <qemu:arg value='pcie-root-port.speed=8'/> <qemu:arg value='-global'/> <qemu:arg value='pcie-root-port.width=16'/> </qemu:commandline> </domain> I think you have a bit to play with now 😄 Quote Link to comment
IamSpartacus Posted March 21, 2019 Author Share Posted March 21, 2019 (edited) 9 minutes ago, bastl said: I think you have a bit to play with now 😄 I sure do, thanks :D. As for your question about the flickering, it happens just sitting at the desktop with nothing open on the taskbar. As for the cores, I really don't know. I'd have to make one card disappear to know which card is assigned to which core die. Would simply removing power (with the system off of course) remove the GPU from being detected? Edited March 21, 2019 by IamSpartacus Quote Link to comment
bastl Posted March 21, 2019 Share Posted March 21, 2019 2 minutes ago, IamSpartacus said: simply removing power Not actual sure if this would work. Some older cards are still recognized without extra power plugged in. Worst case, server won't boot. I bet the layout will be the same as for me. Slot1 is connected to node1 and slot3 to node0. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.