September 27, 20178 yr Hi, I have a problem with GPU passthrough. I am using Unraid 6.3.5 (trial version). I have only one GPU - RX 550. I am trying to pass it through to a VM with Ubuntu, but no luck so far. I do not have additional GPU at home to test. I have tried 1st and 2nd PCIe slot (with and without ACS patch). I am on newest BIOS 0902 (ASUS X370 Pro with Ryzen 1600 and 16GB RAM, RX 550). GPU displays Unraid text console on monitor when I turn on PC. When I create VM with VNC, I can see it is booting from ISO, but when I choose RX 550 gpu and start VM, nothing happens - I can still see login prompt on the monitor. Any ideas for to pass through only one GPU in my case? Info: Model: Custom M/B: ASUSTeK COMPUTER INC. - PRIME X370-PRO CPU: AMD Ryzen 5 1600 Six-Core @ 3200 HVM: Enabled IOMMU: Enabled Cache: 576 kB, 3072 kB, 16384 kB Memory: 16 GB (max. installable capacity 64 GB) Network: bond0: fault-tolerance (active-backup), mtu 1500 eth0: 100 Mb/s, full duplex, mtu 1500 Kernel: Linux 4.9.30-unRAID x86_64 IOMMU: IOMMU group 0 [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 1 [1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453 IOMMU group 2 [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 3 [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 4 [1022:1453] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453 IOMMU group 5 [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 6 [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454 [1022:145a] 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a [1022:1456] 29:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456 [1022:145c] 29:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c IOMMU group 7 [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454 [1022:1455] 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455 [1022:7901] 2a:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) [1022:1457] 2a:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457 IOMMU group 8 [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 9 [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460 [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461 [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462 [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463 [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464 [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465 [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466 [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467 IOMMU group 10 [1022:43b9] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02) [1022:43b5] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02) [1022:43b0] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02) [1022:43b4] 1d:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1022:43b4] 1d:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1022:43b4] 1d:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1022:43b4] 1d:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1022:43b4] 1d:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1022:43b4] 1d:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) [1b21:1343] 25:00.0 USB controller: ASMedia Technology Inc. Device 1343 [8086:1539] 26:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 11 [1002:699f] 28:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Lexa PRO [Radeon RX 550] (rev c7) [1002:aae0] 28:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aae0 Edited October 8, 20178 yr by gelmi
October 8, 20178 yr Linux kernel 4.9 doesn't have full support for Ryzen. Try the latest unRAID 6.4.0-rc instead.
November 23, 20178 yr @gelmi How were you able to solve it? I use a Ryzen and the Asus X370 Pro as well. The only difference is my GPU: NVidia GTX 1080Ti When I start the VM I get a black screen on my primary GPU (unRaif 6.4.0-rc). Would appreciate if you could let me know how you solved it. Edited November 23, 20178 yr by Kenishi
November 23, 20178 yr Wow, that was a fast answer, thanks a lot. Really appreciate it! I'm trying to use Windows 10. Tried installing Ubuntu and opensuse as well, but run into the same issue. Did it work for you without any special changes in the settings, syslinux configuration or xml of the VM?
November 24, 20178 yr Author In the VM I set: * CPU Mode: Host Passthrough * Machine: i 440fx-2.9 * BIOS: SeaBIOS * Hyper-V: Yes * Graphics card and Sound Card: Choose both NVIDIA devices That should be it. Check this and post if it works or not.
November 24, 20178 yr Thanks a lot for your answer. The differences I had was * BIOS was set to: OVMF * Machine was set to: i440fx-2.7 I created a new machine with your settings but got the same problem. As soon as I start the VM, my screen goes black and nothing come up. You only have one GPU, right? Do you also see the Linux terminal on the screen and when you boot up the VM, the same screen goes black and then shows the VM content after that? I wonder why it's working for you with almost the same setup as I have. Something must be different on my side which prevents the graphics card to function correctly. When I connect to the Windows VM using a remote connection, I see my graphics card in the device manager but with a warning sign. "The device doesn't work properly." So it somehow recognise it, but it doesn't work as it should. Edited November 24, 20178 yr by Kenishi
November 24, 20178 yr Author I have 2 GPUs: first PCIe for RX560 and second PCIe for GTS 450. I can use either of them for Windows and Ubuntu VM. Try to reboot PC and start VM one more time. Maybe your GPU suffer from reset bug, so it can be initialized only once in VM after boot. If this does not work, share some more infrormation about your configuration: BIOS version, which PCIe slot do you use, exact RC version. Did you try to dump GPU ROM and feed it with VM configuration?
November 25, 20178 yr On my Asus Prime x370 Pro, I have updated BIOS to the latest 3203, and in this BIOS I enable SVM in order to allow Virtual Machines to run. In the Unraid syslinux.cfg I specify two options for my Ryzen 1600x: rcu_nocbs=0-11 processor.max_cstate=1. With the latest beta of Unraid 6.4, it runs with my Nvidia 1050Ti as passthrough for any OS. For Windows 10, I use OVMF and No to Hyper-V.
November 25, 20178 yr @pederm: Thanks a lot for sharing your setup. Really appreciate it and will try updating my BIOS and the settings right now. Some questions while I'm trying it out: - Do you have a second GPU? - Is the Nvidia 1050Ti your primary GPU? - If so, does your screen with the terminal on it goes black and then shows the Windows 10 VM? @gelmi: My PC specs: - Mainboard: Asus Prime x370 PRO (BIOS Version 0902 <= most stable on my system) - CPU: Ryzen 7 1700X - GPU 1: NVIDIA GeForce GTX 1080 TI (PCIE 1) <= Two monitors connected - GPU 2: NVIDIA GeForce GTX 1080 TI (PCIe 2) <= One monitor connected - RAM: Corsair Vengeance LPX 64GB (4 x 16GB) DDR4-3000 Yes, I already did dump my GPU ROM, using the commands on unRaid as well as in Windows using CPU-Z. Wrote it into the XML file. No luck yet. My second graphics card on PCIe 2 works. But the primary one, on PCIe 1, won't work at all. I see the terminal on my screen. As soon as I start the VM, the screen with the terminal goes black and that's it... Really appreciate your help and all your hints!
November 25, 20178 yr I PCIEX16_1 is an old AMD Redwood graphics card, which is exclusively used by Unraid. In the PCIEX_2 slot is the Nvidia 1050Ti located. A dedicated mouse and keyboard is also used exclusively by Unraid.
November 25, 20178 yr @pederm That's why it's working for you I think. You're not using your primary GPU for the VM. And that's exactly what I try to achieve - using my primary GPU for the VM. Do you know if it's possible to use three GPUs with the X370-PRO? Edited November 25, 20178 yr by Kenishi
November 25, 20178 yr Author OK, that is strange. So, let me get this straight: + You have two identical GPUs, first in slot 1 and second in slot? + The GPU in slot 1 does suffer from black screen problem, but the second GPU (identical NVIDIA) in second slot works? + Also, when you switch the cards, the card from slot 2, whey you put it into slow 1 does not work, right? If that is correct, maybe try to put card from slot 1 to slot 3 (only 8x) just for testing, so the cards are in slots 2 and 3. Check which of them will not work with VM. Maybe it is not the issue of a PCIe slot 1, but rather that the card cannot be passed to VM after it was initialized for host PC unraid console? Do you run Unraid UEFI or EFI boot?
November 25, 20178 yr Just now, gelmi said: + You have two identical GPUs, first in slot 1 and second in slot? 2 You're exactly right. Both are exactly the same GPU model. Just now, gelmi said: + The GPU in slot 1 does suffer from black screen problem, but the second GPU (identical NVIDIA) in second slot works? + Also, when you switch the cards, the card from slot 2, whey you put it into slow 1 does not work, right? Yes, I tried switching them already. The primary GPU (which is in PCIe 1) doesn't work. Doesn't matter which of the two is in there. The one in the second slot works fine. Just now, gelmi said: If that is correct, maybe try to put card from slot 1 to slot 3 (only 8x) just for testing, [...] Maybe it is not the issue of a PCIe slot 1, but rather that the card cannot be passed to VM after it was initialized for host PC unraid console? 2 Sadly I can't use slot 3 because there's not enough space for the GPU. There are cables right underneath of the slot and the GPU is too big. That's what I think as well. The card gets initialized by unraid console and then can't be passed to the VM. I wonder if it's possible to run unraid without it initializing the primary GPU. Just now, gelmi said: Do you run Unraid UEFI or EFI boot? UEFI is disabled (by default) on the bios. I looked into the settings and it's still deactivated - so no UEFI => I think UEFI is still used even if disabled. Tbh, I'm not quite sure about this. Edited November 25, 20178 yr by Kenishi
November 25, 20178 yr Author OK, so if both cards work fine in slot 2, so the problem is with sharing/passing through GPU that is boot GPU for Unraid in slot 1. Have you checked either of these solutions? https://arseniyshestakov.com/2016/03/31/how-to-pass-gpu-to-vm-and-back-without-x-restart/ <------------ the binding/unbinding solution What is your current Unraid RC version?
November 25, 20178 yr I did already try the second solution - it's how I dumped my GPU ROM and edited the XML manually. About the binding/unbinding solution - well, I tried it but failed. echo "0000:0a:00.0" > /sys/bus/pci/devices/0000:0a:00.0/driver/unbind shows "No such file or directory". Also tried it on "/sys/bus/pci/drivers" but there it says "No such device". (0a:00.0 is my primary GPU on PCIe slot 1) I slowly begin to give up. I'm on latest unRAID 6.4.0-rc14. Set it up today from scratch. Edited November 25, 20178 yr by Kenishi
November 25, 20178 yr Author Have you tried to add two GPU NVIDIA cards to one VM at the same time and see how it shows in Device Manager? Or maybe VNC (as primary GPU) and GPU from 1st slot (as a secondary).
November 25, 20178 yr You are a genius! You my dear friend are my hero. I owe you so much! Send me your mail address by PM and I'll send you an Amazon voucher or anything else you use to buy online. Starting only one VM with both GPUs worked. Then I stopped the VM, assigned one gpu to another VM and started again. And BAM it worked! I'll write a start script which does that with an "empty VM" automatically, so I don't have to do it myself.
November 25, 20178 yr Author Glad I could help. Script is a good way to go. The problem is with (re)initialization of NVIDIA cards. NVIDIA does not like when consumer GPUs are used for virtualization. They would like you to use their enterprise cards. I still have to restart or sleep/wake up Unraid system in order to reboot linux VM that uses my old GTS 450.
November 25, 20178 yr Yes, had the same behaviour with my second card - which always worked. Quickly going to sleep mode and booting up again solves it. So happy it's finally working. Thank you so much!
February 26, 20197 yr Hi, sorry I'm new to all of this VM stuff, I have and specific question here, so you can pass through all the GPUs available on your machine to VMs and left the unRAID server without any GPU for itself? thanks in advance. Edited February 26, 20197 yr by CarlosCo typo
March 4, 20197 yr Author Yes, you can run Unraid headless. I have 2 external GPUs and sometimes one is for 1st VM, the other is for 2nd VM, both VMs are running and Unraid still works. The only thing is that after I shut down VM I cannot reassign my primary GPU for Unraid console, but this is due to my GPU. Some GPUs can be switched back via script. I need to restart my tower when I want to use the GPU by Unraid console.
Archived
This topic is now archived and is closed to further replies.