Donach

Hey all wanted to chime in with my experience... I had several issues over the years with GPU passthrough, eventually I lately settled on having working 3060Ti in 2nd slot (x8) PCI-E, which works flawlessly. However, my main goal was to allow two machines running at once, yet the first GPU, whatever I used, was having either reset bug, or Error 43... Now I've tested just today, and my VM resets and shutsdown-powers up flawlessly - using RX 6800. Here are all the settings used, hope I remember all: HW: R9-5900X AsRock Phantom Gaming 4S (X570 mobo) 64GB RAM (4 sticks) UEFI: Enabled 4G, Resize-Bar Booting from UEFI unRaid on USB stick Latest AsRock bios (as of 09/2023 or so). IOMMU, HVM etc on ofc unRaid: Version: 6.12.3 Enabled ACS Override = "both" VFIO allow unsafe interrupts: = "yes" Syslinux config: kernel /bzimage append video=efifb:off isolcpus=3-11,15-23 pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot Ofc make sure to bind EVERYTHING from the GPU to VFIO - RX 6800 has 4 devices on it: GPU, Sound, USB controller + USB serial bus (IDs 0d:00:0 -> 0d:00:3 for me...) VM config: Win10 - I actually didn't install "from scratch", I had an old image lying around with Win10 working from testing the RTX 3080 in 1st PCI-E slot - so I cannot say if this helped in any way... Q35v7.1 Bios: OVMF Hyper-V: YES USB: 3.0 (qemu XHCI) Make sure ALL FOUR devices are passed through to VM I did NOT do any manual edits, I wanna avoid them as much as possible so... To install the driver, I also used the VNC console, which I've now removed after installing Barrier to the VM. With this, I'm able to restart and/or shutdown and bring the VM back up, no issues. Haven't done performance testing yet, I'll compare the two VMs to see if there is any considerable difference (the 2nd VM has the 3060ti in 2nd PCIE slot).

@jammsen What you may be forced to do, is to passthrough whole PCI group to one VM. Although this allows you to pick the groups specifically, not to use one huge group. For example, Groups 31-33 may have to be passed to the same VM to make it working. However, I would still try to pass just the Audio (Group 33) and see what happens, try a few restarts etc.

@jammsen Sure thing. For the video playbacks, I usually had no issues after all set properly (this was actually one of my tests to see if YT videos play without stutter and video-audio are in sync). What I usually had to do for audio is to pass it through, any emulation never worked. This means I passed the onboard audio, and for other VMs, I was using the simple USB 2-jack audio and via USB Hotplug plugin was plugging it in (or just via VM settings). Then I had to go to audio settings and change the sample rate to at least 48 kHz, then there was no crackling in the audio on either of the cards.

@jammsen Well the groups are not that good yet. Try to enable both ACS Override in GUI, then go to the syslinux (or how is it called) and enable multifunction as well. Sorry to hear you had to replace MB, sometimes going for the cheapest option brings more issues that to put in few more bucks ;). As for the slow down, I think its really a disk issue. If you copy something over to the array, from which the VMs are running as well, it slows down like a hell. I had actually issues on my own, like when I plugged in a USB, the VM gets a micro-freeze (like 0.1-0.2 seconds), which is also pretty annoying (not that you would plug in something all the time, but still...) However, when I was copying, I do not remember having such hard time. I was using Samsung 960 EVO as my cache drive. I have encountered different issues though, so be aware of those - like passing through Vega 56 to VM is OK, however Vega has the hardware reset bug, so once you restart the VM, you cannot assign it properly again - no hacks like enabling/disabling worked for me, so I just had to revert to bad old Win 10 :/.

Did you use ACS Override? If not, do. If that is not enough, search my post back in the thread, where I explain how to setup the "multifunction" into unRaid startup. This helped me a lot to split the groups insanely (like really every device having its own group).

About the audio crack - I was experiencing this on any Audio device. I was passing through USB stick with 2jacks - simple fix though: In the Windows Sound configuration, changing the sound quality to higher (I think it was 48kHz - DVD) resolved the issue (as far as I can recall). Thanks for the reply - so just getting newest unRaid RC is enough from what I understood... I will definitely try it sometime soon!

Did you use any dirty-hack to make it working like bare-metal? I tried to setup double Win 10 VMs before the NPT fix was found, and it was horrible - had lags even at very basic games... Thanks for clarifying this.

What driver are you using for the wireless mouse? Tried to install the Microsoft Mouse and Keyboard center? If yes, try various version or uninstall it completely...

Hello, Have you tried various USB ports? I actually never had such issue, passing any USB device worked flawlessly. Be sure to also have the USB adapter installed for the VM and to have correct drivers installed.

Do you have ACS Override enabled? If not, do so. See my post (around 10-15th page), where I explain how to enable it for it to work perfectly.

Hello, I tried to do this npt=0 on unRAID, however I was not able to do so... How can you set this setting? Or more precisely, where? Many thanks.

Hello ColeBrodine, as Pauven said, look at my post on page 13 of this thread - my IOMMU grouping was totally off with the ACS Override enabled also, until I added the "multifunction" option. Since then, works like a charm - I'm able to passthrough anything to the VM, even since the CH6 has 3 PCI-Ex16 ports, I can run 3 VMs, each having it's own GPU. However, I am now struggling with some specific problems - I want to passthrough USB busses, so I can plug and unplug USB devices on the go. Another thing is that I have encountered issues running Europa Universalis IV - extreme game stuttering, solely on VMs. I did not see this with any other games yet.

Hello everybody! I have been also dreaming about making it possible to run 2 VMs on one machine. I can now finally say, I was successful with Ryzen 7 + unRAID! I am now able to run 2 or even 3 VMs with dedicated GPUs. My setup: Ryzen 7 1700 @stock (planning OC) ASUS ROG Crosshair Hero VI 16 GB Corsair 3200 MHz DDR4 Samsung 960 EVO 250GB (cache) Seagate Barracuda 3TB (planning to get one or two more) Radeon HD5770, R5 230 (guests); Nvidia GTS 450 (host, guest also possible) - I am using these lowend GPUs which I had lying around, once all tested, I'm going to get pair of RX480/580. Linux 4.9.19-unRAID x86_64 ACS Override Enabled (downstream,multifunction) - note that multifunction option was MANDATORY for my setup, otherwise the override was working only for top 2 PCI-Ex16 slots. Syslinux config: cat /boot/syslinux/syslinux.cfg default /syslinux/menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot label unRAID OS GUI Mode kernel /bzimage append pcie_acs_override=downstream initrd=/bzroot,/bzroot-gui label unRAID OS Safe Mode (no plugins, no GUI) kernel /bzimage append pcie_acs_override=downstream initrd=/bzroot unraidsafemode label unRAID OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest IOMMU Groups - these actually played out amazingly! I did not expect after lots of hate on the grouping, that it would separate so perfectly, see for yourself! IOMMU group 0 [1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 1 [1022:1453] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453 IOMMU group 2 [1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453 IOMMU group 3 [1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 4 [1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 5 [1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453 IOMMU group 6 [1022:1453] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453 IOMMU group 7 [1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 8 [1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 9 [1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454 IOMMU group 10 [1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452 IOMMU group 11 [1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454 IOMMU group 12 [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 13 [1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460 [1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461 [1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462 [1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463 [1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464 [1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465 [1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466 [1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467 IOMMU group 14 [144d:a804] 01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 IOMMU group 15 [1022:43b9] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02) IOMMU group 16 [1022:43b5] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02) IOMMU group 17 [1022:43b0] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02) IOMMU group 18 [1022:43b4] 1d:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) IOMMU group 19 [1022:43b4] 1d:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) IOMMU group 20 [1022:43b4] 1d:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) IOMMU group 21 [1022:43b4] 1d:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02) IOMMU group 22 [1b21:1343] 21:00.0 USB controller: ASMedia Technology Inc. Device 1343 IOMMU group 23 [8086:1539] 23:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 24 [10de:0dc4] 25:00.0 VGA compatible controller: NVIDIA Corporation GF106 [GeForce GTS 450] (rev a1) IOMMU group 25 [10de:0be9] 25:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1) IOMMU group 26 [1002:677b] 26:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos PRO [Radeon HD 7450] IOMMU group 27 [1002:aa98] 26:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM] IOMMU group 28 [1002:68b8] 27:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770] IOMMU group 29 [1002:aa58] 27:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] IOMMU group 30 [1022:145a] 28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a IOMMU group 31 [1022:1456] 28:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456 IOMMU group 32 [1022:145c] 28:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c IOMMU group 33 [1022:1455] 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455 IOMMU group 34 [1022:7901] 29:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 35 [1022:1457] 29:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457 VMs: Windows 10 8 CPUs (0-7 and 8-15) Machine i440fx-2.7 BIOS: OVMF Hyper-V is OFF Virtio drivers: virtio-win-0.1.126-2.iso keyboard and mouse passed through Audio from USB double jack adapter - I had to set sound quality to at least 48kHz, or there was crackling in the audio (no manufacturer driver available) or Audio passed through from MB ([1022:1457] 29:00.3) With this setup, I am running 2 GPUs with the Radeon cards passed through to VMs. I was, unfortunately, unable to passthrough the Nvidia card due to constant BSOD (System Thread Exception). However, with newer Nvidia card, or another AMD GPU it is possible to run even 3 separate machines from this setup! - unRAID does not mind at all that I am actually taking a GPU from it and it assigns it to the VM correctly - it is even able to boot to the VM, the BSOD appears after installing Nvidia drivers. Hope I'll help at least somebody trying to achieve the same I was able to. Thank you Limetech! I tried to make this happen myself, but I gave up after many unsuccessful days of trying. Your solution worked right away! I'll be happy to buy the unRAID for my purposes :-).