Let's start with my setup:
Motherboard: ASUS X470-F Gaming (Socket AM4) CPU: AMD Ryzen 7 2700X (does not have integrated graphics) Memory: 32GB Storage: 2 x 512GB NVME SSDs (no RAID) GPU0 (top PCIE slot): EVGA GTX 1050 GPU1 (secondary PCIE slot): EVGA GTX 1080Ti Hybrid Network: Single onboard gigabit nic
Everyone's got something different, but the above it working for me just fine.
Here are the hurdles I faced during this journey:
* My Native Instruments Komplete Audio 6 would panic and reset itself in a Windows 10 guest.
- To fix this, I had to look at my IOMMU groups and find a USB controller that doesn't share the same IOMMU group as anything else important, then "stub" it to keep the Unraid underlying OS from claiming it during boot.
* As a result of the above, I became limited in which USB ports I could use for VMs
- But, to be fair, I can now unplug/replug anything I want on the Windows 10 guest. That's the benefit of binding an entire USB controller.
* I found I was completely unable to pass through my primary graphics card to a Linux guest.
- To fix this, I had to stub the card, AND - and this is the most important bit - I had to disable EFI frame buffering (or something like that). These bits are all done in the Unraid kernel startup line.
* My Linux guest had stuttery, demonic audio over HDMI and video would lag when audio was playing.
- To fix this, I had to enable some intel sound options inside the Linux guest, which I found amusing since nothing in my machine is Intel (something about the virt layer needing this).
Let's dig into each.
USB audio interface runs terribly when passed through
This happens because of the emulation layer. The only way will be to find out which USB controller can be stubbed.
To figure out which one, get on Unraid and open a terminal and run:
lspci | grep USB
Take note of the IDs there, then run:
for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done
In this big list you're looking for something like this:
IOMMU group 20
0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
Notice how this USB controller is the ONLY device in this IOMMU group (which just so happens to be group 20 on my machine). This means that we should be able to completely isolate it from the Unraid host OS so we can let the VM claim it completely and without question.
To stub it, we modify the kernel startup line. You can either use terminal and modify /boot/syslinux/syslinux.cfg or you can use the Unraid web UI and go to Main -> click "Flash" under 'Boot Device' -> then edit the 'Unraid OS' section under 'Syslinux Configuration'
Here's what it looks like when I stub ONLY the above USB controller:
kernel /bzimage
append initrd=/bzroot vfio-pci.ids=1022:145f
Save that change, then reboot Unraid. You can now edit your VM and at the bottom, uncheck the USB devices and instead pass through the USB hub itself. That should fix it.
I can't pass through my GPU in the top slot because Unraid is using it
Correct, but we can be aggressive about it and do it anyway. Start by finding the device IDs using lspci:
root@funraid:~# lspci | grep VGA -A1
09:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 3GB] (rev a1)
09:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
0a:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
0a:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
Above, you can see that my GTX 1050 and its associated audio interface are on 09:00.(0|1)
So now I need to find out the device IDs:
root@funraid:~# lspci -n | grep 09:00.
09:00.0 0300: 10de:1c83 (rev a1)
09:00.1 0403: 10de:0fb9 (rev a1)
Awesome. Let's stub those, too... but remember! We need to also disable EFI frame buffering (I think that's what it's called). This means that when you boot Unraid next, you WILL NOT have any console output after the main bootloader (the blue screen) completes - your main GPU that you're passing through (in this case, my GTX 1050) will appear to freeze at that bootloader. If you see the OS even remotely start to boot, you didn't do this right.
In addition to the USB hub, we'll also stub the GPU and its audio interface and disable EFI FB:
kernel /bzimage
append initrd=/bzroot vfio-pci.ids=1022:145f,10de:1c83,10de:0fb9 video=efifb:off
Save that and reboot Unraid, and all will be well. You might - and this is very unlikely, but - you might need to supply a VGA bios from techpowerup for your card, or worst case put the card you are trying to use here into PCIE slot 2 and dump its BIOS from the Unraid OS, but I really don't think you'll have to.
My Linux guest has demonic, stuttery audio and video when trying to play videos and audio is coming over HDMI through my nVidia card.
Me too, and I dug for a bit and found out that I can simply give the Linux guest a modprobe line to clear up this behavior.
Inside of the Linux guest's OS (NOT UNRAID!), create a conf file in /etc/modprobe.d/... and use the EXACT same data I use below. Here's mine:
[root@archer ~]# cat /etc/modprobe.d/snd-hda-intel.conf
options snd-hda-intel enable_msi=1
That's it. Literally one line. Yeah, you have to type that in word for word. Don't worry, I know, you're not running anything Intel - but trust me, it works.
Save that and reboot the Linux guest OS. Everything should be in order.
I really hope this all helps someone. Thanks! :o)
