[Guide] Pass Through Primary GPU (Headless Unraid!), Fix USB Panic Problems, and Linux Guest Audio/Video Stutter


Recommended Posts

Let's start with my setup:

 

Motherboard: ASUS X470-F Gaming (Socket AM4)
CPU: AMD Ryzen 7 2700X (does not have integrated graphics)
Memory: 32GB
Storage: 2 x 512GB NVME SSDs (no RAID)
GPU0 (top PCIE slot): EVGA GTX 1050
GPU1 (secondary PCIE slot): EVGA GTX 1080Ti Hybrid
Network: Single onboard gigabit nic

 

Everyone's got something different, but the above it working for me just fine.

 

Here are the hurdles I faced during this journey:

 

* My Native Instruments Komplete Audio 6 would panic and reset itself in a Windows 10 guest.
- To fix this, I had to look at my IOMMU groups and find a USB controller that doesn't share the same IOMMU group as anything else important, then "stub" it to keep the Unraid underlying OS from claiming it during boot.

 

* As a result of the above, I became limited in which USB ports I could use for VMs
- But, to be fair, I can now unplug/replug anything I want on the Windows 10 guest.  That's the benefit of binding an entire USB controller.

 

* I found I was completely unable to pass through my primary graphics card to a Linux guest.
- To fix this, I had to stub the card, AND - and this is the most important bit - I had to disable EFI frame buffering (or something like that).  These bits are all done in the Unraid kernel startup line.

 

* My Linux guest had stuttery, demonic audio over HDMI and video would lag when audio was playing.
- To fix this, I had to enable some intel sound options inside the Linux guest, which I found amusing since nothing in my machine is Intel (something about the virt layer needing this).

 

Let's dig into each.

 

USB audio interface runs terribly when passed through

 

This happens because of the emulation layer.  The only way will be to find out which USB controller can be stubbed.

 

To figure out which one, get on Unraid and open a terminal and run:

lspci | grep USB

Take note of the IDs there, then run:

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

In this big list you're looking for something like this:

IOMMU group 20
0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]

Notice how this USB controller is the ONLY device in this IOMMU group (which just so happens to be group 20 on my machine).  This means that we should be able to completely isolate it from the Unraid host OS so we can let the VM claim it completely and without question.

 

To stub it, we modify the kernel startup line.  You can either use terminal and modify /boot/syslinux/syslinux.cfg or you can use the Unraid web UI and go to Main -> click "Flash" under 'Boot Device' -> then edit the 'Unraid OS' section under 'Syslinux Configuration'

 

Here's what it looks like when I stub ONLY the above USB controller:

kernel /bzimage
append initrd=/bzroot vfio-pci.ids=1022:145f

Save that change, then reboot Unraid.  You can now edit your VM and at the bottom, uncheck the USB devices and instead pass through the USB hub itself.  That should fix it.

 

I can't pass through my GPU in the top slot because Unraid is using it

 

Correct, but we can be aggressive about it and do it anyway.  Start by finding the device IDs using lspci:

root@funraid:~# lspci | grep VGA -A1
09:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 3GB] (rev a1)
09:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
0a:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
0a:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)

Above, you can see that my GTX 1050 and its associated audio interface are on 09:00.(0|1)

 

So now I need to find out the device IDs:

root@funraid:~# lspci -n | grep 09:00.
09:00.0 0300: 10de:1c83 (rev a1)
09:00.1 0403: 10de:0fb9 (rev a1)

Awesome.  Let's stub those, too... but remember!  We need to also disable EFI frame buffering (I think that's what it's called).  This means that when you boot Unraid next, you WILL NOT have any console output after the main bootloader (the blue screen) completes - your main GPU that you're passing through (in this case, my GTX 1050) will appear to freeze at that bootloader.  If you see the OS even remotely start to boot, you didn't do this right.

 

In addition to the USB hub, we'll also stub the GPU and its audio interface and disable EFI FB:

kernel /bzimage
append initrd=/bzroot vfio-pci.ids=1022:145f,10de:1c83,10de:0fb9 video=efifb:off

Save that and reboot Unraid, and all will be well.  You might - and this is very unlikely, but - you might need to supply a VGA bios from techpowerup for your card, or worst case put the card you are trying to use here into PCIE slot 2 and dump its BIOS from the Unraid OS, but I really don't think you'll have to.

 

My Linux guest has demonic, stuttery audio and video when trying to play videos and audio is coming over HDMI through my nVidia card.

 

Me too, and I dug for a bit and found out that I can simply give the Linux guest a modprobe line to clear up this behavior.

 

Inside of the Linux guest's OS (NOT UNRAID!), create a conf file in /etc/modprobe.d/... and use the EXACT same data I use below.  Here's mine:

[root@archer ~]# cat /etc/modprobe.d/snd-hda-intel.conf 
options snd-hda-intel enable_msi=1

That's it.  Literally one line.  Yeah, you have to type that in word for word.  Don't worry, I know, you're not running anything Intel - but trust me, it works.

Save that and reboot the Linux guest OS.  Everything should be in order.

 

I really hope this all helps someone.  Thanks! :o)

  • Like 2
  • Upvote 2
Link to comment
  • 1 year later...
  • 10 months later...
On 2/2/2019 at 1:30 AM, sixsicsix said:

In addition to the USB hub, we'll also stub the GPU and its audio interface and disable EFI FB:


kernel /bzimage
append initrd=/bzroot vfio-pci.ids=1022:145f,10de:1c83,10de:0fb9 video=efifb:off

Save that and reboot Unraid, and all will be well.  You might - and this is very unlikely, but - you might need to supply a VGA bios from techpowerup for your card, or worst case put the card you are trying to use here into PCIE slot 2 and dump its BIOS from the Unraid OS, but I really don't think you'll have to.


This has saved me from another 3hrs down a rabbithole. In all the guides & threads ive searched, only here did it give me the exact answer i needed when Unraid held one of my GPUs ransom.

Link to comment
  • 6 months later...

unfortunately this does not work for me. im running latest version of unraid. tried stubbing the devices manually in devices page. supplied bios file. and selected correct sound. but does not want to work i have added

vfio-pci.ids=10de:0e0f,10de:128b video=efifb:off

and restarted but i still dont get any sound. if i take the sound option out of the vm settings. the vm starts fine.

Link to comment
  • 10 months later...

Hi,

Just created an account on unraid forums to say thank you. This post is gold !

 

Since 6.9, I'm not sure it is still necessary to stub usb hub in sysconfig. I didn't need it. I just checked the corresponding iommu group in the tools / system devices page then passed it through the VM.

I I'm not mistaken, stubbing the graphic card is still useful as I don't want unraid to use it at start so I can boot my VM on my primary pcie slot without reseting it after a reboot.

Edited by VertPingouin
added precisions for 6.9 from my humble understanding
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.