Endy Posted January 19, 2020 Share Posted January 19, 2020 (edited) I’ve seen a lot of people asking questions and having problems getting graphics cards and onboard usb controllers passed through on Ryzen X570 systems. I had some issues myself that thanks to a couple of people here on the forums I was able to get resolved. This guide is about setting up a Windows 10 vm using a single graphics card with onboard usb passed through. Because I have managed to get it working, I thought it might be a good idea to write a little guide based on my setup in case it can help some other people. While this was specific to my motherboard and hardware, there are parts that should be applicable for other hardware combinations. Let’s start with my hardware. I’m using a Ryzen 7 3800X with the AsRock X570 Taichi. The graphics card is a Geforce GTX 1070. BIOS First step is to make sure the bios is up to date. At the time I am writing this the newest bios for my motherboard is 2.70 dated 12-9-2019. This is the AGESA version: Update AMD AGESA Combo-AM4 126.96.36.199 Patch B Once updated, boot into the bios once then restore default settings, save, reboot and go back into the bios. To enable IOMMU I found this at level1techs: Quote For the X570 Taichi go to Advanced>AMD CBS>NBIO Common Options Make sure you have “IOMMU” set to “Enable” (Auto doesn’t work). Also make sure “Enable AER Cap.” is set to “Enable” Once you’ve set that an option called “ACS Enable” will show up, I left that set to “Auto”. After that’s done you’ll have the normal X570 IOMMU groupings. https://forum.level1techs.com/t/x570-taichi-iommu-groups/145762/2 After some trial and error, I found that in order to passthrough the motherboard usb controllers, I had to change some usb options from Auto to Enabled. Because there is one controller that can’t be passed through, not all of these probably need to be changed, but I’m not sure which ones are for which controllers. Doesn’t seem to be causing me any issues at the moment. Go to Advanced>AMD CBS>FCH Common Options>USB Configuration Options and change XHCI Controller to Enabled. While there go to MCM USB enable and set XHCI2 enable and XHCI3 enable to Enabled. Next is Advanced>AMD CBS>Chipset Common Options>Chipset USB Configuration Options. Change XHCI Controller0 enable and XHCI Controller1 enable to Enabled. The last thing is to set the boot device to the Unraid usb non uefi. (Supposedly booting the usb using uefi can cause problems, but I have not experienced those myself so I can’t comment on that. This is the better safe than sorry approach.) Save settings and reboot. Isolating Cores This next step may be considered optional, but it’s about making sure the vm has the best chance of performing as best it can. In Unraid go to Settings>CPU Pinning. At the bottom you can choose which CPU cores you would like to isolate. Unraid like to use core 0, so I stay away from that one and usually start at the other end. Keep in mind to match the hyperthreads together. In my case I went with cores 4-7 and the matching hyperthreads 12-15. Syslinux Next, in Unraid, go to the Main tab and click on Flash then scroll down to Syslinux Configuration. To edit these click on Menu View and it will change to Raw View. I like to have a separate menu option for hardware passthrough so I just copy everything for ‘label Unraid OS’, delete ‘menu default’ under ‘label Unraid OS’, then paste in the new menu item with the other menu items. Make sure to rename it and that it says ‘menu default’ so that’s the one that Unraid will boot into automatically. The only other change is to add ‘video=efifb:off’ after ‘ initrd=/bzroot’. I admit that I am not sure on the details of why this is necessary, but I believe it has to do with passing through the only graphics card in the system. label Unraid OS HWPT menu default kernel /bzimage append isolcpus=4-7,12-15 initrd=/bzroot video=efifb:off vfio-pci.cfg To stub the graphics card and usb controller I highly recommend using the VFIO-PCI CFG plugin. So far it works really well and makes things much easier. Plus, it gives useful information like if your device can be reset and also which devices are attached to which usb controllers (very useful for figuring out which usb ports belong to which usb controllers). The plugin can be found in Community Apps. Once installed you can find it in the Settings tab. Go there and it’s a good time to double check that the IOMMU groups are looking like they should. Then it’s as easy as selecting which devices you want to stub. Check the graphics card and it’s audio controller. For the onboard usb controllers, it’s tempting to try using the one that is off on in it’s own group. In my case that was this one: Group 33 10:00.3 [RESET] 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller However, that one will not work and was one of the problems I ran into, but thanks to an explanation by @Skitals it makes sense why that one would not work. The address is Bus:Device:Function. Notice that the Function for that usb controller is 3. That means that despite being in it’s own IOMMU group, it belongs to another device and therefore, can’t be passed through. Good thing there are two other usb controllers on this motherboard in this group: Group 20 03:08.0 [RESET] 1022:57a4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4 0a:00.0 [RESET] 1022:1485 Non-Essential Instrumentation : Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP 0a:00.1 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller 0a:00.3 [RESET] 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller Select the three Oa:00 devices. Make sure the usb drive that Unraid uses is not in one of the usb ports for those two controllers and then reboot. For reference, here’s a picture that shows the back panel of the X570 Taichi. The ports colored in red belong to the usb controller that’s off in it’s own group. The yellow and green colored ports belong to the two other usb controllers. I have not yet tried to figure out which controllers the mother usb headers belong to. vm xml – graphics card Before creating the vm, we’ll need a dump of the vbios. While it is possible to download from techpowerup, that may or may not be the right vbios for your exact card. I found it was easiest to put Hiren’s BootCD on a usb drive and swap that with the unraid usb drive, boot into that, dump the vbios with GPUZ and save that to the usb drive. Take that usb drive to another computer and edit the vbios with a hex editor to remove the header as seen in this Spaceinvader One video: https://www.youtube.com/watch?v=1IP-h9IKof0&t=438s Put the Unraid usb drive back in the server and start it up. (Now is a good time to make sure it’s still booting into Legacy mode and not uefi as I noticed mine had switched at this point.) Put the vbios file somewhere on your Unraid server so that the vm will be able to access it. (Maybe a vbios folder in with your ISO folders.) Time to create a Windows 10 vm. Go to the VMS tab, Add VM, Windows 10. I’ll just mention the things that I actually change and the rest of the settings will stay at defaults. Logical CPUs - change this to match the the cores you isolated earlier Initial and Max Memory – change these to the desired amount of RAM for the vm, I’m using 8192 MB for both Machine – Q35 OS Install ISO – point this to your Windows 10 install ISO VirtIO Drivers ISO – might want to make sure you have the latest version (virtio-win-0.1.171.iso as of the time I am writing this) They can be found here under Direct downloads virtio-win iso: https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/index.html Note: You can set your default virtio iso in Settings>VM Manager. Graphics Card – set to your graphics card Graphics ROM BIOS – point this to the vbios file you got from your graphics card earlier Sound Card – set to the graphics card audio controller Other PCI Devices – I had some issues with this but so far I have found two ways that seem to work. The first is to just select the usb controller that ends in .3. (In the reference photo above, I believe it’s the usb ports colored in yellow.) The other way is to select the 2 usb controllers and the non-essential instrumentation. Then the xml also needs to be edited which will be explained below. Uncheck Start VM after creation Click Create Now the xml of the vm needs to be edited, so go in to edit it and click on Form View to change it to XML View. Now we need to follow this video by Spaceinvader One. There is one thing I noticed that was different for my setup and that was that it was the Bus that was different and not the Slot, so I went ahead and changed the Bus to match. https://www.youtube.com/watch?v=QlTVANDndpM&t=601s Usb Controller If both usb controllers were selected, then the xml needs to be edited in a similar fashion as to the way the graphics card xml was edited. Here’s mine for an example: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x2'/> </hostdev> I should also mention that I changed the Bus since before it was audio was 5, non-essentials device was 6, usb controller was 7 and usb controller was 8. With the audio moved to Bus 4 with the graphics card, I changed the other 3 devices to the now vacated Bus 5. That should about cover it. If you are trying to use a previous vm and it doesn’t start and no error comes up and it’s just nothing on the screen, try removing the vm and recreating it. (Thanks to @testdasi for that one.) Edited January 20, 2020 by Endy 4 3 Quote Link to comment
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.