Guide - AsRock X570 Taichi VM w/ Hardware Passthrough


Endy

Recommended Posts

I’ve seen a lot of people asking questions and having problems getting graphics cards and onboard usb controllers passed through on Ryzen X570 systems. I had some issues myself that thanks to a couple of people here on the forums I was able to get resolved. This guide is about setting up a Windows 10 vm using a single graphics card with onboard usb passed through. 

 

Because I have managed to get it working, I thought it might be a good idea to write a little guide based on my setup in case it can help some other people.

 

While this was specific to my motherboard and hardware, there are parts that should be applicable for other hardware combinations.

 

Let’s start with my hardware. I’m using a Ryzen 7 3800X with the AsRock X570 Taichi. The graphics card is a Geforce GTX 1070.

 

BIOS

 

First step is to make sure the bios is up to date. At the time I am writing this the newest bios for my motherboard is 2.70 dated 12-9-2019. This is the AGESA version:

 

Update AMD AGESA Combo-AM4 1.0.0.4 Patch B

 

Once updated, boot into the bios once then restore default settings, save, reboot and go back into the bios.

 

To enable IOMMU I found this at level1techs:

Quote

 

For the X570 Taichi go to

Advanced>AMD CBS>NBIO Common Options

Make sure you have “IOMMU” set to “Enable” (Auto doesn’t work).

Also make sure “Enable AER Cap.” is set to “Enable”

Once you’ve set that an option called “ACS Enable” will show up, I left that set to “Auto”.

 

After that’s done you’ll have the normal X570 IOMMU groupings.

 

 

https://forum.level1techs.com/t/x570-taichi-iommu-groups/145762/2

 

bios_iommu.thumb.jpg.59c3e1f7dfc0b7cf7e11ed572bf01974.jpg

 

After some trial and error, I found that in order to passthrough the motherboard usb controllers, I had to change some usb options from Auto to Enabled. Because there is one controller that can’t be passed through, not all of these probably need to be changed, but I’m not sure which ones are for which controllers. Doesn’t seem to be causing me any issues at the moment.

 

Go to Advanced>AMD CBS>FCH Common Options>USB Configuration Options and change XHCI Controller to Enabled.

 

bios_usb_config_options.thumb.jpg.e0b93b466c1e20b21c7dbf2234863b5c.jpg

 

While there go to MCM USB enable and set XHCI2 enable and XHCI3 enable to Enabled.

 

bios_usb_config_mcm.thumb.jpg.7df80fa3d62e37ede6447511f84cb994.jpg

 

Next is Advanced>AMD CBS>Chipset Common Options>Chipset USB Configuration Options. Change XHCI Controller0 enable and XHCI Controller1 enable to Enabled.

 

bios_chipset_usb_config.thumb.jpg.b0e3cda4152c42b5b7a9e4a46cc9ad37.jpg

 

The last thing is to set the boot device to the Unraid usb non uefi. (Supposedly booting the usb using uefi can cause problems, but I have not experienced those myself so I can’t comment on that. This is the better safe than sorry approach.)

 

bios_boot_config.thumb.jpg.8ad50d679d80f4a3866576dbd3ab206f.jpg

 

Save settings and reboot.

 

Isolating Cores

 

This next step may be considered optional, but it’s about making sure the vm has the best chance of performing as best it can. In Unraid go to Settings>CPU Pinning. At the bottom you can choose which CPU cores you would like to isolate. Unraid like to use core 0, so I stay away from that one and usually start at the other end. Keep in mind to match the hyperthreads together. In my case I went with cores 4-7 and the matching hyperthreads 12-15.

 

cpu_pinning.png.bc14eb45f350abf4a2f6b6b388cdcdaf.png

 

Syslinux

 

Next, in Unraid, go to the Main tab and click on Flash then scroll down to Syslinux Configuration. To edit these click on Menu View and it will change to Raw View.

 

I like to have a separate menu option for hardware passthrough so I just copy everything for ‘label Unraid OS’, delete ‘menu default’ under ‘label Unraid OS’, then paste in the new menu item with the other menu items. Make sure to rename it and that it says ‘menu default’ so that’s the one that Unraid will boot into automatically.

 

The only other change is to add ‘video=efifb:off’ after ‘ initrd=/bzroot’. I admit that I am not sure on the details of why this is necessary, but I believe it has to do with passing through the only graphics card in the system.

label Unraid OS HWPT
menu default
kernel /bzimage
append isolcpus=4-7,12-15 initrd=/bzroot video=efifb:off

 

vfio-pci.cfg

 

To stub the graphics card and usb controller I highly recommend using the VFIO-PCI CFG plugin. So far it works really well and makes things much easier. Plus, it gives useful information like if your device can be reset and also which devices are attached to which usb controllers (very useful for figuring out which usb ports belong to which usb controllers). The plugin can be found in Community Apps.

 

Once installed you can find it in the Settings tab. Go there and it’s a good time to double check that the IOMMU groups are looking like they should. Then it’s as easy as selecting which devices you want to stub.

 

Check the graphics card and it’s audio controller.

 

For the onboard usb controllers, it’s tempting to try using the one that is off on in it’s own group. In my case that was this one:

 

Group 33 10:00.3 [RESET] 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

 

However, that one will not work and was one of the problems I ran into, but thanks to an explanation by @Skitals it makes sense why that one would not work. The address is Bus:Device:Function. Notice that the Function for that usb controller is 3. That means that despite being in it’s own IOMMU group, it belongs to another device and therefore, can’t be passed through.

 

Good thing there are two other usb controllers on this motherboard in this group:

Group 20 03:08.0 [RESET] 1022:57a4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
         0a:00.0 [RESET] 1022:1485 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
         0a:00.1 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
         0a:00.3 [RESET] 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

 

Select the three Oa:00 devices.

 

Make sure the usb drive that Unraid uses is not in one of the usb ports for those two controllers and then reboot.

 

For reference, here’s a picture that shows the back panel of the X570 Taichi. The ports colored in red belong to the usb controller that’s off in it’s own group. The yellow and green colored ports belong to the two other usb controllers. I have not yet tried to figure out which controllers the mother usb headers belong to.

 

X570-Taichi-backplate.jpg.938bff89d0f9af8ffdbf0a3335535a57.jpg

 

vm xml – graphics card

 

Before creating the vm, we’ll need a dump of the vbios. While it is possible to download from techpowerup, that may or may not be the right vbios for your exact card. I found it was easiest to put Hiren’s BootCD on a usb drive and swap that with the unraid usb drive, boot into that, dump the vbios with GPUZ and save that to the usb drive. Take that usb drive to another computer and edit the vbios with a hex editor to remove the header as seen in this Spaceinvader One video:

 

https://www.youtube.com/watch?v=1IP-h9IKof0&t=438s

 

Put the Unraid usb drive back in the server and start it up. (Now is a good time to make sure it’s still booting into Legacy mode and not uefi as I noticed mine had switched at this point.) Put the vbios file somewhere on your Unraid server so that the vm will be able to access it. (Maybe a vbios folder in with your ISO folders.)

 

Time to create a Windows 10 vm. Go to the VMS tab, Add VM, Windows 10. I’ll just mention the things that I actually change and the rest of the settings will stay at defaults.

 

Logical CPUs - change this to match the the cores you isolated earlier

 

Initial and Max Memory – change these to the desired amount of RAM for the vm, I’m using 8192 MB for both

 

Machine – Q35

 

OS Install ISO – point this to your Windows 10 install ISO

 

VirtIO Drivers ISO – might want to make sure you have the latest version (virtio-win-0.1.171.iso as of the time I am writing this) They can be found here under Direct downloads virtio-win iso:

 

https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/index.html

 

Note: You can set your default virtio iso in Settings>VM Manager.

 

Graphics Card – set to your graphics card

 

Graphics ROM BIOS – point this to the vbios file you got from your graphics card earlier

 

Sound Card – set to the graphics card audio controller

 

Other PCI Devices – I had some issues with this but so far I have found two ways that seem to work. The first is to just select the usb controller that ends in .3. (In the reference photo above, I believe it’s the usb ports colored in yellow.) The other way is to select the 2 usb controllers and the non-essential instrumentation. Then the xml also needs to be edited which will be explained below.

 

Uncheck Start VM after creation

 

Click Create

 

Now the xml of the vm needs to be edited, so go in to edit it and click on Form View to change it to XML View.

 

Now we need to follow this video by Spaceinvader One. There is one thing I noticed that was different for my setup and that was that it was the Bus that was different and not the Slot, so I went ahead and changed the Bus to match.

 

https://www.youtube.com/watch?v=QlTVANDndpM&t=601s

 

Usb Controller

 

If both usb controllers were selected, then the xml needs to be edited in a similar fashion as to the way the graphics card xml was edited.

 

Here’s mine for an example:

<hostdev mode='subsystem' type='pci' managed='yes'>
 <driver name='vfio'/>
  <source>
   <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
  </source>
 <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
 <driver name='vfio'/>
  <source>
   <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
  </source>
 <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
 <driver name='vfio'/>
  <source>
   <address domain='0x0000' bus='0x0a' slot='0x00' function='0x3'/>
  </source>
 <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x2'/>
</hostdev>

 

I should also mention that I changed the Bus since before it was audio was 5, non-essentials device was 6, usb controller was 7 and usb controller was 8. With the audio moved to Bus 4 with the graphics card, I changed the other 3 devices to the now vacated Bus 5.

 

That should about cover it.

 

If you are trying to use a previous vm and it doesn’t start and no error comes up and it’s just nothing on the screen, try removing the vm and recreating it. (Thanks to @testdasi for that one.)

Edited by Endy
  • Like 4
  • Thanks 3
Link to comment

Thanks for the write up.

 

I also have the same motherboard but the CPU is the 3950x.

 

After creating the VM did you manage to installed the motherboard drives from ASRock? It won’t let me as it says it’s not a compatible device? I guess it’s because it’s a VM.

 

Where the slots both the same for your GPU? Again, mine is similar being a 1080 Ti with both slots 0 and and bus different.

 

Lastly, in device manager the only thing which isn’t installed is a unknown device? I cannot work out what it is?

 

The other PCIe items installed and working ‘in unraid’ is a SFP+ card, LSI controller and then the USB controller but that’s passed through to Windows. I find it simply easier to use a PCIe USB controller entirely passed through instead of working out which motherboard ones are for which.

 

Cheers

Link to comment
1 hour ago, jazzy192 said:

After creating the VM did you manage to installed the motherboard drives from ASRock? It won’t let me as it says it’s not a compatible device? I guess it’s because it’s a VM.

I didn't really go into actually setting up Windows and installing drivers. Spaceinvader One has a video for that as well. But basically this is where the VirtIO drivers come in and I'd have to watch the video again to tell you what exactly is needed.

 

1 hour ago, jazzy192 said:

Where the slots both the same for your GPU? Again, mine is similar being a 1080 Ti with both slots 0 and and bus different.

Yes, the slots were the same. On a Windows 7 vm, the bus was the same but the slots were different, just like in the video. I do not know why that is. But basically, because the graphics and sound are one device and that's the way we want them to be seen by the vm, both the bus and the slot need to match.

 

1 hour ago, jazzy192 said:

Lastly, in device manager the only thing which isn’t installed is a unknown device? I cannot work out what it is?

This is one of those things that I know I have dealt with in the past, but my memory fails me to the specifics. Again, I think this may be in the video for setting up a Windows 10 vm. I think I have also recently read a post about that here on the forum somewhere?

 

1 hour ago, jazzy192 said:

 

The other PCIe items installed and working ‘in unraid’ is a SFP+ card, LSI controller and then the USB controller but that’s passed through to Windows. I find it simply easier to use a PCIe USB controller entirely passed through instead of working out which motherboard ones are for which.

 

Cheers

I would say with the new vfio-pci.cfg plugin, it is much much easier now to determine which usb port is which and plan accordingly. But it's also good to have options.

Link to comment
43 minutes ago, Endy said:

I didn't really go into actually setting up Windows and installing drivers. Spaceinvader One has a video for that as well. But basically this is where the VirtIO drivers come in and I'd have to watch the video again to tell you what exactly is needed.

 

Yes, the slots were the same. On a Windows 7 vm, the bus was the same but the slots were different, just like in the video. I do not know why that is. But basically, because the graphics and sound are one device and that's the way we want them to be seen by the vm, both the bus and the slot need to match.

 

This is one of those things that I know I have dealt with in the past, but my memory fails me to the specifics. Again, I think this may be in the video for setting up a Windows 10 vm. I think I have also recently read a post about that here on the forum somewhere?

 

I would say with the new vfio-pci.cfg plugin, it is much much easier now to determine which usb port is which and plan accordingly. But it's also good to have options.

 

Cheers the fast reply. 

 

The video your talking of, is that the generic setting up Windows as a daily driver from a few years ago? If so then I have already done that many times as I had dual Xeon build before my Ryzen.

 

Hmm, if I try and have them on the same BUS then I cannot create the VM? Mine have to be on 4 and 5. The error is like below if I try and change the sound card part to 4 also:

 

Quote

VM creation error

XML error: Attempted double use of PCI Address 0000:04:00.0

 

Its not in the video about the unknown device but I have not looked on the forums really yet. Im also 99% sure it wasn't there after installing the VirtIO drivers. Im pretty sure it was when I changed from UEFI to Legacy. Mine was working fine on UEFI, I just changed it after watching the recent SpaceInvader video.

Link to comment

PS my XML is: 

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>a781c519-e8bf-3b12-2dfb-dd39bead760f</uuid>
  <description>Nvidia 1080 Passed Through</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>24</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='20'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='21'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='22'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='23'/>
    <vcpupin vcpu='8' cpuset='8'/>
    <vcpupin vcpu='9' cpuset='24'/>
    <vcpupin vcpu='10' cpuset='9'/>
    <vcpupin vcpu='11' cpuset='25'/>
    <vcpupin vcpu='12' cpuset='10'/>
    <vcpupin vcpu='13' cpuset='26'/>
    <vcpupin vcpu='14' cpuset='11'/>
    <vcpupin vcpu='15' cpuset='27'/>
    <vcpupin vcpu='16' cpuset='12'/>
    <vcpupin vcpu='17' cpuset='28'/>
    <vcpupin vcpu='18' cpuset='13'/>
    <vcpupin vcpu='19' cpuset='29'/>
    <vcpupin vcpu='20' cpuset='14'/>
    <vcpupin vcpu='21' cpuset='30'/>
    <vcpupin vcpu='22' cpuset='15'/>
    <vcpupin vcpu='23' cpuset='31'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/a781c519-e8bf-3b12-2dfb-dd39bead760f_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='24' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:61:46:e8'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/General Storage/NVIDIA.GTX1080Ti.11264.170118.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

Link to comment

 

22 minutes ago, jazzy192 said:

 

Cheers the fast reply. 

 

The video your talking of, is that the generic setting up Windows as a daily driver from a few years ago? If so then I have already done that many times as I had dual Xeon build before my Ryzen.

I think there have been a couple. One by jonp and one by Spaceinvader One.

 

22 minutes ago, jazzy192 said:

 

Hmm, if I try and have them on the same BUS then I cannot create the VM? Mine have to be on 4 and 5. The error is like below if I try and change the sound card part to 4 also:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/General Storage/NVIDIA.GTX1080Ti.11264.170118.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

I think this is your problem. You left off multifunction='on'. So it should look like this:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/General Storage/NVIDIA.GTX1080Ti.11264.170118.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>

And then you may or may not need to change the Bus of the last device because of the number gap from Bus 4 to Bus 6. I'm not sure how that is handled by the vm and it may not be a problem at all.

 

22 minutes ago, jazzy192 said:

Its not in the video about the unknown device but I have not looked on the forums really yet. Im also 99% sure it wasn't there after installing the VirtIO drivers. Im pretty sure it was when I changed from UEFI to Legacy. Mine was working fine on UEFI, I just changed it after watching the recent SpaceInvader video.

I know that I have seen it somewhere, but I just don't recall where.

Link to comment
6 hours ago, uek2wooF said:

Very nice, this thread helped a lot.  Have you had any luck passing onboard sound or bluetooth?  I also have the x570 taichi.  Feels silly buying lots of usb stuff when the board comes with so much already.

I am not even seeing bluetooth listed in the devices (entirely possible it's there and I am just not seeing it) and I don't use it, so I don't think I am gonna be much help there.

The problem with the onboard audio is the same as with that 3rd usb device. It may be possible to get it working, but I haven't tried it. More info here:

 

Link to comment
11 hours ago, uek2wooF said:

Very nice, this thread helped a lot.  Have you had any luck passing onboard sound or bluetooth?  I also have the x570 taichi.  Feels silly buying lots of usb stuff when the board comes with so much already.

Bluetooth is connected to one of the USB controllers, it passes through with USB. On board audio should work with the kernel linked above.

Link to comment
  • 2 weeks later...

@Endy thank you for this very detailed guide.  I have about the same configuration (ASRock X570 Taichi with 3950x) except that I have an AMD Radeon RX 570, and followed your instructions, and after many tries I haven't been able to get my GPU to passthrough correctly - I just get a black screen.  I have also tried an Nvidia GTX 950 with the same results. I am on the latest bios version 2.70.

 

I have posted thread about my issue and wanted to see if you could help? 

 

Edited by mafoster
added additional info
Link to comment

@mafoster I took a quick look and some of it is out of my realm of experience because I have not tried to set up a vm for Catalina or using an AMD video card.

 

There is one thing I did see. It looks like you were able to stub the video card, but you may want to follow this video:

https://www.youtube.com/watch?v=QlTVANDndpM&t=601s

 

Be aware that changes made to the vm in xml are overwritten if you make any other changes using the gui view.

Link to comment
  • 3 months later...
On 2/7/2020 at 5:55 AM, mafoster said:

Thanks @Endy
 

I have tried everything in that @SpaceInvaderOne video but still get a blank screen on boot, the fan start on the GPU and get the error I mentioned in this post:

 

 

Anyone else that could suggest something?

 

Pinpointing the failures needs your current XML to be examined. Could you share your XML so we can take a look?

Edited by Hakabe
Link to comment
  • 3 weeks later...
On 1/19/2020 at 9:54 PM, Endy said:

I’ve seen a lot of people asking questions and having problems getting graphics cards and onboard usb controllers passed through on Ryzen X570 systems. I had some issues myself that thanks to a couple of people here on the forums I was able to get resolved. This guide is about setting up a Windows 10 vm using a single graphics card with onboard usb passed through. 

 

Because I have managed to get it working, I thought it might be a good idea to write a little guide based on my setup in case it can help some other people.

 

While this was specific to my motherboard and hardware, there are parts that should be applicable for other hardware combinations.

 

Let’s start with my hardware. I’m using a Ryzen 7 3800X with the AsRock X570 Taichi. The graphics card is a Geforce GTX 1070.

 

BIOS

 

First step is to make sure the bios is up to date. At the time I am writing this the newest bios for my motherboard is 2.70 dated 12-9-2019. This is the AGESA version:

 


Update AMD AGESA Combo-AM4 1.0.0.4 Patch B

 

Once updated, boot into the bios once then restore default settings, save, reboot and go back into the bios.

 

To enable IOMMU I found this at level1techs:

 

https://forum.level1techs.com/t/x570-taichi-iommu-groups/145762/2

 

bios_iommu.thumb.jpg.59c3e1f7dfc0b7cf7e11ed572bf01974.jpg

 

After some trial and error, I found that in order to passthrough the motherboard usb controllers, I had to change some usb options from Auto to Enabled. Because there is one controller that can’t be passed through, not all of these probably need to be changed, but I’m not sure which ones are for which controllers. Doesn’t seem to be causing me any issues at the moment.

 

Go to Advanced>AMD CBS>FCH Common Options>USB Configuration Options and change XHCI Controller to Enabled.

 

bios_usb_config_options.thumb.jpg.e0b93b466c1e20b21c7dbf2234863b5c.jpg

 

While there go to MCM USB enable and set XHCI2 enable and XHCI3 enable to Enabled.

 

bios_usb_config_mcm.thumb.jpg.7df80fa3d62e37ede6447511f84cb994.jpg

 

Next is Advanced>AMD CBS>Chipset Common Options>Chipset USB Configuration Options. Change XHCI Controller0 enable and XHCI Controller1 enable to Enabled.

 

bios_chipset_usb_config.thumb.jpg.b0e3cda4152c42b5b7a9e4a46cc9ad37.jpg

 

The last thing is to set the boot device to the Unraid usb non uefi. (Supposedly booting the usb using uefi can cause problems, but I have not experienced those myself so I can’t comment on that. This is the better safe than sorry approach.)

 

bios_boot_config.thumb.jpg.8ad50d679d80f4a3866576dbd3ab206f.jpg

 

Save settings and reboot.

 

Isolating Cores

 

This next step may be considered optional, but it’s about making sure the vm has the best chance of performing as best it can. In Unraid go to Settings>CPU Pinning. At the bottom you can choose which CPU cores you would like to isolate. Unraid like to use core 0, so I stay away from that one and usually start at the other end. Keep in mind to match the hyperthreads together. In my case I went with cores 4-7 and the matching hyperthreads 12-15.

 

cpu_pinning.png.bc14eb45f350abf4a2f6b6b388cdcdaf.png

 

Syslinux

 

Next, in Unraid, go to the Main tab and click on Flash then scroll down to Syslinux Configuration. To edit these click on Menu View and it will change to Raw View.

 

I like to have a separate menu option for hardware passthrough so I just copy everything for ‘label Unraid OS’, delete ‘menu default’ under ‘label Unraid OS’, then paste in the new menu item with the other menu items. Make sure to rename it and that it says ‘menu default’ so that’s the one that Unraid will boot into automatically.

 

The only other change is to add ‘video=efifb:off’ after ‘ initrd=/bzroot’. I admit that I am not sure on the details of why this is necessary, but I believe it has to do with passing through the only graphics card in the system.


label Unraid OS HWPT
menu default
kernel /bzimage
append isolcpus=4-7,12-15 initrd=/bzroot video=efifb:off

 

vfio-pci.cfg

 

To stub the graphics card and usb controller I highly recommend using the VFIO-PCI CFG plugin. So far it works really well and makes things much easier. Plus, it gives useful information like if your device can be reset and also which devices are attached to which usb controllers (very useful for figuring out which usb ports belong to which usb controllers). The plugin can be found in Community Apps.

 

Once installed you can find it in the Settings tab. Go there and it’s a good time to double check that the IOMMU groups are looking like they should. Then it’s as easy as selecting which devices you want to stub.

 

Check the graphics card and it’s audio controller.

 

For the onboard usb controllers, it’s tempting to try using the one that is off on in it’s own group. In my case that was this one:

 


Group 33 10:00.3 [RESET] 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

 

However, that one will not work and was one of the problems I ran into, but thanks to an explanation by @Skitals it makes sense why that one would not work. The address is Bus:Device:Function. Notice that the Function for that usb controller is 3. That means that despite being in it’s own IOMMU group, it belongs to another device and therefore, can’t be passed through.

 

Good thing there are two other usb controllers on this motherboard in this group:


Group 20 03:08.0 [RESET] 1022:57a4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
         0a:00.0 [RESET] 1022:1485 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
         0a:00.1 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
         0a:00.3 [RESET] 1022:149c USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

 

Select the three Oa:00 devices.

 

Make sure the usb drive that Unraid uses is not in one of the usb ports for those two controllers and then reboot.

 

For reference, here’s a picture that shows the back panel of the X570 Taichi. The ports colored in red belong to the usb controller that’s off in it’s own group. The yellow and green colored ports belong to the two other usb controllers. I have not yet tried to figure out which controllers the mother usb headers belong to.

 

X570-Taichi-backplate.jpg.938bff89d0f9af8ffdbf0a3335535a57.jpg

 

vm xml – graphics card

 

Before creating the vm, we’ll need a dump of the vbios. While it is possible to download from techpowerup, that may or may not be the right vbios for your exact card. I found it was easiest to put Hiren’s BootCD on a usb drive and swap that with the unraid usb drive, boot into that, dump the vbios with GPUZ and save that to the usb drive. Take that usb drive to another computer and edit the vbios with a hex editor to remove the header as seen in this Spaceinvader One video:

 

https://www.youtube.com/watch?v=1IP-h9IKof0&t=438s

 

Put the Unraid usb drive back in the server and start it up. (Now is a good time to make sure it’s still booting into Legacy mode and not uefi as I noticed mine had switched at this point.) Put the vbios file somewhere on your Unraid server so that the vm will be able to access it. (Maybe a vbios folder in with your ISO folders.)

 

Time to create a Windows 10 vm. Go to the VMS tab, Add VM, Windows 10. I’ll just mention the things that I actually change and the rest of the settings will stay at defaults.

 

Logical CPUs - change this to match the the cores you isolated earlier

 

Initial and Max Memory – change these to the desired amount of RAM for the vm, I’m using 8192 MB for both

 

Machine – Q35

 

OS Install ISO – point this to your Windows 10 install ISO

 

VirtIO Drivers ISO – might want to make sure you have the latest version (virtio-win-0.1.171.iso as of the time I am writing this) They can be found here under Direct downloads virtio-win iso:

 

https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/index.html

 

Note: You can set your default virtio iso in Settings>VM Manager.

 

Graphics Card – set to your graphics card

 

Graphics ROM BIOS – point this to the vbios file you got from your graphics card earlier

 

Sound Card – set to the graphics card audio controller

 

Other PCI Devices – I had some issues with this but so far I have found two ways that seem to work. The first is to just select the usb controller that ends in .3. (In the reference photo above, I believe it’s the usb ports colored in yellow.) The other way is to select the 2 usb controllers and the non-essential instrumentation. Then the xml also needs to be edited which will be explained below.

 

Uncheck Start VM after creation

 

Click Create

 

Now the xml of the vm needs to be edited, so go in to edit it and click on Form View to change it to XML View.

 

Now we need to follow this video by Spaceinvader One. There is one thing I noticed that was different for my setup and that was that it was the Bus that was different and not the Slot, so I went ahead and changed the Bus to match.

 

https://www.youtube.com/watch?v=QlTVANDndpM&t=601s

 

Usb Controller

 

If both usb controllers were selected, then the xml needs to be edited in a similar fashion as to the way the graphics card xml was edited.

 

Here’s mine for an example:


<hostdev mode='subsystem' type='pci' managed='yes'>
 <driver name='vfio'/>
  <source>
   <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
  </source>
 <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
 <driver name='vfio'/>
  <source>
   <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
  </source>
 <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
 <driver name='vfio'/>
  <source>
   <address domain='0x0000' bus='0x0a' slot='0x00' function='0x3'/>
  </source>
 <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x2'/>
</hostdev>

 

I should also mention that I changed the Bus since before it was audio was 5, non-essentials device was 6, usb controller was 7 and usb controller was 8. With the audio moved to Bus 4 with the graphics card, I changed the other 3 devices to the now vacated Bus 5.

 

That should about cover it.

 

If you are trying to use a previous vm and it doesn’t start and no error comes up and it’s just nothing on the screen, try removing the vm and recreating it. (Thanks to @testdasi for that one.)

 

big thank you to Endy, I have a AsRock X570 mobo and these settings will be handy since am very new to unraid.

 

I, too, noticed after doing further bios updates the iommu grouping was better asrock are doing a good job of releasing newer bios so far. But I think I still ran into sam usb controller issue you mentiioned and had tried all 3 but the other 2 did not work (just hanged the vm and crashed unraid) the separate one works better but has been a bit iffy sometimes does not detect my usb soundcard rarely (creative X3), and since the other 2 usb controllers hanged vm when I used the vci plugin I think I may just get a usb 3 separate card instead to reduce headache.

 

I noticed you used machine Q35 and also a non uefi unraid pen flash, does this give better stability ?

 

Either way I will re-read your guide, I currently have a VM running Win10 successfully but think once I got the seperate usb controller sorted

and I use a usb sound card I should be more solid.

 

Also, worth mentioning it's a good idea to enable "VFIO allow unsafe interrupts" under VM advanced tab settings, this solves the usb issue if you are hot swopping devices and won't hang the vm and crash the unraid server.

 

 

 

Link to comment
  • 2 weeks later...

Hey man!

 

i know this is unrelated to this specific topic, but I was wondering if you had an HBA set up on this machine? I have the same CPU and motherboard, but am having trouble getting my HBA to work.

 

Any ideas would be greatly appreciated:)

 

cheers!

Link to comment
11 minutes ago, Crocs said:

Hey man!

 

i know this is unrelated to this specific topic, but I was wondering if you had an HBA set up on this machine? I have the same CPU and motherboard, but am having trouble getting my HBA to work.

 

Any ideas would be greatly appreciated:)

 

cheers!

I have a Dell H310 flashed to IT mode. I just plugged it in and it worked. It was previously used in my old server using forward breakout cables and now it's connected to the backplane in my CSE-846. 

Link to comment
  • 2 weeks later...

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>ALVIN-RYZEN9</name>
  <uuid>9320-0000-0000-000000000000</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>28</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='18'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='19'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='20'/>
    <vcpupin vcpu='6' cpuset='5'/>
    <vcpupin vcpu='7' cpuset='21'/>
    <vcpupin vcpu='8' cpuset='6'/>
    <vcpupin vcpu='9' cpuset='22'/>
    <vcpupin vcpu='10' cpuset='7'/>
    <vcpupin vcpu='11' cpuset='23'/>
    <vcpupin vcpu='12' cpuset='8'/>
    <vcpupin vcpu='13' cpuset='24'/>
    <vcpupin vcpu='14' cpuset='9'/>
    <vcpupin vcpu='15' cpuset='25'/>
    <vcpupin vcpu='16' cpuset='10'/>
    <vcpupin vcpu='17' cpuset='26'/>
    <vcpupin vcpu='18' cpuset='11'/>
    <vcpupin vcpu='19' cpuset='27'/>
    <vcpupin vcpu='20' cpuset='12'/>
    <vcpupin vcpu='21' cpuset='28'/>
    <vcpupin vcpu='22' cpuset='13'/>
    <vcpupin vcpu='23' cpuset='29'/>
    <vcpupin vcpu='24' cpuset='14'/>
    <vcpupin vcpu='25' cpuset='30'/>
    <vcpupin vcpu='26' cpuset='15'/>
    <vcpupin vcpu='27' cpuset='31'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/1f59a1a8-9320-0000-0000-000000000000_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='off'/>
      <vapic state='off'/>
      <spinlocks state='off'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' dies='1' cores='14' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/nvme-ADATA_SX6000LNP_2K0620005595'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.173-2.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:90:62:6f'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/MSI.RTX2080Super.8192.190625.Edited.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc537'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x258a'/>
        <product id='0x002a'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,topoext=on,invtsc=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-reset,hv-frequencies,host-cache-info=on,l3-cache=off,-amd-stibp'/>
  </qemu:commandline>
</domain>

Edited by Unraid Newbie
UUID
Link to comment

the bottom part was the code i am talking about. Qemu commandline. i need to add this in order to start any vm.

 

this is the one i am having trouble with, trying to pass the 2080 super, no luck yet.

 

tried disable hyper-v,

dumped the vbios,

edited the dumped vbios, no luck

tried the download version of vbios

edited the download version

grouped them in the same virtual port, and set multifunction on

tried to create a custom version of driver, still no luck.

booting in legacy mode

 

please help. 

 

Link to comment
  • 3 weeks later...
  • 4 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.