Skitals Posted November 22, 2019 Share Posted November 22, 2019 (edited) I've spent a few weeks getting my X570 AORUS Elite WiFi + 3900X + GTX1070 running to my liking so I thought I would share. These settings are also confirmed working on the AORUS Pro WiFi and AORUS Ultra. It will probably be similar for all the X570 AORUS boards. Here are the settings for USB passthrough, single Nvidia GPU passthrough, and more. Using BIOS F10. This is important as your IOMMU groupings can/will change with AGESA updates. UEFI / BIOS Settings: Tweaker -> Advanced CPU Settings -> SVM Mode -> Enable Settings -> Miscellaneous -> IOMMU -> Enable Settings -> AMD CBS -> ACS Enable -> Enable Settings -> AMD CBS -> Enable AER Cap -> Enable USB Passthrough: Leaving PCIe ACS override disabled, you should have ~34 IOMMU groups (give or take depending on how many PCIe devices you have connected) if you look in Tools > System Devices. There should be 3 USB controllers with the same vendor/device ID (1022:149c). Two of them will be lumped together with a PCI bridge and "Non-Essential Instrumentation." Those are the two we want to pass! The more logical option would be the controller isolated in its own group, but I could NOT get that one to pass. The trick is run your Unraid USB off that third controller, and we can pass the other two controllers together. Run your Unraid USB out of the rear white USB port labeled BIOS. That white USB 3.0 port plus the neighboring 3 blue USB 3.0 ports share a controller. Use these other ports for your keyboard and mouse (to be passed through as devices) and your UPS or whatever else you want Unraid to access. Note the addresses of the two USB controllers AND the "Non-Essential Instrumentation" in that IOMMU. In my case they are 07:00.0, 07:00.1, 07:00.3. Create the file /boot/config/vfio-pci.cfg with the following contents: Quote BIND=07:00.0 07:00.1 07:00.3 When you reboot, these devices will available in the vm xml gui to passthrough under Other PCI Devices. Pass all 3 of them together! If you do not pass the "Non-Essential Instrumentation" Unraid will throw a warning in the logs that the .1 controller is dependent on it and unavailable to reset. When you passthrough all three together you will get no errors/warnings and everything works. Bonus: Bluetooth on this board is a usb device tied to the .3 controller and is passed through along with the controller! Note: When you add or remove PCIe devices, these addresses can/will change. When you add or remove a PCIe device, check Tools > System Devices to see if the USB addresses have changed and update vfio-pci.cfg accordingly. Single (NVIDIA) GPU Passthrough: For single GPU passthrough, you need to disable graphical output in Unraid. From the Main menu, click the name of your Boot Device (flash). Under Syslinux Config -> Unraid OS, add "video=efifb:off" after "append initrd=/bzroot". The line should now read "append initrd=/bzroot video=efifb:off". When you reboot you will notice there is no video output when unraid boots (you will be left with a freeze frame of the boot menu). Your solo GPU is now ready to pass. For Nvidia you will need the vbios for your card. I dumped my own following this tutorial using second gpu. If you can't dump your own, trying following this tutorial to download/modify a working vbios. Now simply pass your GPU, vbios, and the Sound Card that goes with your GPU from the vm xml gui. Fan Speed Sensors and PWM Controllers: See warning below! You can already see your CPU temp (Tctl) using the k10temp driver with Dynamix System Temperature. If you want to see Fan Speeds on your dashboard, or use the Dynamix Auto Fan Control plugin, we can force the it87 driver to load for the it8628 on this board. To force this we need to set another boot flag, "acpi_enforce_resources=lax". Add this the same way as above, after "video=efifb:off". That line in your syslinux.cfg should now read "append initrd=/bzroot video=efifb:off acpi_enforce_resources=lax". Next, add the following line to /boot/config/go Quote modprobe it87 force_id=0x8628 The it87 driver will now load on boot and your fan speeds will be displayed on the Unraid dashboard, and the fan controllers will be available in Dynamix Auto Fan Control. Warning: Setting acpi_enforce_resources to lax is considered risky for reasons explained here. Edited November 25, 2019 by Skitals 3 3 Quote Link to comment
rachid596 Posted November 22, 2019 Share Posted November 22, 2019 Waw, Thank you very much. Can you post your iommu group ? I research a good motherboard for a ryzen 3900x. Quote Link to comment
Skitals Posted November 22, 2019 Author Share Posted November 22, 2019 10 minutes ago, rachid596 said: Waw, Thank you very much. Can you post your iommu group ? I research a good motherboard for a ryzen 3900x. Here they are with F10 bios and above settings. The Fresco Logic FL1100 is a PCIe card. 2 Quote Link to comment
Skitals Posted November 22, 2019 Author Share Posted November 22, 2019 Note that I can not pass the on board audio even though it's in it's own group. I've always used a (USB) Astro Mixamp so it's no loss to me. 1 Quote Link to comment
rachid596 Posted November 22, 2019 Share Posted November 22, 2019 Note that I can not pass the on board audio even though it's in it's own group. I've always used a (USB) Astro Mixamp so it's no loss to me.Thank you very much, almost perfect Board. I would have to Pass the soundcard.Envoyé de mon SM-N960F en utilisant Tapatalk Quote Link to comment
Skitals Posted November 25, 2019 Author Share Posted November 25, 2019 The Aorus boards are on sale at Microcenter and I was within my 30-day return period so I just upgraded to the x570 Aorus Ultra. The IOMMU groups are identical. In fact, I didn't have to change a single thing to get back up and running. The USB controllers are split the same way, with the same 4 ports integrated in the cpu (the ones we can't pass). That means the rear USB Type-C passes (along with the type-c header)! Going from Aorus Elite -> Ultra nets me: +1 m.2 slot (3 total!) +1 PCIe x16 slot (@x8) +1 USB Type-C port (-1 type A) Realtek® ALC1220 -> Realtek® ALC1220-VB (haven't succesfully passed this) Bluetoooth 4.2 -> 5.0 (passes with the usb controllers) Wi-Fi 802.11a/b/g/n/ac -> Wi-Fi 802.11a/b/g/n/ac/ax (no use for this right now) Some other fun stuff, like 2 temp sensor headers (with included thermistors) Quote Link to comment
Sleepyme Posted December 2, 2019 Share Posted December 2, 2019 On 11/25/2019 at 3:05 PM, Skitals said: The Aorus boards are on sale at Microcenter and I was within my 30-day return period so I just upgraded to the x570 Aorus Ultra. The IOMMU groups are identical. In fact, I didn't have to change a single thing to get back up and running. The USB controllers are split the same way, with the same 4 ports integrated in the cpu (the ones we can't pass). That means the rear USB Type-C passes (along with the type-c header)! Going from Aorus Elite -> Ultra nets me: +1 m.2 slot (3 total!) +1 PCIe x16 slot (@x8) +1 USB Type-C port (-1 type A) Realtek® ALC1220 -> Realtek® ALC1220-VB (haven't succesfully passed this) Bluetoooth 4.2 -> 5.0 (passes with the usb controllers) Wi-Fi 802.11a/b/g/n/ac -> Wi-Fi 802.11a/b/g/n/ac/ax (no use for this right now) Some other fun stuff, like 2 temp sensor headers (with included thermistors) Did you need to install the chipset drivers or did windows automatically install them? Quote Link to comment
david279 Posted December 2, 2019 Share Posted December 2, 2019 Dont have to install any chipset drivers when using Vms Quote Link to comment
phat_cow Posted December 2, 2019 Share Posted December 2, 2019 (edited) 20 minutes ago, david279 said: Dont have to install any chipset drivers when using Vms I think I got mine passed through but I tried plugging in a usb device and windows doesn't detect it. Is it normal that it says it's an Nvidia usb controller? Also, in device manager one of the controllers is not working. Any I missing something? I accessed the unraid flash drive via Flash share in windows and created the vfio-pci.cfg file inside the config folder with the following content: "BIND=07:00.0 07:00.1 07:00.3" That should've worked right? Edited December 2, 2019 by phat_cow Quote Link to comment
david279 Posted December 2, 2019 Share Posted December 2, 2019 When you install the Nvidia GPU driver that driver for the USB c port on the rtx card will install. For some reason I thought you were talking about amd chipset drivers.Sent from my SM-N960U using Tapatalk Quote Link to comment
phat_cow Posted December 2, 2019 Share Posted December 2, 2019 Crap those are not usb controllers from my motherboard.....I guess mine didn't passthrough correctly. Quote Link to comment
Sleepyme Posted December 2, 2019 Share Posted December 2, 2019 (edited) Mine didn't work. I tried BIND=07:00.0 07:00.1 07:00.3 and BIND="07:00.0 07:00.1 07:00.3" And only the nvidia usb controllers showed up. Edited December 2, 2019 by Sleepyme Quote Link to comment
Sleepyme Posted December 2, 2019 Share Posted December 2, 2019 Anyone else have any success getting this to work? Quote Link to comment
Sleepyme Posted December 3, 2019 Share Posted December 3, 2019 Is it possible unraid is not loading my vfio-pci.cfg file? Ifnits not loading it. How do i get unraid to load it? Quote Link to comment
Skitals Posted December 3, 2019 Author Share Posted December 3, 2019 I see other (old) bug reports about vfio-pci.cfg not loading, but I don't see an immediate solution. What version of unraid are you running, and what does your syslinux.cfg look like? Try booting unraid in safe mode (essentially bypassing any changes you made to the default boot parameters) and see if the binding works. Quote Link to comment
righardt.marais Posted December 5, 2019 Share Posted December 5, 2019 Hi, I have now unsuccessfully tried for a week to get my gigabyte x5700xt card passed through to a win10 pro vm on my trial unraid. Hardware: GB Aorus Ultra mobo F10 GB x5700xt card 32gb ram 2 x 2tb nvme USB 4 port card 2 x displays Requirements: Want unraid as host os with a win10 gaming vm. X5700xt should be passed for gaming. I want second/third vm for work related win/Linux stuff. Could add second video card if required. Peripherals like mouse, keyb, sound should be passed to main gaming vm. Work VMs will be accessed from gaming vm or laptop via RDP or SSH. Failures: I have gone through the space invader one vids and tried to setup the unraid with split iommu groups. All the virtualisation has been switched on in BIOS. Win10 vm was created and it's fine in vnc graphics mode. As soon as I pass the navi10 card and sound card with the GPU ROM passed, it either hangs or does nothing or vm stops. I have temporarily given up by installing win10 on 1st nvme and putting all games and work related stuff on that hardware. The second nvme I kept open for the possibility of trying/messing around with the unraid setup again. Is there anyone who could guide me step by step regarding the GPU pass through and other USB peripherals. If I get it up and running I will document and redo the whole machine with unraid again. Many thanks Righardt Quote Link to comment
david279 Posted December 5, 2019 Share Posted December 5, 2019 Try legacy mode instead of UEFI boot mode of that's what you are using. Quote Link to comment
Skitals Posted December 5, 2019 Author Share Posted December 5, 2019 4 hours ago, righardt.marais said: Hi, I have now unsuccessfully tried for a week to get my gigabyte x5700xt card passed through to a win10 pro vm on my trial unraid. Hardware: GB Aorus Ultra mobo F10 GB x5700xt card 32gb ram 2 x 2tb nvme USB 4 port card 2 x displays Requirements: Want unraid as host os with a win10 gaming vm. X5700xt should be passed for gaming. I want second/third vm for work related win/Linux stuff. Could add second video card if required. Peripherals like mouse, keyb, sound should be passed to main gaming vm. Work VMs will be accessed from gaming vm or laptop via RDP or SSH. Failures: I have gone through the space invader one vids and tried to setup the unraid with split iommu groups. All the virtualisation has been switched on in BIOS. Win10 vm was created and it's fine in vnc graphics mode. As soon as I pass the navi10 card and sound card with the GPU ROM passed, it either hangs or does nothing or vm stops. I have temporarily given up by installing win10 on 1st nvme and putting all games and work related stuff on that hardware. The second nvme I kept open for the possibility of trying/messing around with the unraid setup again. Is there anyone who could guide me step by step regarding the GPU pass through and other USB peripherals. If I get it up and running I will document and redo the whole machine with unraid again. Many thanks Righardt Did you read the first post? Did you try the boot flag video=efifb:off as indicated? That is not in the SpaceinvaderOne guides and required for this board, hence this thread. You do not need legacy mode, uefi works fine. This is confirmed working on both Nvidia and amd cards. You should not need to pass a vbios with an amd card. Quote Link to comment
Sleepyme Posted December 6, 2019 Share Posted December 6, 2019 Thanks Skitals for bring up about this f10 bios and the amazing iommu group it comes with. I couldn't get the usb controllers (or any devices) to passthrough using the vfio-pci.cfg method. BUt i was able to do it by editing my xml file. Now i don't need to buy a usb pcie card! Yay! Now lets hope we can pass the onboard audio in the future! Quote Link to comment
dmarshman Posted December 6, 2019 Share Posted December 6, 2019 (edited) 10 hours ago, righardt.marais said: Hi, I have now unsuccessfully tried for a week to get my gigabyte x5700xt card passed through to a win10 pro vm on my trial unraid. Hardware: <snip> </snip> Many thanks Righardt If you are not already, try using the "Q35-3.1" machine, rather than the "i440fx-3.1" machine to which the "Windows 10" template defaults. That fixed it for me - [ASRock X570 Taichi, Ryzen 9 3950X and Radeon Vega VII] - in conjunction with the boot flag "video=efifb:off" outlined above. Edited December 6, 2019 by dmarshman Quote Link to comment
Sleepyme Posted December 6, 2019 Share Posted December 6, 2019 Just now, dmarshman said: If you are not already, try using the "Q35-3.1" machine, rather than the "i440fx-3.1" machine to which the "Windows 10" template defaults. That fixed it for me - [ASRock X570 Taichi, Ryzen 9 3950X and Radeon Vega VII] - in conjunction with the boot flag "video=efifb:off" outlined above. Same here but use 4.1 instead Quote Link to comment
dmarshman Posted December 6, 2019 Share Posted December 6, 2019 7 minutes ago, Sleepyme said: Same here but use 4.1 instead Are you running the 6.8.0 RC for that? Quote Link to comment
phat_cow Posted December 6, 2019 Share Posted December 6, 2019 25 minutes ago, dmarshman said: Are you running the 6.8.0 RC for that? Yes, it took me like a month to find out a proper combination of settings to get my GPU to passthrough properly. Luckily I didn't give up Quote Link to comment
righardt.marais Posted December 6, 2019 Share Posted December 6, 2019 8 hours ago, Skitals said: Did you read the first post? Did you try the boot flag video=efifb:off as indicated? That is not in the SpaceinvaderOne guides and required for this board, hence this thread. You do not need legacy mode, uefi works fine. This is confirmed working on both Nvidia and amd cards. You should not need to pass a vbios with an amd card. Hi Skitals Yeh we tried the UEFI boot and then changed it back to normal boot. I have added the boot flag video=efifb:off to the non-gui boot config and it also did not work. I would love to get this working as I have UnRaid running my media storage and its awesome ! Tx Quote Link to comment
righardt.marais Posted December 6, 2019 Share Posted December 6, 2019 2 hours ago, dmarshman said: If you are not already, try using the "Q35-3.1" machine, rather than the "i440fx-3.1" machine to which the "Windows 10" template defaults. That fixed it for me - [ASRock X570 Taichi, Ryzen 9 3950X and Radeon Vega VII] - in conjunction with the boot flag "video=efifb:off" outlined above. Hi 2 hours ago, Sleepyme said: Same here but use 4.1 instead Hi Sleepyme I initially tried the i440fv machine when you create the VM as it comes as the standard config but this did not work for my setup and a friend suggested I use the Q35 machine instead. This also did not work in the end. tx Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.