Gigabyte X570 AORUS Elite / Pro (WiFi) / Ultra Tips & Tricks


Recommended Posts

1 hour ago, Dava2k7 said:

No but it didn’t do anything inside vm do I need the vfio config file in place before doing this? Or Somin?

You need to actually read the first post in this thread. You need to make sure your unraid usb is on the 0b:00.3 USB controller, and the occulus stuff is NOT on that controller. Then you need to pass through 06:00.0, 06:00.1, and 06:00.3. This can all be done very easily with the vfio-pci config plugin. Do not attempt to do anything else until you understand what you are doing.

  • Like 1
Link to comment
  • 2 weeks later...
12 hours ago, sonic_reaction said:

@Skitals In your first post can you mention setting CSM to enable and booting via legacy rather that UEFI? I had major problems passing through a GPU until I enabled that and checked unraid to make sure I was in legacy mode and then everything worked. 

I am all UEFI, no legacy. It might come down to your GPU and which mode(s) it supports, if you are talking about an older card.

Link to comment
  • 2 weeks later...
  • 2 weeks later...

I am using a Asus tuf x570-plus motherboard but the IOMMU-groups seams to be identical. After reading the whole thread I have placed the unraid usb on 

the 0b:00.3 USB controller. I have also passed 06:00.1, and 06:00.3 controllers using the VIFO-PCI plugin.

My Windows 10 VM is using 06:00.1 and it works perfectly. But I have not got my other VM's to be able to use the 06:00.3 controller as nothing happens when I plug anything in the ports. I am not sure which ports is connected to 06:00.3 but I have tried all ports on my unraid server.

Or is it connected to the single USB-C connector ?

 

Do I also need to pass through:

Group 26 06:00.0 1022:1485 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

 

Edit: passing through Group 26 did not change anything. When looking in the log for the other VM I get this:

2020-03-11T16:02:11.664422Z qemu-system-x86_64: vfio: Cannot reset device 0000:06:00.0, depends on group 27 which is not owned.

 

aa.PNG

Edited by Cliff
Link to comment
  • 2 weeks later...

Hi all i have same problem with 3900x and MSI B450 tomahawk max

now i have 2 gpu card (i want 1 vm use 1 gpu card ,GTX660 and GTX650)

i try edit video=efifb:off and watch Spaceinvader One "How to passthrough an nvidia gpu as primary gpu in unRAID"

to enter command to use dump file but still can't use gpu card

now i should try which part to use the gpu card.....thank you all

Sorry my bad english....

Link to comment
1 hour ago, david279 said:

Are those GPUs uefi compatible? If not good luck passing them through, probably won't work.

How to check GPUs uefi compatible? i think is compatible because i use GTX650 can passthrough with old computer

this is my PCI Devices and IOMMU Groups

Quote

IOMMU group 0:[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 1:[1022:1483] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 2:[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 3:[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 4:[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 5:[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 6:[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 7:[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 8:[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 9:[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 10:[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 11:[1022:1484] 00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 12:[1022:1484] 00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 13:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 14:[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0

[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1

[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2

[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3

[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4

[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5

[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6

[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7

IOMMU group 15:[1022:43d5] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01)

IOMMU group 16:[1022:43c8] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)

IOMMU group 17:[1022:43c6] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)

IOMMU group 18:[1022:43c7] 20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

IOMMU group 19:[1022:43c7] 20:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

IOMMU group 20:[1022:43c7] 20:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

IOMMU group 21:[10ec:8168] 22:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

IOMMU group 22:[10de:0fc6] 25:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GTX 650] (rev a1)

IOMMU group 23:[10de:0e1b] 25:00.1 Audio device: NVIDIA Corporation GK107 HDMI Audio Controller (rev a1)

IOMMU group 24:[10de:11c0] 26:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1)

IOMMU group 25:[10de:0e0b] 26:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)

IOMMU group 26:[1022:148a] 27:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 27:[1022:1485] 28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 28:[1022:1486] 28:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP

IOMMU group 29:[1022:149c] 28:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 30:[1022:1487] 28:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

IOMMU group 31:[1022:7901] 30:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 32:[1022:7901] 31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

this is my vm xml

Quote

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>d2d767e0-926c-26df-82d7-a49f70260e5c</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/d2d767e0-926c-26df-82d7-a49f70260e5c_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows 10 + Office 2019 20200201.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.173.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='7' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:a2:1f:63'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x26' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/appdata/GTX660.dump'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>
 

My system info

Quote

Model: Custom

M/B: Micro-Star International Co., Ltd B450 TOMAHAWK MAX (MS-7C02) Version 1.0 - s/n: JC16755566

BIOS: American Megatrends Inc. Version 3.50. Dated: 11/07/2019

CPU: AMD Ryzen 9 3900X 12-Core @ 3800 MHz

HVM: Enabled

IOMMU: Enabled

Cache: 768 KiB, 6144 KiB, 65536 KiB

Memory: 32 GiB DDR4 (max. installable capacity 128 GiB)

Network: bond0: fault-tolerance (active-backup), mtu 1500
 eth0: 1000 Mbps, full duplex, mtu 1500

Kernel: Linux 5.5.8-Unraid x86_64

OpenSSL: 1.1.1d

 

Thank you all

Link to comment
15 minutes ago, david279 said:

Any reason you did not passthrough the GPU video and sound component? All i see is the video part. Have you tried passing through both?

Same.....also can't passthrough the GPU now i use q35-4.2 is can't see GPU card just show is unknow hardware,

but i remember if use i440FX is can see the GPU Card but error 43

this problem i try 7 days......

Edited by eddy0616
Link to comment
3 minutes ago, eddy0616 said:

Same.....also can't passthrough the GPU now i use q35-4.2 is can't see GPU card just show is unknow hardware,

but i remember if use i440FX is can see the GPU Card but error 43

this problem i try 7 days......

Try to make a linux vm to see if its a windows issue. Try ubuntu or something...

 

Also make sure you windows iso is up to date,

Edited by david279
Link to comment
14 hours ago, david279 said:

Try to make a linux vm to see if its a windows issue. Try ubuntu or something...

 

Also make sure you windows iso is up to date,

Edited 14 hours ago by david279

i try ubuntu is can show the video use to gpu card,it mean is my win10 problem ?

Link to comment
3 hours ago, eddy0616 said:

error 43.

Code 43 is a generic catch-all for a problem, and doesn't specifically refer to anything in particular.  Last time I had it was with a brand-new 1050.  RMA'd it for a different card and the replacement worked right out of the box.  Mind you, the original card did work correctly in a Windows box.

 

Another time, I was getting it, I got around it by simply installing the nVidia software manually instead of going through Windows Update to have it set up the driver.

Link to comment
50 minutes ago, Squid said:

Code 43 is a generic catch-all for a problem, and doesn't specifically refer to anything in particular.  Last time I had it was with a brand-new 1050.  RMA'd it for a different card and the replacement worked right out of the box.  Mind you, the original card did work correctly in a Windows box.

 

Another time, I was getting it, I got around it by simply installing the nVidia software manually instead of going through Windows Update to have it set up the driver.

Thank you you reply,i install a new win10 and find nVidia Software drives,but still same problem,my Win 10 now is use Q35,OVMF......

Link to comment
  • 2 months later...

Thanks for this information.

I have the same set up and also is working great, I have noticed though that When you try restart a vm it goes into a paused state and needs to be force stopped and restarted.

Logs show gpu related issue vfio_err_notifier_handler(0000:0a:00.1) Unrecoverable error detected

I have an Rx580 passed though.

Any help would be amazing.

Edited by steveBBB
Error
Link to comment
  • 2 months later...
On 12/18/2019 at 10:02 AM, Skitals said:

No experience with aida64, but here are my results (aida64 v6.20.5300)

Memory Read: 53295 MB/s

Memory Write: 44206 MB/s

Memory Copy: 51345 MB/s

Memory Latency: 87.4 ns

 

Edit: I had noticed high cpu usage while the guest report 1%, changing this from no to yes helped a bit and brough latency "down" to 83.3 ns. Still seems a bit high, yes?

 

Edit 2: Okay, I got memory latency "down" to 79.0ns AND reduced my idle cpu usage to practically 0% by making these two changes:

Switched the usb controller to 3.0 nec.

Turned hpet back OFF and added these hyperv flags:

Here are the new results:

 

Memory Read: 56165 MB/s

Memory Write: 44619 MB/s

Memory Copy: 52043 MB/s

Memory Latency: 79.0 ns

Hey!

I have a rather similar setup: 3900x, Gigabyte X570 Pro Wifi, 64GB Ram 3200MHz, RTX 2070Super & 1070ti.

I am passing through 10 Cores (leaving the first two across for Unraid) and both GPUs to a windows VM. 

My performance is rather good but at time feels a bit sluggish (very hard to put into numbers). After Effects/Lightroom and some other Adobe software especially feels quite sluggish.

 

I ran AIDA in trial mode and got result like you in your first test. Tried to implement your hyperv flags but not sure if I have done it right.

I just added them into the XML under hyperv

 

<hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
	  <vpindex state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
    </hyperv>

Not sure If thats correct.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.