AMD Ryzen Threadripper PRO 3955WX - VM Problems (Unable to power on device, stuck in D3)

Putschgi

By Putschgi
in VM Templates

Recommended Posts

Putschgi Noob

Putschgi

Posted
Posted

Hi guys

I updated my gaming machine with a threadripper, unfortunately the GPU passthrough from the 1st GFK is no longer possible ... can someone help me?

 

Specs:

M/B: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI Version Rev 1.xx - s/n: 210483465700039

BIOS: American Megatrends Inc. Version 0405. Dated: 03/17/2021

CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores @ 3900 MHz

HVM: Enabled

IOMMU: Enabled

Cache: 1 MB, 8 MB, 64 MB

Memory: 128 GiB DDR4 (max. installable capacity 512 GiB)

GFK: 4x RTX 2060 EVO and 1x GT 710

 

use case: A 4 player couch gaming machine like an X-Box The GT 710 GFK combines the 4 images into one via NDI streams.

 

Problem: The top GFK (slot 1) installed on the MB does not want to start since the change from an i9-9900K

 

Error message on the VM Log:

pci,host=0000:41:00.0,id=hostdev0,bus=pci.0,addr=0x6: Failed to mmap 0000:41:00.0 BAR 1. Performance may be slow

2021-06-23T16:07:48.918447Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

 

XML: 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Game_1</name>
  <uuid>c6a0f7ea-92a5-9e75-c7f8-f127a7994073</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>24641536</memory>
  <currentMemory unit='KiB'>24641536</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='16'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='17'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='18'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/c6a0f7ea-92a5-9e75-c7f8-f127a7994073_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='3' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Game_1/vdisk.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Game_1/vdisk2.img'/>
      <target dev='hdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:15:0a:5f'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='de-ch'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/218088.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

i Have also atached the Diagnostics and the Syslog (with cleaning it overfilled)

therealbeast-diagnostics-20210623-1757.zip syslog

Link to comment
  • 4 weeks later...
DotBowder Noob

DotBowder

Posted
Posted

Hello, I am experiencing the same issue, but with proxmox as a hypervisor instead of unraid.

 

I posted this issue on the level1techs forum, https://forum.level1techs.com/t/vfio-passthrough-issues-threadripper-pro-asus-wrx80e-sage/173542

 

I have updated BIOS & BMC with no luck. I do not yet know of a solution to this.

 

---

 

Something worthy of note: When I attempt to pass a graphics card through a slot 2-7 that has been bifurcated to 4x4x4x4x, but my graphics card is touching all 16x pins, I get the same error.  I verified in my BIOS the top slot is not bifurcated, so I'm wondering if there's some oopsies in the firmware regarding bifurcation of that top slot.

 

Link to comment
  • 2 weeks later...
frodr Enthusiast

frodr

Posted
Posted (edited)
On 6/23/2021 at 8:08 PM, Putschgi said:

Hi guys

I updated my gaming machine with a threadripper, unfortunately the GPU passthrough from the 1st GFK is no longer possible ... can someone help me?

 

Specs:

M/B: ASUSTeK COMPUTER INC. Pro WS WRX80E-SAGE SE WIFI Version Rev 1.xx - s/n: 210483465700039

BIOS: American Megatrends Inc. Version 0405. Dated: 03/17/2021

CPU: AMD Ryzen Threadripper PRO 3955WX 16-Cores @ 3900 MHz

HVM: Enabled

IOMMU: Enabled

Cache: 1 MB, 8 MB, 64 MB

Memory: 128 GiB DDR4 (max. installable capacity 512 GiB)

GFK: 4x RTX 2060 EVO and 1x GT 710

 

use case: A 4 player couch gaming machine like an X-Box The GT 710 GFK combines the 4 images into one via NDI streams.

 

Problem: The top GFK (slot 1) installed on the MB does not want to start since the change from an i9-9900K

 

Error message on the VM Log:

pci,host=0000:41:00.0,id=hostdev0,bus=pci.0,addr=0x6: Failed to mmap 0000:41:00.0 BAR 1. Performance may be slow

2021-06-23T16:07:48.918447Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

 

XML: 



<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Game_1</name>
  <uuid>c6a0f7ea-92a5-9e75-c7f8-f127a7994073</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>24641536</memory>
  <currentMemory unit='KiB'>24641536</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='16'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='17'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='18'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/c6a0f7ea-92a5-9e75-c7f8-f127a7994073_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='3' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Game_1/vdisk.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Game_1/vdisk2.img'/>
      <target dev='hdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:15:0a:5f'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='de-ch'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/218088.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

i Have also atached the Diagnostics and the Syslog (with cleaning it overfilled)

therealbeast-diagnostics-20210623-1757.zip 92.34 kB · 1 download syslog 421.9 kB · 1 download

 

 

Bios 0602 is out. I am running RTX 2080-ti as second GPU on a Win 10 VM, that works fine. 

Edited by frodr
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing