• [6.10.rc7] No login web UI after trying to start new VM 11?


    casperse
    • Minor

    "Reproduceable problem"

    First let me say I don't know if this is specific to this release? (VM for Windows 11 is new feature)

    So please move it if you feel it shouldn't be here!

     

    I have a problem where if I try to start my new VM windows 11 setup

    Then the UI freezes and I cannot login to UNRAID anymore (From anywher!)

     

    BUT all my existing Dockers and VM's are running 100% just fine! so RAM/CPU must have enough resources available!

    Even my other VM and AMP gaming server runs perfect!

     

    My only option is to SSH in to Unraid and do a:

    powerdown -r

     

    And Fortunately I didn't have auto start on this VM 🙂

    And afterwards I can login again

     

    I can reproduce the problem easily, I have tried removing cores & adding less RAM?

     

    So if you have any special SSH commands to get the diagnostic file or other logs I should do to trouble shoot this then let me know!

     

    MY XML from the Windows 11 VM

    (If I have never seen a VM totally stop the entire access to the UI before? not even the login works)

    image.thumb.png.33a1f83f3971bb59d8c987026db28b73.png

     

     

     




    User Feedback

    Recommended Comments

    Testing different cards and Bios dumps that worked before now have it not freezing the entire Unraid but get this error message (Normally seen on AMD but I am running Xeon?)


    2022-05-13T14:47:31.447653Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

     

    -smp 8,sockets=1,dies=1,cores=4,threads=2 \
    -uuid f29f275c-5941-8a32-f3de-fb21812cbab9 \
    -display none \
    -no-user-config \
    -nodefaults \
    -chardev socket,id=charmonitor,fd=49,server=on,wait=off \
    -mon chardev=charmonitor,id=monitor,mode=control \
    -rtc base=localtime \
    -no-hpet \
    -no-shutdown \
    -boot strict=on \
    -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 \
    -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 \
    -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x7.0x1 \
    -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x7.0x2 \
    -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 \
    -blockdev '{"driver":"file","filename":"/mnt/cache_shares/game-server/vdisk.qcow2","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
    -blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
    -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=libvirt-3-format,id=virtio-disk2,bootindex=1,write-cache=on \
    -blockdev '{"driver":"file","filename":"/mnt/user/isos/Win11_Danish_x64v1.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
    -blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
    -device ide-cd,bus=ide.0,unit=0,drive=libvirt-2-format,id=ide0-0-0,bootindex=2 \
    -blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.217-1.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
    -blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
    -device ide-cd,bus=ide.0,unit=1,drive=libvirt-1-format,id=ide0-0-1 \
    -netdev tap,fd=50,id=hostnet0 \
    -device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:f9:6f:ba,bus=pci.0,addr=0x2 \
    -chardev pty,id=charserial0 \
    -device isa-serial,chardev=charserial0,id=serial0,index=0 \
    -chardev socket,id=charchannel0,fd=48,server=on,wait=off \
    -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
    -chardev 'socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/6-Windows 11 GAMER-VM--swtpm.sock' \
    -tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
    -device tpm-tis,tpmdev=tpm-tpm0,id=tpm0 \
    -device usb-tablet,id=input0,bus=usb.0,port=1 \
    -audiodev '{"id":"audio1","driver":"none"}' \
    -device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/cache_shares/game-server/Bios_dumps/ASUS_DUAL_RTX3060_12G_V2_GA104.rom \
    -device vfio-pci,host=0000:01:00.1,id=hostdev1,bus=pci.0,addr=0x6 \
    -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
    -msg timestamp=on
    char device redirected to /dev/pts/5 (label charserial0)
    2022-05-13T14:47:31.447557Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
    2022-05-13T14:47:31.447653Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

     

    The card is selected to VFIO

    image.thumb.png.45c23bdd921dde1d5732bbc584090fa9.png

     

    Link to comment

    After some more experimenting I found that this total freeze is only happening if I try to start the VM Windows 11

    With the passthrough graphic card! if I remove it and select VNC the VM starts without freezing the Unraid web UI

    (And I don't have to SSH in and do a powerdown -r)

     

    I have also done a Bios dump of the actual card

    I also made sure that the card is not the one selected in the Bios

    I also tried a different card whit a bios dump that worked before?

     

    Then after following these changes to the XML:

    image.png.a981b7c4b7120d02636abfbc86b7253a.png

     

    So finally got to the Windows 11 and I can see that the GPU is in state 43

    image.png.9e6bee024094f77477af5a9f772d33ba.png

    I managed to install the Nvidia drivers (Because the card was listed) and after a reboot I am back to only have the remote display adapter 

     

    image.png.3933a257e338148cb51644260fa38f97.png

     

    Any input on how to fix this?

    Is it just that the card isn't supported? 3060 card

     

    Link to comment

    Seems like a misconfiguration, did you attach to vfio the gpu, both audio and video part? stuck in d3 suggests it couldn't be resetted, maybe because of the audio part.

    Moreover it is always suggested to attach also the audio part of the gpu to the vm when you passthrough the video (and any other component of the gpu, like usb controller) so the drivers of the gpu can work as expected.

    Once logged in ssh type:

    diagnostics

    the zipped file will be saved into /boot/logs folder.

     

    Ps: with recent nvidia drivers you don't need to hide the hypervisor. Code 43 does not always mean that the nvidia drivers are not loading because the gpu is inside a vm, it simply means that the device cannot be started for whatever reason.

    Edited by ghost82
    Link to comment
    32 minutes ago, ghost82 said:

    Seems like a misconfiguration, did you attach to vfio the gpu, both audio and video part? stuck in d3 suggests it couldn't be resetted, maybe because of the audio part.

    Moreover it is always suggested to attach also the audio part of the gpu to the vm when you passthrough the video (and any other component of the gpu, like usb controller) so the drivers of the gpu can work as expected.

    Once logged in ssh type:

    diagnostics

    the zipped file will be saved into /boot/logs folder.

     

    Ps: with recent nvidia drivers you don't need to hide the hypervisor. Code 43 does not always mean that the nvidia drivers are not loading because the gpu is inside a vm, it simply means that the device cannot be started for whatever reason.

     

    Attach vfio like this:

    image.thumb.png.06af841ccaf4c1c403350102d6565a6c.png

     

    Yes GPU + Audio like this:

    image.thumb.png.0dc13caba0008f6b4db094ba35983906.png

     

    And the SSH diagnostic attached below

     

    And my XML file:

    <?xml version='1.0' encoding='UTF-8'?>
    <domain type='kvm'>
      <name>Windows 11 GAMER-VM-PC</name>
      <uuid>f29f275c-5941-8a32-f3de-fb21812cbab9</uuid>
      <description>Windows 11 Pro 64 DK GAMERPC</description>
      <metadata>
        <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
      </metadata>
      <memory unit='KiB'>8388608</memory>
      <currentMemory unit='KiB'>8388608</currentMemory>
      <memoryBacking>
        <nosharepages/>
      </memoryBacking>
      <vcpu placement='static'>8</vcpu>
      <cputune>
        <vcpupin vcpu='0' cpuset='2'/>
        <vcpupin vcpu='1' cpuset='8'/>
        <vcpupin vcpu='2' cpuset='3'/>
        <vcpupin vcpu='3' cpuset='9'/>
        <vcpupin vcpu='4' cpuset='4'/>
        <vcpupin vcpu='5' cpuset='10'/>
        <vcpupin vcpu='6' cpuset='5'/>
        <vcpupin vcpu='7' cpuset='11'/>
      </cputune>
      <os>
        <type arch='x86_64' machine='pc-i440fx-6.2'>hvm</type>
        <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
        <nvram>/etc/libvirt/qemu/nvram/f29f275c-5941-8a32-f3de-fb21812cbab9_VARS-pure-efi-tpm.fd</nvram>
      </os>
      <features>
        <acpi/>
        <apic/>
        <hyperv mode='custom'>
          <relaxed state='on'/>
          <vapic state='on'/>
          <spinlocks state='on' retries='8191'/>
          <vendor_id state='on' value='2D77A8B352F1'/>
        </hyperv>
        <kvm>
          <hidden state='on'/>
        </kvm>
        <vmport state='off'/>
        <ioapic driver='kvm'/>
      </features>
      <cpu mode='host-passthrough' check='partial' migratable='on'>
        <topology sockets='1' dies='1' cores='4' threads='2'/>
        <cache mode='passthrough'/>
      </cpu>
      <clock offset='localtime'>
        <timer name='hypervclock' present='yes'/>
        <timer name='hpet' present='no'/>
      </clock>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>restart</on_crash>
      <devices>
        <emulator>/usr/local/sbin/qemu</emulator>
        <disk type='file' device='disk'>
          <driver name='qemu' type='qcow2' cache='writeback'/>
          <source file='/mnt/cache_shares/game-server/vdisk.qcow2'/>
          <target dev='hdc' bus='virtio'/>
          <boot order='1'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
        </disk>
        <disk type='file' device='cdrom'>
          <driver name='qemu' type='raw'/>
          <source file='/mnt/user/isos/Win11_Danish_x64v1.iso'/>
          <target dev='hda' bus='ide'/>
          <readonly/>
          <boot order='2'/>
          <address type='drive' controller='0' bus='0' target='0' unit='0'/>
        </disk>
        <disk type='file' device='cdrom'>
          <driver name='qemu' type='raw'/>
          <source file='/mnt/user/isos/virtio-win-0.1.217-1.iso'/>
          <target dev='hdb' bus='ide'/>
          <readonly/>
          <address type='drive' controller='0' bus='0' target='0' unit='1'/>
        </disk>
        <controller type='usb' index='0' model='ich9-ehci1'>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci1'>
          <master startport='0'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci2'>
          <master startport='2'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci3'>
          <master startport='4'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
        </controller>
        <controller type='pci' index='0' model='pci-root'/>
        <controller type='ide' index='0'>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
        </controller>
        <controller type='virtio-serial' index='0'>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
        </controller>
        <interface type='bridge'>
          <mac address='52:54:00:f9:6f:ba'/>
          <source bridge='br0'/>
          <model type='virtio-net'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
        </interface>
        <serial type='pty'>
          <target type='isa-serial' port='0'>
            <model name='isa-serial'/>
          </target>
        </serial>
        <console type='pty'>
          <target type='serial' port='0'/>
        </console>
        <channel type='unix'>
          <target type='virtio' name='org.qemu.guest_agent.0'/>
          <address type='virtio-serial' controller='0' bus='0' port='1'/>
        </channel>
        <input type='tablet' bus='usb'>
          <address type='usb' bus='0' port='1'/>
        </input>
        <input type='mouse' bus='ps2'/>
        <input type='keyboard' bus='ps2'/>
        <tpm model='tpm-tis'>
          <backend type='emulator' version='2.0' persistent_state='yes'/>
        </tpm>
        <audio id='1' type='none'/>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
          </source>
          <rom file='/mnt/cache_shares/game-server/Bios_dumps/ASUS_DUAL_RTX3060_FIXED.rom'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
        </hostdev>
        <memballoon model='virtio'>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
        </memballoon>
      </devices>
    </domain>

     

    Thanks for reaching out! - I have tried so many things! 🙂

     

    plexzone-diagnostics-20220514-2250.zip

    Link to comment

    I'm not sure I fully understood the issue, it seems you have 2 issues:

    1. unraid gui freezes when you launch the windows 11 gamer vm, but apart the gui it's working since you can ssh into it

    2. your 3060 gpu doesn't work inside the vm

     

    Is this correct?

     

    On 5/13/2022 at 4:56 PM, casperse said:

    2022-05-13T14:47:31.447653Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

    I'm not seeing anymore this error in the diagnostics you attached (stuck in d3 is usually caused by gpu in use by something else, wrong vbios).

     

    On the vm side, I could suggest to:

    1. dump your vbios and use it

    2. Use q35 machine type instead of i440fx

    3. Change layout of target gpu as multifunction (audio+video): same bus, same slot, different function with addition of multifunction='on' at the and of the '<address type' line of the video block.

     

     

    Edited by ghost82
    Link to comment
    2 hours ago, ghost82 said:

    I'm not sure I fully understood the issue, it seems you have 2 issues:

     

    1. unraid gui freezes when you launch the windows 11 gamer vm, but apart the gui it's working since you can ssh into it

    Yes I found that if I added the GPU to the VM it would freeze the entire Unraid UI only SSH would giive access

    But I now know that this only happens if I didn't do the "Bind selected IOMMU groups to VFIO at boot" 

    (Still think that it should be possible to accidently do this without a total freeze 🙃)

     

    2. your 3060 gpu doesn't work inside the vm

    Yes it seems that no matter what I do I cant get the rxt3060 to passthrough?

     

     

    1. dump your vbios and use it

    I have done that using the GPU-Z.2.46.0 - moving the card to a desktop

    I also did a fixed VBIOS removing the top with a hexeditor like some suggest (Tried booting with both VBIOS)

     

    2. Use q35 machine type instead of i440fx

    I just created a new Win 11 using Q35 (I couldnt change the existing VM I got this error:

    image.png.bdd40389543c0cec963a0d5f478de11c.png

    So I created a new VM with Q35

    <?xml version='1.0' encoding='UTF-8'?>
    <domain type='kvm' id='15'>
      <name>Windows 11</name>
      <uuid>15c7fb89-8be2-5ae0-4c84-150ebba46f87</uuid>
      <metadata>
        <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
      </metadata>
      <memory unit='KiB'>8388608</memory>
      <currentMemory unit='KiB'>8388608</currentMemory>
      <memoryBacking>
        <nosharepages/>
      </memoryBacking>
      <vcpu placement='static'>8</vcpu>
      <cputune>
        <vcpupin vcpu='0' cpuset='2'/>
        <vcpupin vcpu='1' cpuset='8'/>
        <vcpupin vcpu='2' cpuset='3'/>
        <vcpupin vcpu='3' cpuset='9'/>
        <vcpupin vcpu='4' cpuset='4'/>
        <vcpupin vcpu='5' cpuset='10'/>
        <vcpupin vcpu='6' cpuset='5'/>
        <vcpupin vcpu='7' cpuset='11'/>
      </cputune>
      <resource>
        <partition>/machine</partition>
      </resource>
      <os>
        <type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
        <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
        <nvram>/etc/libvirt/qemu/nvram/15c7fb89-8be2-5ae0-4c84-150ebba46f87_VARS-pure-efi-tpm.fd</nvram>
      </os>
      <features>
        <acpi/>
        <apic/>
        <hyperv mode='custom'>
          <relaxed state='on'/>
          <vapic state='on'/>
          <spinlocks state='on' retries='8191'/>
          <vendor_id state='on' value='1234567890ab'/>
        </hyperv>
        <kvm>
          <hidden state='on'/>
        </kvm>
        <vmport state='off'/>
        <ioapic driver='kvm'/>
      </features>
      <cpu mode='host-passthrough' check='none' migratable='on'>
        <topology sockets='1' dies='1' cores='4' threads='2'/>
        <cache mode='passthrough'/>
      </cpu>
      <clock offset='localtime'>
        <timer name='hypervclock' present='yes'/>
        <timer name='hpet' present='no'/>
      </clock>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>restart</on_crash>
      <devices>
        <emulator>/usr/local/sbin/qemu</emulator>
        <disk type='file' device='cdrom'>
          <driver name='qemu' type='raw'/>
          <source file='/mnt/user/isos/virtio-win-0.1.217-1.iso' index='2'/>
          <backingStore/>
          <target dev='hdb' bus='sata'/>
          <readonly/>
          <alias name='sata0-0-1'/>
          <address type='drive' controller='0' bus='0' target='0' unit='1'/>
        </disk>
        <disk type='file' device='disk'>
          <driver name='qemu' type='qcow2' cache='writeback'/>
          <source file='/mnt/cache_shares/game-server/vdisk.qcow2' index='1'/>
          <backingStore/>
          <target dev='hdc' bus='sata'/>
          <boot order='1'/>
          <alias name='sata0-0-2'/>
          <address type='drive' controller='0' bus='0' target='0' unit='2'/>
        </disk>
        <controller type='usb' index='0' model='ich9-ehci1'>
          <alias name='usb'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci1'>
          <alias name='usb'/>
          <master startport='0'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci2'>
          <alias name='usb'/>
          <master startport='2'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1' multifunction='on'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci3'>
          <alias name='usb'/>
          <master startport='4'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
        </controller>
        <controller type='pci' index='0' model='pcie-root'>
          <alias name='pcie.0'/>
        </controller>
        <controller type='pci' index='1' model='pcie-root-port'>
          <model name='pcie-root-port'/>
          <target chassis='1' port='0x8'/>
          <alias name='pci.1'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
        </controller>
        <controller type='pci' index='2' model='pcie-root-port'>
          <model name='pcie-root-port'/>
          <target chassis='2' port='0x9'/>
          <alias name='pci.2'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
        </controller>
        <controller type='pci' index='3' model='pcie-root-port'>
          <model name='pcie-root-port'/>
          <target chassis='3' port='0x12'/>
          <alias name='pci.3'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
        </controller>
        <controller type='pci' index='4' model='pcie-root-port'>
          <model name='pcie-root-port'/>
          <target chassis='4' port='0x13'/>
          <alias name='pci.4'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
        </controller>
        <controller type='virtio-serial' index='0'>
          <alias name='virtio-serial0'/>
          <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
        </controller>
        <controller type='sata' index='0'>
          <alias name='ide'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
        </controller>
        <interface type='bridge'>
          <mac address='52:54:00:f9:6f:ba'/>
          <source bridge='br0'/>
          <target dev='vnet20'/>
          <model type='virtio-net'/>
          <alias name='net0'/>
          <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
        </interface>
        <serial type='pty'>
          <source path='/dev/pts/5'/>
          <target type='isa-serial' port='0'>
            <model name='isa-serial'/>
          </target>
          <alias name='serial0'/>
        </serial>
        <console type='pty' tty='/dev/pts/5'>
          <source path='/dev/pts/5'/>
          <target type='serial' port='0'/>
          <alias name='serial0'/>
        </console>
        <channel type='unix'>
          <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-15-Windows 11/org.qemu.guest_agent.0'/>
          <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
          <alias name='channel0'/>
          <address type='virtio-serial' controller='0' bus='0' port='1'/>
        </channel>
        <input type='tablet' bus='usb'>
          <alias name='input0'/>
          <address type='usb' bus='0' port='1'/>
        </input>
        <input type='mouse' bus='ps2'>
          <alias name='input1'/>
        </input>
        <input type='keyboard' bus='ps2'>
          <alias name='input2'/>
        </input>
        <tpm model='tpm-tis'>
          <backend type='emulator' version='2.0' persistent_state='yes'/>
          <alias name='tpm0'/>
        </tpm>
        <audio id='1' type='none'/>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
          </source>
          <alias name='hostdev0'/>
          <rom file='/mnt/cache_shares/game-server/Bios_dumps/ASUS_DUAL_RTX3060_FIXED.rom'/>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
          </source>
          <alias name='hostdev1'/>
          <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
        </hostdev>
        <memballoon model='none'/>
      </devices>
      <seclabel type='dynamic' model='dac' relabel='yes'>
        <label>+0:+100</label>
        <imagelabel>+0:+100</imagelabel>
      </seclabel>
    </domain>

    After booting and I have the 3060 card set:

    image.thumb.png.2751d3cba6ac4e8143297599d5c6768d.png

     

    Windows 11 starts up and I don't have any card in the device manager anymore?

    image.png.1f8bfe8968b19dee649be0903f838fc9.png

     

    3. Change layout of target gpu as multifunction (audio+video): same bus, same slot, different function with addition of multifunction='on' at the and of the '<address type' line of the video block.

     

    Not sure what you mean? should I change the way the IMMOU is split here?

    image.thumb.png.277d6f0d84c4af8159c23af0d26001c4.png

    To use multifunction?

     

    I am thinking that maybe the Bios dump is wrong?

    Or the VFIO is wrong? should I input the GPU ID's here during boot?

    image.thumb.png.9d5ed23b2862cf0904066d795aa0b485.png

     

    Should I do a NEW:
    kernel /bzimage
    append vfio-pci.ids=10de:2487,10de:228b pcie_acs_override=downstream,multifunction initrd=/bzroot,/bzroot-gui pci=nommconf

     

    Thanks again for helping! (I have been troubleshooting this for over a week now!)

    Edited by casperse
    Link to comment
    28 minutes ago, casperse said:

    3. Change layout of target gpu as multifunction (audio+video): same bus, same slot, different function with addition of multifunction='on' at the and of the '<address type' line of the video block.

     

    Not sure what you mean? should I change the way the IMMOU is split here?

    Replace this:

        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
          </source>
          <alias name='hostdev0'/>
          <rom file='/mnt/cache_shares/game-server/Bios_dumps/ASUS_DUAL_RTX3060_FIXED.rom'/>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
          </source>
          <alias name='hostdev1'/>
          <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
        </hostdev>

    With this:

        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
          </source>
          <alias name='hostdev0'/>
          <rom file='/mnt/cache_shares/game-server/Bios_dumps/ASUS_DUAL_RTX3060_FIXED.rom'/>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
          </source>
          <alias name='hostdev1'/>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
        </hostdev>

     

    28 minutes ago, casperse said:

    Should I do a NEW:
    kernel /bzimage
    append vfio-pci.ids=10de:2487,10de:228b pcie_acs_override=downstream,multifunction initrd=/bzroot,/bzroot-gui pci=nommconf

    No, do not add vfio-pci.ids, vfio isalready managed.

    What do you use pci=nommconf for?Remove it, because this is for memory mapped pci configuration registers, and your gpu may need them.

    So append line will be:

    append pcie_acs_override=downstream,multifunction initrd=/bzroot,/bzroot-gui

     

    You may need to enable allow unsafe interrupts, it's a setting in unraid, if I remember well in the same page you have the Pcie acs override setting.

     

    After these changes reboot the server.

    Edited by ghost82
    • Thanks 1
    Link to comment
    1 hour ago, ghost82 said:

    Replace this:

        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
          </source>
          <alias name='hostdev0'/>
          <rom file='/mnt/cache_shares/game-server/Bios_dumps/ASUS_DUAL_RTX3060_FIXED.rom'/>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
          </source>
          <alias name='hostdev1'/>
          <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
        </hostdev>

    With this:

        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
          </source>
          <alias name='hostdev0'/>
          <rom file='/mnt/cache_shares/game-server/Bios_dumps/ASUS_DUAL_RTX3060_FIXED.rom'/>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
          </source>
          <alias name='hostdev1'/>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
        </hostdev>

     

    No, do not add vfio-pci.ids, vfio isalready managed.

    What do you use pci=nommconf for?Remove it, because this is for memory mapped pci configuration registers, and your gpu may need them.

    So append line will be:

    append pcie_acs_override=downstream,multifunction initrd=/bzroot,/bzroot-gui

     

    You may need to enable allow unsafe interrupts, it's a setting in unraid, if I remember well in the same page you have the Pcie acs override setting.

     

    After these changes reboot the server.

     

    I removed the pci=nommconf from the startup (I think this was part of my pfsense NIC passthrough guide?)

    After setting the VM to multifunction all my drives was gone when rebooting!

    image.thumb.png.394bedaa1a74ed215cceed4acd0342bc.png

    Scary stuff

    Fortunately changing it back to "Both" and rebooting the drives was back again!

    I did get the unsafe interrupts enabled!

     

    So I am back to testing passthrough again, hooping the removal of the "pci=nommconf" might have fixed it?

    Booting now

    Link to comment

    Hi Finally it works!

    (It is still a mystery why all my drives would disappear every time I tried to change my VM settings to multifunction?)

     

    Trying to re-cap what I ended up doing:

    Anyway I got passthrough working with "pc-i440fx-6.2"

    And the changes to VM settings to "allow unsafe interrupts"

    Also I did use the VBIOS dump with the HEX edited part not the full dump.

    Also think the removal of the "pci=nommconf" did have something to do with the problem

    and of course the added lines in the XML:

      <features>
        <acpi/>
        <apic/>
        <hyperv mode='custom'>
          <relaxed state='on'/>
          <vapic state='on'/>
          <spinlocks state='on' retries='8191'/>
          <vendor_id state='on' value='1234567890ab'/>
        </hyperv>
        <kvm>
          <hidden state='on'/>
        </kvm>
        <vmport state='off'/>
        <ioapic driver='kvm'/>
      </features>

     

    So my final XML file like this, working and pretty quick boot! :

    <?xml version='1.0' encoding='UTF-8'?>
    <domain type='kvm'>
      <name>Windows 11 - GAMERVMPC</name>
      <uuid>58ffcf6c-1a1d-63a5-7524-3283f15d131b</uuid>
      <description>Windows 11 Pro 64 DK GAMERPC</description>
      <metadata>
        <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
      </metadata>
      <memory unit='KiB'>8388608</memory>
      <currentMemory unit='KiB'>8388608</currentMemory>
      <memoryBacking>
        <nosharepages/>
      </memoryBacking>
      <vcpu placement='static'>8</vcpu>
      <cputune>
        <vcpupin vcpu='0' cpuset='2'/>
        <vcpupin vcpu='1' cpuset='8'/>
        <vcpupin vcpu='2' cpuset='3'/>
        <vcpupin vcpu='3' cpuset='9'/>
        <vcpupin vcpu='4' cpuset='4'/>
        <vcpupin vcpu='5' cpuset='10'/>
        <vcpupin vcpu='6' cpuset='5'/>
        <vcpupin vcpu='7' cpuset='11'/>
      </cputune>
      <os>
        <type arch='x86_64' machine='pc-i440fx-6.2'>hvm</type>
        <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
        <nvram>/etc/libvirt/qemu/nvram/58ffcf6c-1a1d-63a5-7524-3283f15d131b_VARS-pure-efi-tpm.fd</nvram>
      </os>
      <features>
        <acpi/>
        <apic/>
        <hyperv mode='custom'>
          <relaxed state='on'/>
          <vapic state='on'/>
          <spinlocks state='on' retries='8191'/>
          <vendor_id state='on' value='1234567890ab'/>
        </hyperv>
        <kvm>
          <hidden state='on'/>
        </kvm>
        <vmport state='off'/>
        <ioapic driver='kvm'/>
      </features>
      <cpu mode='host-passthrough' check='none' migratable='on'>
        <topology sockets='1' dies='1' cores='4' threads='2'/>
        <cache mode='passthrough'/>
      </cpu>
      <clock offset='localtime'>
        <timer name='hypervclock' present='yes'/>
        <timer name='hpet' present='no'/>
      </clock>
      <on_poweroff>destroy</on_poweroff>
      <on_reboot>restart</on_reboot>
      <on_crash>restart</on_crash>
      <devices>
        <emulator>/usr/local/sbin/qemu</emulator>
        <disk type='file' device='disk'>
          <driver name='qemu' type='qcow2' cache='writeback'/>
          <source file='/mnt/cache_shares/game-server/vdisk.qcow2'/>
          <target dev='hdc' bus='virtio'/>
          <boot order='1'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
        </disk>
        <disk type='file' device='cdrom'>
          <driver name='qemu' type='raw'/>
          <source file='/mnt/user/isos/Win11_Danish_x64v1.iso'/>
          <target dev='hda' bus='ide'/>
          <readonly/>
          <boot order='2'/>
          <address type='drive' controller='0' bus='0' target='0' unit='0'/>
        </disk>
        <disk type='file' device='cdrom'>
          <driver name='qemu' type='raw'/>
          <source file='/mnt/user/isos/virtio-win-0.1.217-1.iso'/>
          <target dev='hdb' bus='ide'/>
          <readonly/>
          <address type='drive' controller='0' bus='0' target='0' unit='1'/>
        </disk>
        <controller type='usb' index='0' model='ich9-ehci1'>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci1'>
          <master startport='0'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci2'>
          <master startport='2'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
        </controller>
        <controller type='usb' index='0' model='ich9-uhci3'>
          <master startport='4'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
        </controller>
        <controller type='pci' index='0' model='pci-root'/>
        <controller type='pci' index='1' model='pci-bridge'>
          <model name='pci-bridge'/>
          <target chassisNr='1'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
        </controller>
        <controller type='pci' index='2' model='pci-bridge'>
          <model name='pci-bridge'/>
          <target chassisNr='2'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
        </controller>
        <controller type='pci' index='3' model='pci-bridge'>
          <model name='pci-bridge'/>
          <target chassisNr='3'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
        </controller>
        <controller type='ide' index='0'>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
        </controller>
        <controller type='virtio-serial' index='0'>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
        </controller>
        <interface type='bridge'>
          <mac address='52:54:00:f9:6f:ba'/>
          <source bridge='br0'/>
          <model type='virtio-net'/>
          <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
        </interface>
        <serial type='pty'>
          <target type='isa-serial' port='0'>
            <model name='isa-serial'/>
          </target>
        </serial>
        <console type='pty'>
          <target type='serial' port='0'/>
        </console>
        <channel type='unix'>
          <target type='virtio' name='org.qemu.guest_agent.0'/>
          <address type='virtio-serial' controller='0' bus='0' port='1'/>
        </channel>
        <input type='tablet' bus='usb'>
          <address type='usb' bus='0' port='1'/>
        </input>
        <input type='mouse' bus='ps2'/>
        <input type='keyboard' bus='ps2'/>
        <tpm model='tpm-tis'>
          <backend type='emulator' version='2.0' persistent_state='yes'/>
        </tpm>
        <audio id='1' type='none'/>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
          </source>
          <rom file='/mnt/cache_shares/game-server/Bios_dumps/ASUS_DUAL_RTX3060_FIXED.rom'/>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0' multifunction='on'/>
        </hostdev>
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <driver name='vfio'/>
          <source>
            <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
          </source>
          <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x1'/>
        </hostdev>
        <memballoon model='none'/>
      </devices>
    </domain>

     

    I tried getting a 3Dmark benchmark but this wouldn't work on my VM

    image.png.fbb4bb50d59a65831374393ff9ac7c71.png

     

    Even with these changes it didn't work (Someone reported this as a fix)

        <hyperv mode='custom'>
          <relaxed state='on'/>
          <vapic state='on'/>
          <spinlocks state='on' retries='8191'/>
          <vendor_id state='on' value='1234567890ab'/>
          <vpindex state='on'/>
          <runtime state='on'/>
          <synic state='on'/>
          <stimer state='on'>
            <direct state='on'/>
          </stimer>
          <reset state='on'/>
          <frequencies state='on'/>
          <reenlightenment state='on'/>
          <tlbflush state='on'/>
          <ipi state='on'/>
          <evmcs state='on'/>
        </hyperv>  

     

    BIG THANKS TO @ghost82 - I would never have gotten this to work without you!

    (Hope this will help others 🙂 tried to document the changes I did)

    • Like 1
    Link to comment
    3 minutes ago, casperse said:

    I tried getting a 3Dmark benchmark but this wouldn't work on my VM

    Are you running it through a remote desktop connection or are you sit in front of your monitor connected to the passed through gpu?

    Link to comment

    So glad to hear this.  Was in the middle of researching more on this last night and had to turn in before I could figure out a solution for you.  You can probably imagine my surprise and excitement when today I see you have fixed it on your own!!

    Link to comment

    I couldn't have done this without the help from ghost82!

    And the great Unraid forum with the many great post sharing everyone's old and new VM creation solutions 

     

    Only have two things that I didn't find out /re-solved:

    1. Changing VM settings from "Both" to "Multifunction" would remove all my drives after reboot every time! (Only setting it back to "Both" would resolve it after a new reboot) - Scary stuff! 
    2. (Maybe by design?) But selecting a GPU in the dropdown list during a VM 11 creation with passthrough WITHOUT first doing the "Bind selected to VFIO at Boot" for the GPU - in the device settings would crash the entire Unraid UI every time (Only SSH would work after that)
    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.