January 3, 20233 yr Hello, Im trying to create empty VM that uses passed through NVMe that has windows 11 on it, but i get this error internal error: qemu unexpectedly closed the monitor: 2023-01-03T01:17:01.313315Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:02.0","id":"hostdev0","bus":"pci.0","addr":"0x2"}: Failed to mmap 0000:00:02.0 BAR 2. Performance may be slow 2023-01-03T01:17:01.313537Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:02.0","id":"hostdev0","bus":"pci.0","addr":"0x2"}: IGD device 0000:00:02.0 has no ROM, legacy mode disabled 2023-01-03T01:17:01.626909Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:02:00.0","id":"hostdev1","bus":"pci.0","addr":"0x6"}: vfio 0000:02:00.0: failed to add PCI capability 0x11[0x50]@0xb0: table & pba overlap, or they don't fit in BARs, or don't align Edited January 3, 20233 yr by Hexenhammer
January 3, 20233 yr Not sure you are on the proper sub-forum. Here is for running a virtualized UNRAID on any plateform EXSI, OMV, Unraid, etc. You seem to want to run a Windows VM on Unraid ?
January 3, 20233 yr Author 9 hours ago, ChatNoir said: Not sure you are on the proper sub-forum. Here is for running a virtualized UNRAID on any plateform EXSI, OMV, Unraid, etc. You seem to want to run a Windows VM on Unraid ? Yep, is there a different sub-forum for that?
January 3, 20233 yr 9 minutes ago, Hexenhammer said: is there a different sub-forum for that? There is, but I've moved your thread earlier to the correct subforum.
January 3, 20233 yr Community Expert 14 hours ago, Hexenhammer said: Guys any idea whats going on? Is this a bug? There is this old kernel bug which Limetech has been involved with. Looks like a firmeware issue with the nvme controller. https://bugzilla.kernel.org/show_bug.cgi?id=202055#c43 Try this workaround to see if that fixes. Also unlikely to get iGPU to work in windows, I have 126000K UHD770 works ok in linux but not currently in Windows. <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> > ... > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> > </source> > <alias name='ua-sm2262'/> > <address type='pci' domain='0x0000' bus='0x02' slot='0x00' > function='0x0'/> > </hostdev> > ... > <qemu:commandline> > <qemu:arg value='-set'/> > <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/> > </qemu:commandline> > > > (NB: "ua-" is a required prefix when specifying an alias) > > A new virtual BAR appears in the guest hosting the MSI-X table and QEMU > starts normally so long as the guest doesn't exceed 15 vCPUs. > > The vCPU/pCPU count limitations are obviously not ideal, but hopefully this > provides some degree of workaround for typical configurations.
January 3, 20233 yr Author 6 hours ago, JorgeB said: Try without passing-through the iGPU, use VNC. I tested with GPU, without GPU With PCie, without PCIE
January 3, 20233 yr Author 2 minutes ago, SimonF said: There is this old kernel bug which Limetech has been involved with. Looks like a firmeware issue with the nvme controller. https://bugzilla.kernel.org/show_bug.cgi?id=202055#c43 Try this workaround to see if that fixes. Also unlikely to get iGPU to work in windows, I have 126000K UHD770 works ok in linux but not currently in Windows. <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> > ... > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> > </source> > <alias name='ua-sm2262'/> > <address type='pci' domain='0x0000' bus='0x02' slot='0x00' > function='0x0'/> > </hostdev> > ... > <qemu:commandline> > <qemu:arg value='-set'/> > <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/> > </qemu:commandline> > > > (NB: "ua-" is a required prefix when specifying an alias) > > A new virtual BAR appears in the guest hosting the MSI-X table and QEMU > starts normally so long as the guest doesn't exceed 15 vCPUs. > > The vCPU/pCPU count limitations are obviously not ideal, but hopefully this > provides some degree of workaround for typical configurations. Hi, where do i copy this to? Thanks
January 3, 20233 yr Community Expert 29 minutes ago, Hexenhammer said: Hi, where do i copy this to? Thanks Edit the vm and change to XML view. Find the NVME drive mine is bus 0x05. yours will be 0x02 <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain> to add alias and qemu lines. You will loose the changes if you make any updates for this VM via the GUI. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <alias name='ua-sm2262'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/> </qemu:commandline> </domain>
January 3, 20233 yr Author 11 minutes ago, SimonF said: Edit the vm and change to XML view. Find the NVME drive mine is bus 0x05. yours will be 0x02 <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain> to add alias and qemu lines. You will loose the changes if you make any updates for this VM via the GUI. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <alias name='ua-sm2262'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/> </qemu:commandline> </domain> Getting this error system-x86_64: -set device.ua-sm2262.x-msix-relocation=bar2: there is no device "ua-sm2262" defined and if i chnage bus to 0x02 then this Device 0000:02:06.0 not found: could not access /sys/bus/pci/devices/0000:02:06.0/config: No such file or directory <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows 11</name> <uuid>bf15b8e2-225d-d84f-2b67-00d3e3e61b20</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>9437184</memory> <currentMemory unit='KiB'>9437184</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='7'/> <vcpupin vcpu='4' cpuset='8'/> <vcpupin vcpu='5' cpuset='9'/> <vcpupin vcpu='6' cpuset='10'/> <vcpupin vcpu='7' cpuset='11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/bf15b8e2-225d-d84f-2b67-00d3e3e61b20_VARS-pure-efi-tpm.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='8' threads='1'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='pci' index='1' model='pci-bridge'> <model name='pci-bridge'/> <target chassisNr='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <model name='pci-bridge'/> <target chassisNr='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:b2:bb:98'/> <source bridge='virbr0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> </tpm> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </source> <alias name='ua-sm2262'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/> </qemu:commandline> </domain>
January 3, 20233 yr Community Expert 17 hours ago, Hexenhammer said: 0000:02:00.0 you source should this not with 0x06 <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
January 3, 20233 yr Author 14 minutes ago, SimonF said: you source should this not with 0x06 <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> Thanks for help, this is rather embarrassing asking so many questions. Now i get internal error: process exited while connecting to monitor: qemu-system-x86_64: -set device.ua-sm2262.x-msix-relocation=bar2: there is no device "ua-sm2262" defined <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <alias name='ua-sm2262'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/> </qemu:commandline> </domain> P.S. You looked for a fix for iGPU in windows VM?, I found this and enabled it, so far no iGPU errors, but i cant start the VM due to NVMe https://forums.unraid.net/topic/99478-solved-gpu-passthrough-issue-bar-0-cant-reserve/ add the following code to the /flash/config/go file: #!/bin/bash #fix video for VM echo 0 > /sys/class/vtconsole/vtcon0/bind echo 0 > /sys/class/vtconsole/vtcon1/bind echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind Those lines unbind the console & by adding it to /flash/config/go will execute this piece of code on every boot of the system. Edited January 3, 20233 yr by Hexenhammer
January 3, 20233 yr Community Expert 49 minutes ago, Hexenhammer said: Thanks for help, this is rather embarrassing asking so many questions. Now i get internal error: process exited while connecting to monitor: qemu-system-x86_64: -set device.ua-sm2262.x-msix-relocation=bar2: there is no device "ua-sm2262" defined Not a problem, not sure why the Alias is not working will have to investigate. Patch work around was a few years back so may have changed. Are you able to update the firmware on the nvme? re iGPU it passes through ok, just the driver cannot find it correctly code 43. its is bound to vfio so should have the binding but will try to see if that makes a difference.
January 3, 20233 yr Which version of unraid? You need to follow Simon advice but depending on the libvirt version the qemu:arg setting may not work, libvirt removed -set and added other workarounds for this. This issue is related to your nvme firmware, as already stated. You can find other discussions in this forum related to this issue, with possible fixes depending on the libvirt version. If you don't find them just write your unraid version.
January 3, 20233 yr Community Expert 12 minutes ago, ghost82 said: Which version of unraid? You need to follow Simon advice but depending on the libvirt version the qemu:arg setting may not work, libvirt removed -set and added other workarounds for this. This issue is related to your nvme firmware, as already stated. You can find other discussions in this forum related to this issue, with possible fixes depending on the libvirt version. If you don't find them just write your unraid version. I think 6.11.5 looking at the template. so will be qemu 7.1
January 3, 20233 yr ok, good catch, then I'm quite sure the -set option wont work. I'm on mobile now if a possible solution wont be posted I'll write it tomorrow. Edited January 3, 20233 yr by ghost82
January 3, 20233 yr Author 41 minutes ago, ghost82 said: Which version of unraid? You need to follow Simon advice but depending on the libvirt version the qemu:arg setting may not work, libvirt removed -set and added other workarounds for this. This issue is related to your nvme firmware, as already stated. You can find other discussions in this forum related to this issue, with possible fixes depending on the libvirt version. If you don't find them just write your unraid version. How do i know which version i have? Im on latest unRaid, fresh install Version 6.11.5 2022-11-20 are you talking about Machine drop down menu? I tested q35-7.1 and i440fx-7.1 Edited January 3, 20233 yr by Hexenhammer
January 3, 20233 yr Community Expert 21 minutes ago, ghost82 said: ok, good catch, then I'm quite sure the -set option wont work. I'm on mobile now if a possible solution wont be posted I'll write it tomorrow. found your post. <qemu:override> <qemu:device alias='YOURNVMEALIAS'> <qemu:frontend> <qemu:property name='x-msix-relocation' type='string' value='bar2'/> </qemu:frontend> </qemu:device> </qemu:override> Edited January 3, 20233 yr by SimonF
January 3, 20233 yr Community Expert 6 minutes ago, Hexenhammer said: How do i know which version i have? Im on latest unRaid, fresh install Version 6.11.5 2022-11-20 are you talking about Machine drop down menu? I tested q35-7.1 and i440fx-7.1 no qemu vers and libvirt vers. can be seen in vm settings and turn on advance 6.11.5 is qemu 7.1 libvirt 8.7 Edited January 3, 20233 yr by SimonF
January 3, 20233 yr Author 5 minutes ago, SimonF said: no qemu vers and libvirt vers. can be seen in vm settings and turn on advance 6.11.5 is qemu 7.1 libvirt 8.7 Off topic, but By chance do you know how to see LSI HBA controller temp? When i was messing with VMs, suddenly my controller disappeared from unraid and 8 HDDs gone [everything came back after reboot], i assumed maybe it was because i messed with PCie setting in VM and it somehow kicked out the card? Or it may be high temps [it has a fan attached 4700rpm 40mm] Edited January 3, 20233 yr by Hexenhammer
January 3, 20233 yr Community Expert 5 minutes ago, Hexenhammer said: Off topic, but By chance do you know how to see LSI HBA controller temp? When i was messing with VMs, suddenly my controller disappeared from unraid and 8 HDDs gone [everything came back after reboot], i assumed maybe it was because i messed with PCie setting in VM and it somehow kicked out the card? Or it may be high temps [it has a fan attached 4700rpm 40mm] I don't think hbas have temp sensors.
January 3, 20233 yr Author 3 minutes ago, SimonF said: I don't think hbas have temp sensors. apparently there is, but its linux command, storcli /c0 show all | grep -i temperature But unraid has no storcli https://forums.servethehome.com/index.php?threads/lsi-9206-16e-reading-temp-in-linux.29201/ EDIT: After flashing to IT and making it a HBA these tools no longer work. I think the IT-firmware also lacks functions to read the actual temperature. Sas2ircu can be used to query the IT-firmware but theres not much information to display since most functions are not available in that firmware. Edited January 3, 20233 yr by Hexenhammer
January 3, 20233 yr Community Expert 3 minutes ago, Hexenhammer said: apparently there is, but its linux command, I storcli /c0 show all | grep -i temperature But unraid has no storcli https://forums.servethehome.com/index.php?threads/lsi-9206-16e-reading-temp-in-linux.29201/ if you read to the bottom does not work with it mode only ir.
January 4, 20233 yr Solution 11 hours ago, SimonF said: found your post. <qemu:override> <qemu:device alias='YOURNVMEALIAS'> <qemu:frontend> <qemu:property name='x-msix-relocation' type='string' value='bar2'/> </qemu:frontend> </qemu:device> </qemu:override> Exactly, try this, copy/paste and save xml: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Windows 11</name> <uuid>bf15b8e2-225d-d84f-2b67-00d3e3e61b20</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/> </metadata> <memory unit='KiB'>9437184</memory> <currentMemory unit='KiB'>9437184</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='7'/> <vcpupin vcpu='4' cpuset='8'/> <vcpupin vcpu='5' cpuset='9'/> <vcpupin vcpu='6' cpuset='10'/> <vcpupin vcpu='7' cpuset='11'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader> <nvram>/etc/libvirt/qemu/nvram/bf15b8e2-225d-d84f-2b67-00d3e3e61b20_VARS-pure-efi-tpm.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='8' threads='1'/> <cache mode='passthrough'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:b2:bb:98'/> <source bridge='virbr0'/> <model type='virtio-net'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-gb'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <tpm model='tpm-tis'> <backend type='emulator' version='2.0' persistent_state='yes'/> </tpm> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </source> <boot order='1'/> <alias name='ua-sm2262'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:override> <qemu:device alias='ua-sm2262'> <qemu:frontend> <qemu:property name='x-msix-relocation' type='string' value='bar2'/> </qemu:frontend> </qemu:device> </qemu:override> </domain> If it doesn't work attach diagnostics. If part of the code is stripped it could be an unraid issue. You didn't have any graphics defined in the xml, I added basic qxl + vnc, so connect to the vm with builtin novnc. Once and if you fix the nvme issue you can try to passthrough the igpu.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.