HELP, cant create an empty VM to passthrough NVMe: VM creation error


Go to solution Solved by ghost82,

Recommended Posts

Hello,

Im trying to create empty VM that uses passed through NVMe that has windows 11 on it, but i get this error

 

 

internal error: qemu unexpectedly closed the monitor: 2023-01-03T01:17:01.313315Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:02.0","id":"hostdev0","bus":"pci.0","addr":"0x2"}: Failed to mmap 0000:00:02.0 BAR 2. Performance may be slow
2023-01-03T01:17:01.313537Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:00:02.0","id":"hostdev0","bus":"pci.0","addr":"0x2"}: IGD device 0000:00:02.0 has no ROM, legacy mode disabled
2023-01-03T01:17:01.626909Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:02:00.0","id":"hostdev1","bus":"pci.0","addr":"0x6"}: vfio 0000:02:00.0: failed to add PCI capability 0x11[0x50]@0xb0: table & pba overlap, or they don't fit in BARs, or don't align

 

 

image.thumb.png.13681a48490dd5f8761701bedd8916e8.png

 

 

Edited by Hexenhammer
Link to comment
14 hours ago, Hexenhammer said:

Guys any idea whats going on? Is this a bug?

There is this old kernel bug which Limetech has been involved with. Looks like a firmeware issue with the nvme controller.

 

https://bugzilla.kernel.org/show_bug.cgi?id=202055#c43

 

Try this workaround to see if that fixes. Also unlikely to get iGPU to work in windows, I have 126000K UHD770 works ok in linux but not currently in Windows.

 

 <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> ...
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>       </source>
>       <alias name='ua-sm2262'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> function='0x0'/>
>     </hostdev>
> ...
>   <qemu:commandline>
>     <qemu:arg value='-set'/>
>     <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
>   </qemu:commandline>
> 
> 
> (NB: "ua-" is a required prefix when specifying an alias)
> 
> A new virtual BAR appears in the guest hosting the MSI-X table and QEMU
> starts normally so long as the guest doesn't exceed 15 vCPUs.
> 
> The vCPU/pCPU count limitations are obviously not ideal, but hopefully this
> provides some degree of workaround for typical configurations.

 

  • Like 1
Link to comment
2 minutes ago, SimonF said:

There is this old kernel bug which Limetech has been involved with. Looks like a firmeware issue with the nvme controller.

 

https://bugzilla.kernel.org/show_bug.cgi?id=202055#c43

 

Try this workaround to see if that fixes. Also unlikely to get iGPU to work in windows, I have 126000K UHD770 works ok in linux but not currently in Windows.

 

 <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
> ...
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
>       </source>
>       <alias name='ua-sm2262'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> function='0x0'/>
>     </hostdev>
> ...
>   <qemu:commandline>
>     <qemu:arg value='-set'/>
>     <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
>   </qemu:commandline>
> 
> 
> (NB: "ua-" is a required prefix when specifying an alias)
> 
> A new virtual BAR appears in the guest hosting the MSI-X table and QEMU
> starts normally so long as the guest doesn't exceed 15 vCPUs.
> 
> The vCPU/pCPU count limitations are obviously not ideal, but hopefully this
> provides some degree of workaround for typical configurations.

 

 

Hi,

where do i copy this to?

 

Thanks

Link to comment
29 minutes ago, Hexenhammer said:

 

Hi,

where do i copy this to?

 

Thanks

Edit the vm and change to XML view.

 

Find the NVME drive mine is bus 0x05. yours will be 0x02

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

to add alias and qemu lines. You will loose the changes if you make any updates for this VM via the GUI.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
 
   <qemu:commandline>
     <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
   </qemu:commandline>
</domain>

 

Link to comment
11 minutes ago, SimonF said:

Edit the vm and change to XML view.

 

Find the NVME drive mine is bus 0x05. yours will be 0x02

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

to add alias and qemu lines. You will loose the changes if you make any updates for this VM via the GUI.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
 
   <qemu:commandline>
     <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
   </qemu:commandline>
</domain>

 

 

 

Getting this error

 

system-x86_64: -set device.ua-sm2262.x-msix-relocation=bar2: there is no device "ua-sm2262" defined

 

and if i chnage bus to 0x02 then this

 

Device 0000:02:06.0 not found: could not access /sys/bus/pci/devices/0000:02:06.0/config: No such file or directory

 

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows 11</name>
  <uuid>bf15b8e2-225d-d84f-2b67-00d3e3e61b20</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>9437184</memory>
  <currentMemory unit='KiB'>9437184</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='9'/>
    <vcpupin vcpu='6' cpuset='10'/>
    <vcpupin vcpu='7' cpuset='11'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/bf15b8e2-225d-d84f-2b67-00d3e3e61b20_VARS-pure-efi-tpm.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='8' threads='1'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='pci' index='1' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:b2:bb:98'/>
      <source bridge='virbr0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
  </qemu:commandline>
</domain>

 

Link to comment
14 minutes ago, SimonF said:

you source should this not with 0x06

 

<address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>

 

Thanks for help, this is rather embarrassing asking so many questions.

 

Now i get 

internal error: process exited while connecting to monitor: qemu-system-x86_64: -set device.ua-sm2262.x-msix-relocation=bar2: there is no device "ua-sm2262" defined

 

 

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
  </qemu:commandline>
</domain>

 

 

P.S. You looked for a fix for iGPU in windows VM?, I found this and enabled it, so far no iGPU errors, but i cant start the VM due to NVMe

https://forums.unraid.net/topic/99478-solved-gpu-passthrough-issue-bar-0-cant-reserve/

 

add the following code to the /flash/config/go file:

 

#!/bin/bash

#fix video for VM
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

 

Those lines unbind the console & by adding it to /flash/config/go will execute this piece of code on every boot of the system.

 


 

 

Edited by Hexenhammer
Link to comment
49 minutes ago, Hexenhammer said:

 

Thanks for help, this is rather embarrassing asking so many questions.

 

Now i get 

internal error: process exited while connecting to monitor: qemu-system-x86_64: -set device.ua-sm2262.x-msix-relocation=bar2: there is no device "ua-sm2262" defined

Not a problem, not sure why the Alias is not working will have to investigate. Patch work around was a few years back so may have changed.

 

Are you able to update the firmware on the nvme?

 

re iGPU it passes through ok, just the driver cannot find it correctly code 43. its is bound to vfio so should have the binding but will try to see if that makes a difference.

Link to comment

Which version of unraid?

You need to follow Simon advice but depending on the libvirt version the qemu:arg setting may not work, libvirt removed -set and added other workarounds for this.

This issue is related to your nvme firmware, as already stated.

You can find other discussions in this forum related to this issue, with possible fixes depending on the libvirt version.

If you don't find them just write your unraid version.

Link to comment
12 minutes ago, ghost82 said:

Which version of unraid?

You need to follow Simon advice but depending on the libvirt version the qemu:arg setting may not work, libvirt removed -set and added other workarounds for this.

This issue is related to your nvme firmware, as already stated.

You can find other discussions in this forum related to this issue, with possible fixes depending on the libvirt version.

If you don't find them just write your unraid version.

I think 6.11.5 looking at the template. so will be qemu 7.1

  • Like 1
Link to comment
41 minutes ago, ghost82 said:

Which version of unraid?

You need to follow Simon advice but depending on the libvirt version the qemu:arg setting may not work, libvirt removed -set and added other workarounds for this.

This issue is related to your nvme firmware, as already stated.

You can find other discussions in this forum related to this issue, with possible fixes depending on the libvirt version.

If you don't find them just write your unraid version.

 

How do i know which version i have?

Im on latest unRaid, fresh install Version 6.11.5 2022-11-20

 

are you talking about Machine drop down menu? I tested q35-7.1 and i440fx-7.1

Edited by Hexenhammer
Link to comment
21 minutes ago, ghost82 said:

ok, good catch, then I'm quite sure the -set option wont work. I'm on mobile now if a possible solution wont be posted I'll write it tomorrow.

found your post.

 

<qemu:override> <qemu:device alias='YOURNVMEALIAS'> <qemu:frontend> <qemu:property name='x-msix-relocation' type='string' value='bar2'/> </qemu:frontend> </qemu:device> </qemu:override>

 

 

Edited by SimonF
Link to comment
6 minutes ago, Hexenhammer said:

 

How do i know which version i have?

Im on latest unRaid, fresh install Version 6.11.5 2022-11-20

 

are you talking about Machine drop down menu? I tested q35-7.1 and i440fx-7.1

no qemu vers and libvirt vers. can be seen in vm settings and turn on advance 6.11.5 is qemu 7.1 libvirt 8.7

Edited by SimonF
Link to comment
5 minutes ago, SimonF said:

no qemu vers and libvirt vers. can be seen in vm settings and turn on advance 6.11.5 is qemu 7.1 libvirt 8.7

 

Off topic, but By chance  do you know how to see LSI HBA controller temp?

 

When i was messing with VMs, suddenly my controller disappeared from unraid and 8 HDDs gone [everything came back after reboot], i assumed maybe it was because i messed with PCie setting in VM and it somehow kicked out the card?

Or it may be high temps [it has a fan attached 4700rpm 40mm]

Edited by Hexenhammer
Link to comment
5 minutes ago, Hexenhammer said:

 

Off topic, but By chance  do you know how to see LSI HBA controller temp?

 

When i was messing with VMs, suddenly my controller disappeared from unraid and 8 HDDs gone [everything came back after reboot], i assumed maybe it was because i messed with PCie setting in VM and it somehow kicked out the card?

Or it may be high temps [it has a fan attached 4700rpm 40mm]

I don't think hbas have temp sensors.

Link to comment
3 minutes ago, SimonF said:

I don't think hbas have temp sensors.

 

apparently there is, but its linux command,

 

storcli /c0 show all | grep -i temperature

 

But unraid has no storcli

 

 

https://forums.servethehome.com/index.php?threads/lsi-9206-16e-reading-temp-in-linux.29201/

 

 

EDIT:

After flashing to IT and making it a HBA these tools no longer work.
I think the IT-firmware also lacks functions to read the actual temperature.
Sas2ircu can be used to query the IT-firmware but theres not much information to display since most functions are not available in that firmware.

 

Edited by Hexenhammer
Link to comment
  • Solution
11 hours ago, SimonF said:

found your post.

 

<qemu:override> <qemu:device alias='YOURNVMEALIAS'> <qemu:frontend> <qemu:property name='x-msix-relocation' type='string' value='bar2'/> </qemu:frontend> </qemu:device> </qemu:override>

 

Exactly, try this, copy/paste and save xml:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows 11</name>
  <uuid>bf15b8e2-225d-d84f-2b67-00d3e3e61b20</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>9437184</memory>
  <currentMemory unit='KiB'>9437184</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <vcpupin vcpu='4' cpuset='8'/>
    <vcpupin vcpu='5' cpuset='9'/>
    <vcpupin vcpu='6' cpuset='10'/>
    <vcpupin vcpu='7' cpuset='11'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/bf15b8e2-225d-d84f-2b67-00d3e3e61b20_VARS-pure-efi-tpm.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='8' threads='1'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:b2:bb:98'/>
      <source bridge='virbr0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-gb'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <qemu:override>
    <qemu:device alias='ua-sm2262'>
      <qemu:frontend>
        <qemu:property name='x-msix-relocation' type='string' value='bar2'/>
      </qemu:frontend>
    </qemu:device>
  </qemu:override>
</domain>

 

If it doesn't work attach diagnostics.

If part of the code is stripped it could be an unraid issue.

You didn't have any graphics defined in the xml, I added basic qxl + vnc, so connect to the vm with builtin novnc.

Once and if you fix the nvme issue you can try to passthrough the igpu.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.