RC3 VM Alias Handler Change?

1 minute ago, Jryski said:

So this goes back to the original issue, the alias command is removed when you attempt to run this

Can you attach the diagnostics for this?

I think the alias is not removed, but the issue only related to -set and json format, but it seems I'm wrong..

 

 

You could try also this:

1. replace this:

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2'/>
  </qemu:commandline>

 

with this:

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2'/>
  </qemu:commandline>

 

2. In addition to deleting the hostdev block for the nvme also delete this:

    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>

 

Diagnostic attached. Pulled it before I made the changes you suggested.

 

This is after making the suggested edits.

 

internal error: qemu unexpectedly closed the monitor: 2022-03-22T15:01:51.766791Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2: vfio 0000:03:00.0: failed to open /dev/vfio/21: Operation not permitted

unimatrixzero-diagnostics-20220322-1100.zip

1 minute ago, Jryski said:

This is after making the suggested edits.

 

internal error: qemu unexpectedly closed the monitor: 2022-03-22T15:01:51.766791Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2: vfio 0000:03:00.0: failed to open /dev/vfio/21: Operation not permitted

ok, seems better now, can you attach diagnostics with the latest edits applied please?

Progress is progress.

unimatrixzero-diagnostics-20220322-1107.zip

Damn...I suspect that the pcie-root-port with index 3 is automatically added again as a controller block...

    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>

 

did you delete it?

Yes, I deleted it, and I just tested again, If I remove it, update, and try to start the VM, it adds it back.

Thank you for your tests and patience, white flag for me (for now!) 🏳️  🏳️  🏳️  🏳️

I wrote in gitlab bugtracker too, you can follow discussion here:

https://gitlab.com/libvirt/libvirt/-/issues/287

 

I will come back if there will be a solution for this, or if I will have more ideas..

Thanks for all the assistance, I had fun and learned quite a bit about what's possible. Next scheduled restart I'll unstub the drive and try your other idea of passing it through on a virtual controller.

Another idea...use qemu:capabilities in the xml to remove json for devices...

 

So, remove the hostdev block, use qemu:arg for vfio injection and qemu:capabilities to remove json

 

Try to replace this:

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2'/>
  </qemu:commandline>

 

With this:

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2'/>
  </qemu:commandline>
  <qemu:capabilities>
    <qemu:del capability='device.json'/>
  </qemu:capabilities>

 

Or in alternative with this (depending on the libvirt/qemu versions):

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2'/>
  </qemu:commandline>
  <qemu:capabilities>
    <qemu:del capability='device.json+hotplug'/>
  </qemu:capabilities>

 

With some luck json should be disabled (???)

Edited March 22, 2022 by ghost82

First method results:

internal error: qemu unexpectedly closed the monitor: qxl_send_events: spice-server bug: guest stopped, ignoring 2022-03-22T16:46:48.094874Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2: vfio 0000:03:00.0: failed to open /dev/vfio/21: Operation not permitted

 

2nd:

internal error: invalid qemu namespace capability 'device.json+hotplug'

ok, last test for this, back to the beginning:

1. Restore the hostdev block:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>

 

2. Restore the -set qemu:arg and disable json:

  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
  </qemu:commandline>
  <qemu:capabilities>
    <qemu:del capability='device.json'/>
  </qemu:capabilities>

 

Can you attach diagnostics after this? just to look at if json is disabled or not.

Certainly

unimatrixzero-diagnostics-20220322-1259.zip

30 minutes ago, Jryski said:

Certainly

Thanks, but after my edits of the last post

 

Quote

failed to open /dev/vfio/21: Operation not permitted

This is the same error as before, it seems related to permissions (hopefully related to the way you are passing through the nvme with the qemu:arg instead of hostdev block), so if it worked before with the hostdev block I cannot see any reason it shouldn't work anymore (I'm referring to the permission error with iommu group 21 --> nvme controller)...And by disabling json for devices we should be with the "old" working qemu/libvirt behaviors (i.e. with aliases correctly seen by qemu:arg)...

 

Moreover from your latest diagnostics it seems that qemu:capabilities --> del did the trick, now devices are passed from libvirt to qemu in non json format and aliases are all there...hopefully it will work..

 

Another note: aliases are correctly passed from libvirt to qemu: the fact that they cannot be seen in the gui is another thing and I don't know from what it depends, but this is not relevant, as I said, from the logs, aliases are correctly passed.

Edited March 22, 2022 by ghost82

That did it. I'm super impressed.

I've included the diagnostics with the VM running, your last edits worked perfectly... this post should get stickied, as I know I'm not alone in trying to do this. Thank you for all the help. 

unimatrixzero-diagnostics-20220322-1404.zip

May have put my party hat on too soon. I can see the drive now in the VM, but so far I can't make any changes to it, so no creating partitions or formatting, or installing windows.

Did it work before the unraid upgrade or is this new hardware?

Any error during format/installation?

Windows throws an error "We couldn't create a new partition error: 0xfdd10070"
 

This is the first time I've got this particular passthrough working, This nvme controller sm2262 has been an issue all along, samsung NVME I had in it before worked fine.

 

I don't think that this is related to the fix.

However, are you able to see Repair your computer button, after yiu get the error and have access to a command prompt from the installation disk?if so, you could try to use diskpart utility and try to format the disk there.

Otherwise you can create a new vm with the nvme passed through and gparted live disk and try to format the nvme there.

Edited March 23, 2022 by ghost82

About masking device.json:

 

PLEASE NOTE THAT THIS IS A TEMPORARY WORKAROUND AND THAT FOR FUTURE LIBVIRT/QEMU VERSIONS MASKING DEVICE.JSON MAY BREAK OTHER FEATURES.

PATCHES FOR PROPERLY OVERRIDE ARE ON THEIR WAY, SO HOPEFULLY THE NEXT LIBVIRT VERSION (8.2.0?) WILL ALLOW THESE OVERRIDES WITHOUT MASKING JSON AND UNRAID COULD INCLUDE THE NEW LIBVIRT VERSION.

 

https://listman.redhat.com/archives/libvir-list/2022-March/229463.html

 

Pinging also @limetech for this.

 

---

If and when the patches will be merged, I think that the fix will be:

    <qemu:override>
      <qemu:device alias='YOURALIASHERE'>
        <qemu:frontend>
          <qemu:property name='x-msix-relocation' type='string' value='bar2'/>
        </qemu:frontend>
      </qemu:device>
    </qemu:override>

 

Replacing this code block:

  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.YOURALIASHERE.x-msix-relocation=bar2'/>
  </qemu:commandline>
  <qemu:capabilities>
    <qemu:del capability='device.json'/>
  </qemu:capabilities>

 

Edited March 23, 2022 by ghost82

I will update if I can get this working via whatever means. Thanks again for all the information, I'll update this post after the next libvert/qemu update with how I update the XML. I tell you what though, I REALLY wish they'd add a toggle to make the XML sticky when a GUI change is made... Thanks again for all the work.

 

Same issue in DISKPART, it fails due to "i/o errors" wondering if it has something to do with the fact that the drive is being passed as an entire controller, but also the xml refers to it as a generic PCI device, not a SCSI controller? Grasping at straws. At this point, I'm ready to call this non-working until the updates are here. I can see the drive, but no luck making changes.

The VM also takes 10 minutes to get to the windows installer screen, and hangs at each step. I've attempted to limit the cores down to 1 which was a workaround in the past, as well as reduce the ram down to minimum required for windows 11 to allow the install. Despite all our effort, I believe it's time to wait for official support.

On 3/23/2022 at 9:46 AM, ghost82 said:

PATCHES FOR PROPERLY OVERRIDE ARE ON THEIR WAY, SO HOPEFULLY THE NEXT LIBVIRT VERSION (8.2.0?)

Patches are currently applied:

https://gitlab.com/libvirt/libvirt/-/commit/38ab5c9ead58332acc5129fc6066b054d6e6e38c

 

Features will be available in the next libvirt release (8.2.0).

Edited March 28, 2022 by ghost82

Libvirt 8.2.0 was officially released yesterday with the override fix.

 

Quote

qemu: Allow overrides of device properties via the qemu namespace

Users wishing to override or modify properties of devices configured by libvirt can use the <qemu:deviceOverride> QEMU namespace element to specify the overrides instead of relying on the argv passthrough of the -set qemu commandline option which no longer works with new qemu.

 

Edited April 2, 2022 by ghost82

Just for info, now unraid 6.10RC7 includes libvirt 8.2.0, so this applies from this version (example referred to x-msix-relocation):

    <qemu:override>
      <qemu:device alias='YOURALIASHERE'>
        <qemu:frontend>
          <qemu:property name='x-msix-relocation' type='string' value='bar2'/>
        </qemu:frontend>
      </qemu:device>
    </qemu:override>