• RC3 VM Alias Handler Change?

    Arbadacarba

    By Arbadacarba

      • Minor

    I'm running a pfsense VM with a 2port NIC and an Atheros Wifi card passed through.

     

    (I have a slightly unreliable POINT to POINT LTE connection, and occasional mission critical internet requirements - So the Wifi card is set to automatically connect to my cell phone hotspot)

     

    I had trouble getting this Wifi card to work with an error message referencing hostdev1 or something not fitting in BARs.

     

    image.thumb.png.de7558e8c54f0e1dbb3bbc404e5bbf0b.png

     

    I searched, I posted, I got a great answer from Ghost82 who translated the info I had found into the xml:

        <qemu:commandline>
          <qemu:arg value='-set'/>
          <qemu:arg value='device.hostdev1.x-msix-relocation=bar2'/>
        </qemu:commandline>
      </domain>

     

    This, with minimal fiddling (hostdev0 not hostdev1) worked perfectly.

    20220220_2240_pfSense - Cerberus.xml

     

    Right up until I updated to RC3.

     

    First the VFIO passthrough changed in System Devices, and then after I re-enabled that it still would not boot the VM.

    But now it doesn't refer to any hostdev id... it refers to it as bellow

      

    image.png.c84cd227207afc7bb7a0ef74d52395ae.png

     

    If I remove the Wifi card the 2 NIC ports get alias' lines, but if I add the Wifi card back in nothing does.

    Working - No WIFI.txt

    If I edit the xml to give the Wifi card an alias and put the comandline part back in.

    WIFI Added with Command.txt

     

    I get an error claiming there is no hostdev0

     

    image.png.c4ac339a0ae6781c02775e9eb7ab5e8d.png

     

    and when I look back in the XML the alias' are all gone.

    Alias missing.txt

     

    Ghost82 suggests editing the XML in the cli but I haven't tried that yet. Would it make a difference? Should it?

     

    Has there been some change t the way alias' are handled? I don't understand why they disappear.

     

    Thanks for any help

     

    Arbadacarba

     

    jupiter-diagnostics-20220315-0016.zip

    Go to reports Prereleases

    User Feedback

    Recommended Comments



    Kilroy83

    Hi same problem here since 6.10rc3. With macinabox VM 

    where i set this:

     

    <qemu:arg value='device.sata0-0-1.rotation_rate=1'/>

     

    The VM won’t start with error « no device sata0.0.1….

    i try to had a label but the setting not stay in place 

    Edited by Kilroy83
    Link to comment
    Jryski

    All my VM's using a passed-through NVME drive that required an alias and the -set have stopped working. Getting the same "No xxxx device defined" error and I've lost use of 4 VM's Is there a work around?

     

    Edited by Jryski
    Link to comment
    ghost82
    8 minutes ago, Jryski said:

    Is there a work around?

    Which was/were the -set command/s?

    If you are lucky enough there could be a direct entry in libvirt.

    Link to comment
    Jryski

    I was using a workaround posted here. Post #42. Here's a snippit of the code.
     

    <qemu:commandline>

    <qemu:arg value='-set'/>

    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>

    </qemu:commandline>

    Edited by Jryski
    Added post detail.
    Link to comment
    ghost82

    Unfortunately no luck with libvirt for x-msix-relocation.

    If you share your diagnostics file to see the direct qemu command for your vm, I think you can run your vm directly from the terminal, using directly qemu binary instead of relying on libvirt.

    We can try to translate the current command for qemu, avoiding json format, which prevents the -set argument to work.

    So, try to run the vm so that you have the command logged, don't bother if there is the error, then share diagnostics.

    Link to comment
    ghost82

    With absolute no warranties :D

    This is for your NVME Test RC4 vm.

    run a terminal window in unraid, paste this and hit enter, and check if the vm runs: 

    /usr/local/sbin/qemu \
-name 'guest=NVME Test RC4,debug-threads=on' \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-12-NVME Test RC4/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/34e523fe-c879-24d5-3b20-2ec6635682b6_VARS-pure-efi-tpm.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-6.2,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on,topoext=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \
-m 4096 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":4294967296}' \
-overcommit mem-lock=off \
-smp 12,sockets=1,dies=1,cores=6,threads=2 \
-uuid 34e523fe-c879-24d5-3b20-2ec6635682b6 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=36,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pcie.0","addr":"0x7"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Win11_English_x64v1.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-2-format","id":"sata0-0-0","bootindex":2}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-1-format","id":"sata0-0-1"}' \
-netdev tap,fd=38,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:10:ed:31","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0"}' \
-chardev socket,id=charchannel0,fd=39,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-chardev 'socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.sock' \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:1,websocket=5701,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \
-device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2 \
-device '{"driver":"vfio-pci","host":"0000:0a:00.3","id":"hostdev0","bus":"pci.4","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on

     

     

    What I did:

    I changed these: 

    -device '{"driver":"vfio-pci","host":"0000:03:00.0","id":"ua-sm2262","bus":"pci.3","addr":"0x0"}' \
-set device.ua-sm2262.x-msix-relocation=bar2 \

     

    into this: 

    -device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2 \

     

    avoiding json format.

     

    With this command, however, you will lose cpu pinning, not possible with qemu command line, without some additional scripts.

    Edited by ghost82
    Link to comment
    ghost82

    Sorry, you may need to start swtpm first with this command: 

    usr/bin/swtpm socket --daemon --ctrl 'type=unixio,path=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.sock,mode=0600' --tpmstate dir=/var/lib/libvirt/swtpm/34e523fe-c879-24d5-3b20-2ec6635682b6/tpm2,mode=0600 --log 'file=/var/log/swtpm/libvirt/qemu/NVME Test RC4-swtpm.log' --terminate --tpm2 --pid 'file=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.pid'

     

    Link to comment
    Jryski

    Thank you for offering this workaround. Here's the output. 

     

    2022-03-21T13:43:27.382125Z qemu-system-x86_64: Unable to read /var/lib/libvirt/qemu/domain-12-NVME Test RC4/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-12-NVME Test RC4/master-key.aes”: No such file or directory

     

     

    Link to comment
    Jryski
    7 minutes ago, ghost82 said:

    Sorry, you may need to start swtpm first with this command: 

    usr/bin/swtpm socket --daemon --ctrl 'type=unixio,path=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.sock,mode=0600' --tpmstate dir=/var/lib/libvirt/swtpm/34e523fe-c879-24d5-3b20-2ec6635682b6/tpm2,mode=0600 --log 'file=/var/log/swtpm/libvirt/qemu/NVME Test RC4-swtpm.log' --terminate --tpm2 --pid 'file=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.pid'

     

    bash: usr/bin/swtpm: No such file or directory

    Link to comment
    ghost82
    54 minutes ago, Jryski said:

    bash: usr/bin/swtpm: No such file or directory

    What about: 

    /usr/bin/swtpm socket --daemon --ctrl 'type=unixio,path=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.sock,mode=0600' --tpmstate dir=/var/lib/libvirt/swtpm/34e523fe-c879-24d5-3b20-2ec6635682b6/tpm2,mode=0600 --log 'file=/var/log/swtpm/libvirt/qemu/NVME Test RC4-swtpm.log' --terminate --tpm2 --pid 'file=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.pid'

    with a / in front of usr.

     

    1 hour ago, Jryski said:

    2022-03-21T13:43:27.382125Z qemu-system-x86_64: Unable to read /var/lib/libvirt/qemu/domain-12-NVME Test RC4/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-12-NVME Test RC4/master-key.aes”: No such file or directory

    Try to run the command without this: 

    -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-12-NVME Test RC4/master-key.aes"}' \

     

    Sorry but it's the first time I'm trying to run qemu without libvirt..

    Link to comment
    Jryski

    No need to apologize, your help is appreciated and I'm sure I'm not alone with this issue.

     

    the / in front of user seemed to do the trick it took the command.

    I removed the line as you suggested and this is the output.

    2022-03-21T14:54:26.279631Z qemu-system-x86_64: -chardev socket,id=charmonitor,fd=36,server=on,wait=off: File descriptor '36' is not a socket

     

    Link to comment
    ghost82

    ok, try this, let's try to simplify a bit:

    1. run the swtpm socket: 

    /usr/bin/swtpm socket --daemon --ctrl 'type=unixio,path=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.sock,mode=0600' --tpmstate dir=/var/lib/libvirt/swtpm/34e523fe-c879-24d5-3b20-2ec6635682b6/tpm2,mode=0600 --log 'file=/var/log/swtpm/libvirt/qemu/

     

    2. Save the attached script somewhere

    3. chmod a+x: 

    chmod a+x /path/to/VMtest.sh

     

    4. Try to run the script: 

    cd /path/to/VMtest
./VMtest.sh

     

    VMtest.sh

    Link to comment
    Jryski

    Hmm, seems like maybe it's trying to run in the wrong environment?

     

    😞 No such file or directory: setlocale: LC_ALL: cannot change locale (C)bin/bash: warning: setlocale: LC_ALL: cannot change locale (C
    ': No such file or directory
    ./VMtest.sh: line 10: -name: command not found
    ./VMtest.sh: line 11: -S: command not found
    ./VMtest.sh: line 12: -blockdev: command not found
    ./VMtest.sh: line 13: -blockdev: command not found
    ./VMtest.sh: line 14: -blockdev: command not found
    ./VMtest.sh: line 15: -blockdev: command not found
    ./VMtest.sh: line 16: -machine: command not found
    ./VMtest.sh: line 17: -accel: command not found
    ./VMtest.sh: line 18: -cpu: command not found
    ./VMtest.sh: line 19: -m: command not found
    ./VMtest.sh: line 20: -overcommit: command not found
    ./VMtest.sh: line 21: -smp: command not found
    ./VMtest.sh: line 22: -uuid: command not found
    ./VMtest.sh: line 23: -no-user-config: command not found
    ./VMtest.sh: line 24: -nodefaults: command not found
    ./VMtest.sh: line 25: -rtc: command not found
    ./VMtest.sh: line 26: -no-hpet: command not found
    ./VMtest.sh: line 27: -no-shutdown: command not found
    ./VMtest.sh: line 28: -boot: command not found
    ./VMtest.sh: line 29: -device: command not found
    ./VMtest.sh: line 30: -device: command not found
    ./VMtest.sh: line 31: -device: command not found
    ./VMtest.sh: line 32: -device: command not found
    ./VMtest.sh: line 33: -device: command not found
    ./VMtest.sh: line 34: -device: command not found
    ./VMtest.sh: line 36: -blockdev: command not found
    ./VMtest.sh: line 37: -blockdev: command not found
    ./VMtest.sh: line 38: -device: command not found
    ./VMtest.sh: line 39: -blockdev: command not found
    ./VMtest.sh: line 40: -blockdev: command not found
    ./VMtest.sh: line 41: -device: command not found
    ./VMtest.sh: line 42: -netdev: command not found
    ./VMtest.sh: line 43: -device: command not found
    ./VMtest.sh: line 44: -tpmdev: command not found
    ./VMtest.sh: line 45: -chardev: command not found
    ./VMtest.sh: line 46: -device: command not found
    ./VMtest.sh: line 47: -device: command not found
    ./VMtest.sh: line 48: -audiodev: command not found
    ./VMtest.sh: line 49: -vnc: command not found
    ./VMtest.sh: line 50: -k: command not found
    ./VMtest.sh: line 51: -device: command not found
    ./VMtest.sh: line 52: -device: command not found
    ./VMtest.sh: line 53: -device: command not found
    ./VMtest.sh: line 54: -msg: command not found

    Link to comment
    ghost82

    My god...I'm quite lost :D

    try with the command instead of the script, after running swtpm.

      

    /usr/local/sbin/qemu \
-name 'guest=NVME Test RC4,debug-threads=on' \
-S \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/34e523fe-c879-24d5-3b20-2ec6635682b6_VARS-pure-efi-tpm.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-6.2,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on,topoext=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \
-m 4096 \
-overcommit mem-lock=off \
-smp 12,sockets=1,dies=1,cores=6,threads=2 \
-uuid 34e523fe-c879-24d5-3b20-2ec6635682b6 \
-no-user-config \
-nodefaults \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pcie.0","addr":"0x7"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Win11_English_x64v1.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-2-format","id":"sata0-0-0","bootindex":2}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-1-format","id":"sata0-0-1"}' \
-netdev bridge,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:10:ed:31","bus":"pci.1","addr":"0x0"}' \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-chardev 'socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.sock' \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:1,websocket=5701,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \
-device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2 \
-device '{"driver":"vfio-pci","host":"0000:0a:00.3","id":"hostdev0","bus":"pci.4","addr":"0x0"}' \
-msg timestamp=on

     

    Link to comment
    Jryski

    Hahaha, I'm in a bit over my head. 

     

    failed to parse default acl file `/etc/qemu/bridge.conf'
    2022-03-21T16:56:18.190042Z qemu-system-x86_64: -netdev bridge,id=hostnet0: bridge helper failed

     

     

    • Like 1
    Link to comment
    ghost82

    Feel free to stop when you want, take into account that I can go on till eternity :D

    Since this is due to br0, what if you try without network? 

    /usr/local/sbin/qemu \
-name 'guest=NVME Test RC4,debug-threads=on' \
-S \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/34e523fe-c879-24d5-3b20-2ec6635682b6_VARS-pure-efi-tpm.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-6.2,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on,topoext=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \
-m 4096 \
-overcommit mem-lock=off \
-smp 12,sockets=1,dies=1,cores=6,threads=2 \
-uuid 34e523fe-c879-24d5-3b20-2ec6635682b6 \
-no-user-config \
-nodefaults \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pcie.0","addr":"0x7"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Win11_English_x64v1.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-2-format","id":"sata0-0-0","bootindex":2}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.215-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-1-format","id":"sata0-0-1"}' \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-chardev 'socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/12-NVME Test RC4-swtpm.sock' \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:1,websocket=5701,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \
-device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2 \
-device '{"driver":"vfio-pci","host":"0000:0a:00.3","id":"hostdev0","bus":"pci.4","addr":"0x0"}' \
-msg timestamp=on

     

    ---

    BTW, when you want to stop, you could think about attaching your nvme to a virtual sata controller (pass it theough with by-id, with sata bus) instead of passing through the whole nvme controller; that should work at expenses of some speed (not noticeable at all I think..)

    Link to comment
    Jryski

    2022-03-21T18:22:42.872554Z qemu-system-x86_64: Memory backend 'pc.ram' not found

     

    I'm fine with continuing, I don't want to waste your time!

    I'm just at the point of thinking that if it takes this much effort to make it work, it may not be worth the effort. I'm of the mindset that this should be out of the box supported, and it kind of was until this update. 

     

    I was trying to run a VM that is nearly as good as a bare metal PC, but reduces the physical requirments of having multiple PC's. 
     

    I've centrally located this box, and have run optical/active displayport/HDMi and USB where needed to have 1 PC that covers 3 rooms... for most of this a vdisk is fine, but for editing, gaming, etc... NVME speed is highly desirable. That said, if a virtual sata controller is **close**, I'd be happy.

     

    • Like 1
    Link to comment
    ghost82

    UPDATE: before trying this, try this:

    https://forums.unraid.net/bug-reports/prereleases/rc3-vm-alias-handler-change-r1768/?do=findComment&comment=17974

     

    I'm shooting in the dark here...keeping deleting things from the command is not good..

    Let's try something new...as I said you can pass a raw disk attached to a virtual controller in qemu/libvirt.

    In my post above I wrote to attach it to a virtual sata controller, that will work for sure; however, since qemu 6.0, nvme drive emulation was added; I never tried this, but as far as I understand, you should be able to pass the raw disk by specifying its id and attach it to an emulated nvme controller (instead of sata, or scsi, or virtio).

    Try this:

     

    1. uncheck "attach to vfio at boot" the nvme controller (unbind 03:00.0), and save

    2. Reboot unraid

    3. in unraid terminal find the id of your nvme drive: 

    ls /dev/disk/by-id/

     

    You should see your nvme disk and its partitions:

    nvmeXnY is the whole disk

    nvmeXnYpZ is a partition on the disk

     

    You are interested in the whole block, so nvmeXnY, where X and Y are numbers.

     

    As an example, with this output: 

    lrwxrwxrwx 1 root root 15 Jun  5 10:26 nvme-SAMSUNG -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 13 Jun  5 10:26 nvme-nvme.144d-533158574e594147393032353338-53414d53554e47204d5a5650563235364844474c2d3030303030-00000001 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 Jun  5 10:26 nvme-nvme.144d-533158574e594147393032353338-53414d53554e47204d5a5650563235364844474c2d3030303030-00000001-part1 -> ../../nvme0n1p1

     

    Your id will be: 

    nvme-nvme.144d-533158574e594147393032353338-53414d53554e47204d5a5650563235364844474c2d3030303030-00000001

     

    4. in the vm xml, delete this: 

        <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>

     

    and this: 

      <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
  </qemu:commandline>

     

    5. in the vm xml add the emulated nvme, before </domain> 

      <qemu:commandline>
    <qemu:arg value='-drive'/>
    <qemu:arg value='file=/dev/disk/by-id/HEREYOURDISKID,format=raw,if=none,id=NVME1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='nvme,drive=NVME1,serial=nvme-1'/>
  </qemu:commandline>

     

    Replace HEREYOURDISKID with the nvme id.

     

    6. start the vm

     

    Since the nvme controller is now emulated, you shouldn't need the x-msix fix.

    This should give near the same performance of the passed through nvme controller.

    Edited by ghost82
    Link to comment
    ghost82
    13 hours ago, Jryski said:

    I'm of the mindset that this should be out of the box supported

    I'm quite confident that this will be a temporary workaround and that a new version of libvirt will make it to work again in the future, but who knows when...

    Link to comment
    ghost82

    NOT WORKING

     

    I thought about another workaround...

    Why not using the -device arg in libvirt without using the -set attribute?

     

    So in alternative, starting from the original vm:

     

    1. keep 00:03.0 attached to vfio at boot

    2. delete from the vm xml:

      

        <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>

     

    and this: 

      <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>
  </qemu:commandline>

     

    3. in the vm xml add the real nvme controller, before </domain> 

      <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2'/>
  </qemu:commandline>

     

    4. start the vm

     

     

    ------

    @Arbadacarba this should work also for your network card (i.e. do not use the hostdev block to passthrough the device in libvirt xml, but use qemu arg to specify the vfio-pci device).

    For you, based on your attached file "WIFI Added with Command.txt":

    1. keep 00:05.0 attached to vfio at boot

    2. delete from the vm xml: 

        <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

     

    and this: 

        <qemu:commandline>
      <qemu:arg value='-set'/>
      <qemu:arg value='device.hostdev0.x-msix-relocation=bar2'/>
    </qemu:commandline>

     

    3. in the vm xml add the network card, before </domain> 

      <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=0000:05:00.0,id=hostdev0,bus=pci.4,addr=0x0,x-msix-relocation=bar2'/>
  </qemu:commandline>

     

    4. start the vm

    Edited by ghost82
    Link to comment
    Jryski

    You have a much firmer grasp on the inner workings of this that I do, I loosely understand the relationship between QEMU,KVM, and libvert. My knowledge ends on what duties can be shared and/or handed off to one of the others, so I very much appreciate the amount of effort you're putting into this. I didn't try unbinding the NVME yet, I tried your updated post and here's what happened.

     

    So when I remove the <hostdev> block for the drive, it unchecks the box in the gui and then throws this error.

     

    Execution error

    internal error: qemu unexpectedly closed the monitor: 2022-03-22T13:55:44.188067Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2: Bus 'pci.3' not found

     

    As for the network card, I'm not passing anything through for that, I've just been using the virtio-net device.

     

    If you're interested in looking at this in real time I could set up a screen share so you can see instant feedback, mind you, this is me tinkering, and at this point primarily with your help. It's not mission critical.

    • Like 1
    Link to comment
    ghost82
    2 minutes ago, Jryski said:

    internal error: qemu unexpectedly closed the monitor: 2022-03-22T13:55:44.188067Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2: Bus 'pci.3' not found

     

    sorry try to add also the alias pci.3 for the pcie-root-port with Index=3: 

        <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>

     

    6 minutes ago, Jryski said:

    As for the network card, I'm not passing anything through for that, I've just been using the virtio-net device

    Yes, that was the case for user "Arbadacarba", a different case which needs x-msix fix.

    Link to comment
    Jryski

    So this goes back to the original issue, the alias command is removed when you attempt to run this, so it throws the same error, and when you go back to look at the XML, <alias name='pci.3'/> is removed from the config.

     

    internal error: qemu unexpectedly closed the monitor: 2022-03-22T14:30:46.267608Z qemu-system-x86_64: -device vfio-pci,host=0000:03:00.0,id=ua-sm2262,bus=pci.3,addr=0x0,x-msix-relocation=bar2: Bus 'pci.3' not found

    • Like 1
    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.

    ×

  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.

    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.