macmus

System specs - Unraid 7.2.4, kernel 6.12.54 - AMD Threadripper / Matisse platform (ASUS board) - QEMU 9.2.3 / libvirt 11.7.0 - RTX 4060 Gigabyte (AD107, 01:00.0) — passed through to Windows 10 VM - RTX 2080 Ti (TU102, 4d:00.0) — passed through to separate Windows 11 VM (working fine, no issues) - RTX 4060 is the primary/boot VGA on the host — console access must be preserved --- Problem My Windows 10 VM with RTX 4060 passthrough fails to start after the VM has been stopped or crashed, with this error: ``` Execution error internal error: Unknown PCI header type '127' for device '0000:01:00.0' ``` A full Unraid server reboot always recovers it — the VM starts fine on first boot. The problem only happens when trying to restart the VM without rebooting the host. --- IOMMU grouping — clean, no sharing issues ``` IOMMU Group 74 01:00.0 NVIDIA Corporation AD107 [GeForce RTX 4060] [10de:2882] (rev a1) IOMMU Group 74 01:00.1 NVIDIA Corporation AD107 HD Audio Controller [10de:22be] (rev a1) ``` Both functions are isolated in their own group. No ACS override needed. --- Current boot args ``` BOOT_IMAGE=/bzimage vfio-pci.ids=1022:148c,10de:1ad6,10de:1ad7 video=efifb:off isolcpus=1-14,25-38 initrd=/bzroot kvm_amd.nested=1 ``` Note: 10de:2882 and 10de:22be (the RTX 4060) are NOT in vfio-pci.ids. The 2080 Ti functions 10de:1ad6 and 10de:1ad7 are listed but not the GPU itself either — those were added for its USB/UCSI controllers. --- Driver state after VM shutdown ``` ls: cannot access '/sys/bus/pci/devices/0000:01:00.0/driver': No such file or directory ls: cannot access '/sys/bus/pci/devices/0000:01:00.1/driver': No such file or directory ``` No driver is bound to the GPU between VM sessions. On boot it is detected correctly: ``` [ 0.392439] pci 0000:01:00.0: [10de:2882] type 00 class 0x030000 PCIe Legacy Endpoint [ 0.435513] pci 0000:01:00.0: vgaarb: setting as boot VGA device [ 2.086958] vfio_pci: add [1022:148c[ffffffff:ffffffff]] class 0x000000/00000000 ``` The RTX 4060 is being picked up as boot VGA by the host, and is NOT being claimed by vfio-pci at boot (only the 2080 Ti USB controllers are). --- dmesg after VM shutdown — GPU reset failure ``` vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs [repeats ~30 times] vfio-pci 0000:01:00.0: timed out waiting for pending transaction; performing function level reset anyway vfio-pci 0000:01:00.1: Unable to change power state from D0 to D3hot, device inaccessible pcieport 0000:00:01.1: broken device, retraining non-functional downstream link at 2.5GT/s pcieport 0000:00:01.1: retraining failed pcieport 0000:00:01.1: Data Link Layer Link Active not set in 100 msec vfio-pci 0000:01:00.0: Unable to change power state from D3cold to D0, device inaccessible vfio-pci 0000:01:00.1: Unable to change power state from D3cold to D0, device inaccessible ``` The PCIe link to the GPU drops entirely after the VM stops. On the next VM start QEMU reads all 0xFF from the config space (device unreachable) and reports header type 127. --- VM XML — relevant passthrough section ```xml <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/GeForceRTX4060.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> ``` The custom vBIOS ROM file is only 61KB — which seems far too small for an RTX 4060 (I'd expect 200KB+). Not sure if this is contributing. --- What I think is happening The RTX 4060 AD107 appears to have a reset bug similar to the AMD reset bug — after VM shutdown the FLR fails, the PCIe link drops, and the card is completely inaccessible until a full power cycle. The 2080 Ti on the same system passes through without any such issues. --- Questions 1. Is there a known fix for ths reset BUG? 2. Would adding pcie=noaer to boot args help suppress the PCIe error cascade that drops the link? I don't want to brick my Unraid server, but still be able to access it over console. 3. Is my vBIOS ROM ok? I'm not sure how did i get it, since it was long ago. Should I remove it or switch to <rom bar='off'/>? Where i can download a working 4060 bios ? 4. Can I safely add 10de:2882,10de:22be to vfio-pci.ids without losing host console/display output — given this GPU is also the host's primary display output? Thanks in advance. Happy to run any additional diagnostics.

I found this errors in my server logs folder, i don't remember seeing this before. Do you know what it is and how critial is this ? Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.WT1A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.MT1A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.WT2A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.MT2A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.WT3A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.MT3A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.WT4A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.MT4A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CA.MT5A], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.WT1B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.MT1B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.WT2B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.MT2B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.WT3B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.MT3B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.WT4B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.MT4B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CB.MT5B], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.WT1C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.MT1C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.WT2C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.MT2C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.WT3C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.MT3C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.WT4C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.MT4C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CC.MT5C], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.WT1D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.MT1D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.WT2D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.MT2D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.WT3D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.MT3D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.WT4D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.MT4D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.I2CD.MT5D], AE_ALREADY_EXISTS (20220331/dswload2-326) Sep 22 09:56:06 Tower kernel: ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20220331/psobject-220) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): AE_AML_PACKAGE_LIMIT, Index (0x000000007) is beyond end of object (length 0x6) (20220331/exoparg2-393) Sep 22 09:56:06 Tower kernel: ACPI Error: Aborting method \_SB.S0D2.D2A0.BYUP.BYD8.XHC1.RHUB.PRT6._PLD due to previous error (AE_AML_PACKAGE_LIMIT) (20220331/psparse-529) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): AE_AML_PACKAGE_LIMIT, Index (0x000000007) is beyond end of object (length 0x6) (20220331/exoparg2-393) Sep 22 09:56:06 Tower kernel: ACPI Error: Aborting method \_SB.S0D2.D2A0.BYUP.BYD8.XHC1.RHUB.PRT6._PLD due to previous error (AE_AML_PACKAGE_LIMIT) (20220331/psparse-529) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): AE_AML_PACKAGE_LIMIT, Index (0x000000007) is beyond end of object (length 0x6) (20220331/exoparg2-393) Sep 22 09:56:06 Tower kernel: ACPI Error: Aborting method \_SB.S0D2.D2A0.BYUP.BYD8.XHC1.RHUB.PRT6._PLD due to previous error (AE_AML_PACKAGE_LIMIT) (20220331/psparse-529) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): AE_AML_PACKAGE_LIMIT, Index (0x000000007) is beyond end of object (length 0x6) (20220331/exoparg2-393) Sep 22 09:56:06 Tower kernel: ACPI Error: Aborting method \_SB.S0D2.D2A0.BYUP.BYD8.XHC1.RHUB.PRT6._PLD due to previous error (AE_AML_PACKAGE_LIMIT) (20220331/psparse-529) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): AE_AML_PACKAGE_LIMIT, Index (0x000000007) is beyond end of object (length 0x6) (20220331/exoparg2-393) Sep 22 09:56:06 Tower kernel: ACPI Error: Aborting method \_SB.S0D2.D2A0.BYUP.BYD8.XHC1.RHUB.PRT6._PLD due to previous error (AE_AML_PACKAGE_LIMIT) (20220331/psparse-529) Sep 22 09:56:06 Tower kernel: ACPI BIOS Error (bug): AE_AML_PACKAGE_LIMIT, Index (0x000000007) is beyond end of object (length 0x6) (20220331/exoparg2-393) Sep 22 09:56:06 Tower kernel: ACPI Error: Aborting method \_SB.S0D2.D2A0.BYUP.BYD8.XHC1.RHUB.PRT6._PLD due to previous error (AE_AML_PACKAGE_LIMIT) (20220331/psparse-529) Sep 22 09:56:20 Tower ntpd[2345]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Sep 22 09:56:20 Tower ntpd[2345]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

If i do the backup of pendrive it should be safe to to plug the resoration copy or there is stuff written to array/cache during the upgrade too?

I'm good at troubleshooting, but usually the jump between major releases may cause issues, hence I am preemptively asking. This is my main server on which VMs i do all my daily work, so it's kidda crital :-) but on the other hand i don't want to be left behind with updates to the point that i will need to do a reinstall or some such. Do you think it's better to update to 7.2, once it comes out ?

I must admit I have been slacking with upgrades, but that is because everything is working, and I am constantly using 3-5 VMs, which run 24/7 for various tasks. I know that the longer I wait, the more dangerous the upgrade is going to be, but how safe is it for me to do the upgrade now just by "clicking" the upgrade button? Do I need to know anything beforehand? I'm using multiple Docker images and 2 pass-through VMs from 2 graphics cards.

updated to latest unraid version, still only one VM is getting backed up from the list selected. Does anyone have this issue ?

Did u find any solution ?

I finally did a passthrough of two graphic cards I have GF2080Ti and P2000 to the VMs for streaming. I wanted have encoding capabilities. I'm wondering if it's possible to do something like: so ether of those cards I can pass to multiple VMs?

I bought P2000 and it finally worked on both cards.. for some reason G710 did not like to be passthrough

I have same problem and I noticed Sep 3 12:29:10 Tower kernel: pci 0000:01:00.0: vgaarb: setting as boot VGA device Sep 3 12:29:10 Tower kernel: pci 0000:01:00.0: vgaarb: bridge control possible Sep 3 12:29:10 Tower kernel: pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none Sep 3 12:29:10 Tower kernel: pci 0000:4d:00.0: vgaarb: bridge control possible Sep 3 12:29:10 Tower kernel: pci 0000:4d:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none Sep 3 12:29:10 Tower kernel: vgaarb: loaded I'm using GT710 in my slot

you you give me a hand on my issue ?

2080TI I was able to passthrough which is in 3rd slot. GT710 from primary slot I'm not able to passthrough.

so far I am able to passthrough that card, vm boots, but I'm getting Code 43 this is what I am using: kernel /bzimage append vfio-pci.ids=1022:148c,10de:1ad6,10de:1ad7 video=efifb:off isolcpus=2-11,26-35 initrd=/bzroot kvm_amd.nested=1 tower-diagnostics-20230904-1848.zip <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='12'> <name>WinAP</name> <uuid>c52df59f-c118-5e6a-ba7a-97e4cf49c880</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='14'/> <vcpupin vcpu='1' cpuset='38'/> <vcpupin vcpu='2' cpuset='15'/> <vcpupin vcpu='3' cpuset='39'/> <vcpupin vcpu='4' cpuset='16'/> <vcpupin vcpu='5' cpuset='40'/> <vcpupin vcpu='6' cpuset='17'/> <vcpupin vcpu='7' cpuset='41'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/c52df59f-c118-5e6a-ba7a-97e4cf49c880_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='4' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/vm_ssd/domains_ssd/WinAP_082023.img' index='2'/> <backingStore/> <target dev='hdc' bus='virtio'/> <serial>vdisk1</serial> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso' index='1'/> <backingStore/> <target dev='hdb' bus='ide'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:8f:32:e8'/> <source bridge='br0'/> <target dev='vnet11'/> <model type='virtio-net'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-12-WinAP/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

Can I do same for primary card ? I have currently two cards in my server GT710 and 2080Ti. The 2080ti I'm able to passthrough using append vfio-pci.ids=1022:148c,10de:1ad6,10de:1ad7 video=efifb:off isolcpus=2-11,26-35 pic-ids are devicesID for 2080 but I am not able to passthrough GT710 ? Should I also list it within VFIO ?

Ok I was able to fix pausing I think .. it seems it was not related to graphic card.. but still .. Did everything like in this clip and did not help .. still getting error code 43 <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='6'> <name>WinAP</name> <uuid>c52df59f-c118-5e6a-ba7a-97e4cf49c880</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='14'/> <vcpupin vcpu='1' cpuset='38'/> <vcpupin vcpu='2' cpuset='15'/> <vcpupin vcpu='3' cpuset='39'/> <vcpupin vcpu='4' cpuset='16'/> <vcpupin vcpu='5' cpuset='40'/> <vcpupin vcpu='6' cpuset='17'/> <vcpupin vcpu='7' cpuset='41'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/c52df59f-c118-5e6a-ba7a-97e4cf49c880_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='4' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source file='/mnt/vm_ssd/domains_ssd/WinAP_082023.img' index='2'/> <backingStore/> <target dev='hdc' bus='virtio'/> <serial>vdisk1</serial> <boot order='1'/> <alias name='virtio-disk2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso' index='1'/> <backingStore/> <target dev='hdb' bus='ide'/> <readonly/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:8f:32:e8'/> <source bridge='br0'/> <target dev='vnet5'/> <model type='virtio-net'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-6-WinAP/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/isos/MSI.GT710.2048.160112.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

I updated my unraid today to 6.12.4 and was able to select now GF710 (which is in the primary slot). I was able to assign to VM this 710 card and boot the VM. Windows 10 successfully starts with it, and detects it but it shows an error in the device manager (Windows has stopped this device error) and I'm not able to install any drivers for this graphic card. 2080TI in the 3rd slot after the upgrade works without issues and passes successfully as a secondary card to my other VM.

anyone has any idea?

I'm sorry for not being clear. I have in list selected 3 VMs GameServer, UbuntuServer and WinAP. When I run manually the backup now it finds all 3 servers, but it performs backup only of a first one from the list. In this case GameServer is the first server. If I uncheck it from the list It will backup Ubuntu but not WinAP etc.. You can see it all in logs.

pasted log above. It just doesn't start it ? 2023-08-22 20:26:13 information: started attempt to backup GameServer, UbuntuServer, WinAP to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto It find GameServer and does the backup of UbuntuServer and WinAP

When I select multiple VMs for backup and I select 3 servers for backup it only does backup of first and then finish the job ? Why it does not try to proceed to next VM and is doing only 1st one ? It was suppose to run backup on GameServer, UbuntuServer, WinAP but it only did on GameServer why? 2023-08-22 20:26:13 information: snapshot_fallback is 0. snapshots will fallback to standard backups. 2023-08-22 20:26:13 information: pause_vms is 0. vms will be shutdown for standard backups. 2023-08-22 20:26:13 information: enable_reconstruct_write is 0. reconstruct write will not be enabled by this script. 2023-08-22 20:26:13 information: compare_files is 0. files will not be compared after backups. 2023-08-22 20:26:13 information: backup_xml is 1. vms will have their xml configurations backed up. 2023-08-22 20:26:13 information: backup_nvram is 1. vms will have their nvram backed up. 2023-08-22 20:26:13 information: backup_vdisks is 1. vms will have their vdisks backed up. 2023-08-22 20:26:13 information: start_vm_after_backup is 0. vms will not be started following successful backup. 2023-08-22 20:26:13 information: start_vm_after_failure is 0. vms will not be started following an unsuccessful backup. 2023-08-22 20:26:13 information: disable_delta_sync is 0. rsync will be used to perform delta sync backups. 2023-08-22 20:26:13 information: rsync_only is 0. cp will be used when applicable. 2023-08-22 20:26:13 information: actually_copy_files is 1. files will be copied. 2023-08-22 20:26:13 information: clean_shutdown_checks is 20. this is probably a sufficient number of shutdown checks. 2023-08-22 20:26:13 information: seconds_to_wait is 30. this is probably a sufficient number of seconds to wait between shutdown checks. 2023-08-22 20:26:13 information: keep_error_log_file is 1. error log files will be kept. 2023-08-22 20:26:13 information: number_of_error_log_files_to_keep is 10. this is probably a sufficient error number of log files to keep. 2023-08-22 20:26:13 information: started attempt to backup GameServer, UbuntuServer, WinAP to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto 2023-08-22 20:26:13 information: GameServer can be found on the system. attempting backup. 2023-08-22 20:26:13 information: removing old local GameServer.xml. removed 'GameServer.xml' 2023-08-22 20:26:13 information: creating local GameServer.xml to work with during backup. 2023-08-22 20:26:13 information: /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer exists. continuing. 2023-08-22 20:26:13 information: copy of backup of /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_1043_20230804_2115_Gameserver_082023.img vdisk to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_2026_20230804_2115_Gameserver_082023.img starting. '/mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_1043_20230804_2115_Gameserver_082023.img' -> '/mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_2026_20230804_2115_Gameserver_082023.img' 2023-08-22 20:40:57 information: copy of /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_1043_20230804_2115_Gameserver_082023.img to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_2026_20230804_2115_Gameserver_082023.img complete. 2023-08-22 20:40:57 information: extension for /mnt/user/isos/debian-11.7.0-amd64-netinst.iso on GameServer was found in vdisks_extensions_to_skip. skipping disk. 2023-08-22 20:40:57 information: skip_vm_shutdown is false. beginning vm shutdown procedure. 2023-08-22 20:40:57 information: GameServer is running. vm desired state is shut off. Domain GameServer is being shutdown 2023-08-22 20:40:57 information: performing 20 30 second cycles waiting for GameServer to shutdown cleanly. 2023-08-22 20:40:57 information: cycle 1 of 20: waiting 30 seconds before checking if the vm has entered the desired state. 2023-08-22 20:41:27 information: GameServer is shut off. vm desired state is shut off. can_backup_vm set to y. 2023-08-22 20:41:27 information: actually_copy_files is 1. 2023-08-22 20:41:27 information: can_backup_vm flag is y. starting backup of GameServer configuration, nvram, and vdisk(s). sending incremental file list GameServer.xml sent 7,277 bytes received 35 bytes 14,624.00 bytes/sec total size is 7,167 speedup is 0.98 2023-08-22 20:41:27 information: copy of GameServer.xml to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_2026_GameServer.xml complete. sending incremental file list a5374532-5d73-88d9-d03e-506821f267df_VARS-pure-efi.fd sent 131,245 bytes received 35 bytes 262,560.00 bytes/sec total size is 131,072 speedup is 1.00 2023-08-22 20:41:27 information: copy of /etc/libvirt/qemu/nvram/a5374532-5d73-88d9-d03e-506821f267df_VARS-pure-efi.fd to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_2026_a5374532-5d73-88d9-d03e-506821f267df_VARS-pure-efi.fd complete. sending incremental file list 20230804_2115_Gameserver_082023.img sent 694,108,869 bytes received 3,277,035 bytes 1,248,676.64 bytes/sec total size is 53,687,091,200 speedup is 76.98 2023-08-22 20:50:45 information: copy of /mnt/vm_ssd/domains_ssd/20230804_2115_Gameserver_082023.img to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_2026_20230804_2115_Gameserver_082023.img complete. 2023-08-22 20:50:45 information: backup of /mnt/vm_ssd/domains_ssd/20230804_2115_Gameserver_082023.img vdisk to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230822_2026_20230804_2115_Gameserver_082023.img complete. 2023-08-22 20:50:45 information: extension for /mnt/user/isos/debian-11.7.0-amd64-netinst.iso on GameServer was found in vdisks_extensions_to_skip. skipping disk. 2023-08-22 20:50:45 information: the extensions of the vdisks that were backed up are img. 2023-08-22 20:50:45 information: vm_state is shut off. vm_original_state is running. starting GameServer. Domain GameServer started 2023-08-22 20:50:46 information: backup of GameServer to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer completed. 2023-08-22 20:50:46 information: cleaning out backups older than 90 days in location ONLY if newer files exist in /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/ 2023-08-22 20:50:46 information: did not find any config files to remove. 2023-08-22 20:50:46 information: did not find any nvram files to remove. 2023-08-22 20:50:46 information: did not find any vdisk image files to remove. 2023-08-22 20:50:46 information: did not find any vm log files to remove. 2023-08-22 20:50:46 information: cleaning out backups over 2 in location /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/ 2023-08-22 20:50:46 information: removed '/mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230820_0000_GameServer.xml' config file. 2023-08-22 20:50:46 information: removed '/mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/GameServer/20230820_0000_a5374532-5d73-88d9-d03e-506821f267df_VARS-pure-efi.fd' nvram file. /tmp/vmbackup/scripts/default/user-script.sh: line 2899: 082023: value too great for base (error token is "082023") 2023-08-22 20:50:46 information: finished attempt to backup GameServer, UbuntuServer, WinAP to /mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto. 2023-08-22 20:50:46 information: cleaning out logs over 1. 2023-08-22 20:50:46 information: removed '/mnt/user/NOWADZIELNIA/_Domain_Backup/backup_auto/logs/20230822_1139_unraid-vmbackup.log'. 2023-08-22 20:50:46 information: cleaning out error logs over 10. 2023-08-22 20:50:46 information: did not find any error log files to remove. 2023-08-22 20:50:46 Stop logging to log file. 2023-08-22 20:50:46 Found /tmp/vmbackup/scripts/default.pid. Attempting to kill process. 2023-08-22 20:50:46 Killed user script with pid 93908 for default. 2023-08-22 20:50:46 Removed: /tmp/vmbackup/scripts/default/user-script.sh 2023-08-22 20:50:46 Removed: /tmp/vmbackup/scripts/default.pid

Can I passthrough RTX4000 or P2000 as the primary card?

I have a 2080Ti passthrough to one of my VMs as a secondary graphic card on the server. Another card I have (primary card) is GT710. I have tried to pass it through to the VM according to spaceinvader guide (with bios dump), but I failed. The purpose was to install Parsec or Moonlight for better latency. Now I'm looking for a replacement of GeForce GT 710 with which I could do that. Any suggestion on which card can be used, and is it possible to be passthrough event being a primary card? It has to be one slot card.

Is it even worth risking the update ?

I have app data, docker image and libvrt backed up .. Are those getting impacted too ? I don't have anything complicated, but I have one gaming VM with a NVDA 2080Ti passthrough and 2 nvme passthrough,

but what if I plan to move from 6.9.2 could it blow up my current setup?