RTX 4090 Passthrough - I'm dying


Go to solution Solved by Kodon,

Recommended Posts

Hey Unraid magicians!

 

I can not get my Gigabyte 4090 Gaming OC passed through to my Windows 11 VM no matter what I try.

 

For the last two days I have been trying everything I could find in what feels like all possible combinations.

Any help is greatly appreciated!

 

So far I have tried:

 

- Dumping the vbios and adding it as original or hex edited

- Disabling rBAR in Bios

- Binding to VFIO

- adding this to the VM config:

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  [...]
<qemu:commandline>
  <qemu:arg value='-fw_cfg'/>
  <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
</qemu:commandline>

 

The display I attached just stays blank. When I connect via TeamViewer I connect to the tinyest resolution possible and can't find the GPU in the device manager, so it's not a driver issue in the VM.

 

I'm really losing my sanity on this. I have been Using a T1000 card without any issues before.

 

Thanks for any input!

 

arctictower-diagnostics-20230528-2042.zip

Link to comment
Quote

May 28 20:30:13 ArcticTower kernel: pci 0000:01:00.0: BAR 1: assigned to efifb

 

Quote

May 28 20:30:13 ArcticTower kernel: Command line: BOOT_IMAGE=/bzimage vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction initrd=/bzroot

 

--> video=efifb:off

Edited by ghost82
Link to comment
33 minutes ago, ghost82 said:

--> video=efifb:off

 

Heya! Thanks for the input.

 

I changed the setting in the syslinux config. But it didn't change anything...

New Log entries:


 

May 30 12:24:50 ArcticTower kernel: Command line: BOOT_IMAGE=/bzimage vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction video=efifb:off initrd=/bzroot

[...]

May 30 12:24:50 ArcticTower kernel: pci 0000:01:00.0: BAR 1: assigned to efifb

 

Link to comment
On 5/28/2023 at 7:49 PM, Kodon said:

Hey Unraid magicians!

 

I can not get my Gigabyte 4090 Gaming OC passed through to my Windows 11 VM no matter what I try.

 

For the last two days I have been trying everything I could find in what feels like all possible combinations.

Any help is greatly appreciated!

 

So far I have tried:

 

- Dumping the vbios and adding it as original or hex edited

- Disabling rBAR in Bios

- Binding to VFIO

- adding this to the VM config:

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  [...]
<qemu:commandline>
  <qemu:arg value='-fw_cfg'/>
  <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
</qemu:commandline>

 

The display I attached just stays blank. When I connect via TeamViewer I connect to the tinyest resolution possible and can't find the GPU in the device manager, so it's not a driver issue in the VM.

 

I'm really losing my sanity on this. I have been Using a T1000 card without any issues before.

 

Thanks for any input!

 

arctictower-diagnostics-20230528-2042.zip 108.26 kB · 2 downloads

Go to Tools, system devices. Click on 4090 and tick all the box’s next to Nvidia, such as audio and the gpu.

 

then simply select the graphics card and the audio Checkbox’s in the VM edit tab. Leave everything else as default, this will work fine.

Link to comment
On 6/1/2023 at 10:31 PM, Nano said:

Go to Tools, system devices. Click on 4090 and tick all the box’s next to Nvidia, such as audio and the gpu.

 

then simply select the graphics card and the audio Checkbox’s in the VM edit tab. Leave everything else as default, this will work fine.

 

I took this to start over from scratch. I reset the BIOS and reformatted the Unraid USB to get rid of all settings.

Leaving everything at default and "ticking the boxes" and... still not working :c

 

vfio-pci Log

 

Spoiler
Loading config from /boot/config/vfio-pci.cfg
BIND=0000:01:00.0|10de:2684 0000:01:00.1|10de:22ba
---
Processing 0000:01:00.0 10de:2684
Vendor:Device 10de:2684 found at 0000:01:00.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.0
/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.1

Binding...
Successfully bound the device 10de:2684 at 0000:01:00.0 to vfio-pci
---
Processing 0000:01:00.1 10de:22ba
Vendor:Device 10de:22ba found at 0000:01:00.1

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.0
/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.1

Binding...
0000:01:00.0 already bound to vfio-pci
0000:01:00.1 already bound to vfio-pci
Successfully bound the device 10de:22ba at 0000:01:00.1 to vfio-pci
---
vfio-pci binding complete

Devices listed in /sys/bus/pci/drivers/vfio-pci:
lrwxrwxrwx 1 root root    0 Jun  8 16:51 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root    0 Jun  8 16:51 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1

 

 

VM Log

 

Spoiler
-device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-2-format","id":"sata0-0-0","bootindex":2}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.229-1.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-1-format","id":"sata0-0-1"}' \
-netdev tap,fd=36,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:0f:ce:cc","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=34,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/1-Games-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,websocket=5700,audiodev=audio1 \
-k de \
-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.4","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring
2023-06-08T17:27:08.650464Z qemu-system-x86_64: terminating on signal 15 from pid 2869 (/usr/sbin/libvirtd)
2023-06-08 17:27:08.879+0000: shutting down, reason=shutdown
2023-06-08 17:28:10.970+0000: Starting external device: TPM Emulator
/usr/bin/swtpm socket --ctrl type=unixio,path=/run/libvirt/qemu/swtpm/2-Games-swtpm.sock,mode=0600 --tpmstate dir=/var/lib/libvirt/swtpm/a2694598-c0a6-e077-916f-47926aa66729/tpm2,mode=0600 --log file=/var/log/swtpm/libvirt/qemu/Games-swtpm.log --terminate --tpm2
2023-06-08 17:28:10.983+0000: starting up libvirt version: 8.7.0, qemu version: 7.1.0, kernel: 5.19.17-Unraid, hostname: ArcticTower
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME=/var/lib/libvirt/qemu/domain-2-Games \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-Games/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-Games/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-Games/.config \
/usr/local/sbin/qemu \
-name guest=Games,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-Games/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/a2694598-c0a6-e077-916f-47926aa66729_VARS-pure-efi-tpm.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-7.1,usb=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \
-m 16384 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":17179869184}' \
-overcommit mem-lock=off \
-smp 16,sockets=1,dies=1,cores=8,threads=2 \
-uuid a2694598-c0a6-e077-916f-47926aa66729 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=35,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x7.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x7"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x7.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x7.0x2"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Games/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.3","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \
-netdev tap,fd=36,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:0f:ce:cc","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=34,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/2-Games-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:01:00.1","id":"hostdev1","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/002","id":"hostdev2","bus":"usb.0","port":"2"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/003","id":"hostdev3","bus":"usb.0","port":"3"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)

 

 

Unraid Log

 

Spoiler
Jun  8 10:27:08 ArcticTower  avahi-daemon[2328]: Interface vnet0.IPv6 no longer relevant for mDNS.
Jun  8 10:27:08 ArcticTower  avahi-daemon[2328]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe0f:cecc.
Jun  8 10:27:08 ArcticTower kernel: br0: port 2(vnet0) entered disabled state
Jun  8 10:27:08 ArcticTower kernel: device vnet0 left promiscuous mode
Jun  8 10:27:08 ArcticTower kernel: br0: port 2(vnet0) entered disabled state
Jun  8 10:27:08 ArcticTower  avahi-daemon[2328]: Withdrawing address record for fe80::fc54:ff:fe0f:cecc on vnet0.
Jun  8 10:28:10 ArcticTower kernel: br0: port 2(vnet1) entered blocking state
Jun  8 10:28:10 ArcticTower kernel: br0: port 2(vnet1) entered disabled state
Jun  8 10:28:10 ArcticTower kernel: device vnet1 entered promiscuous mode
Jun  8 10:28:10 ArcticTower kernel: br0: port 2(vnet1) entered blocking state
Jun  8 10:28:10 ArcticTower kernel: br0: port 2(vnet1) entered forwarding state
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x26@0xc1c
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x27@0xd00
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x25@0xe00
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x25@0x160
Jun  8 10:28:12 ArcticTower  acpid: input device has been disconnected, fd 6
Jun  8 10:28:12 ArcticTower  acpid: input device has been disconnected, fd 7
Jun  8 10:28:12 ArcticTower  acpid: input device has been disconnected, fd 8
Jun  8 10:28:12 ArcticTower  avahi-daemon[2328]: Joining mDNS multicast group on interface vnet1.IPv6 with address fe80::fc54:ff:fe0f:cecc.
Jun  8 10:28:12 ArcticTower  avahi-daemon[2328]: New relevant interface vnet1.IPv6 for mDNS.
Jun  8 10:28:12 ArcticTower  avahi-daemon[2328]: Registering new address record for fe80::fc54:ff:fe0f:cecc on vnet1.*.
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 1/KVM/12762 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 4/KVM/12765 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 3/KVM/12764 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 7/KVM/12768 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 9/KVM/12770 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 8/KVM/12769 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 11/KVM/12772 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 10/KVM/12771 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 13/KVM/12774 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 15/KVM/12776 took a split_lock trap at address: 0x7fde008c
Jun  8 10:29:12 ArcticTower kernel: usb 1-7: USB disconnect, device number 2
Jun  8 10:29:13 ArcticTower kernel: usb 1-7: new low-speed USB device number 14 using xhci_hcd
Jun  8 10:29:13 ArcticTower kernel: input: PixArt USB Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/0003:093A:2510.0007/input/input11
Jun  8 10:29:13 ArcticTower kernel: hid-generic 0003:093A:2510.0007: input,hidraw0: USB HID v1.11 Mouse [PixArt USB Optical Mouse] on usb-0000:00:14.0-7/input0

 

 

I really don't know what could be wrong.

Screenshot 2023-06-08 200018.png

Link to comment
29 minutes ago, Kodon said:

 

I took this to start over from scratch. I reset the BIOS and reformatted the Unraid USB to get rid of all settings.

Leaving everything at default and "ticking the boxes" and... still not working :c

 

vfio-pci Log

 

  Hide contents
Loading config from /boot/config/vfio-pci.cfg
BIND=0000:01:00.0|10de:2684 0000:01:00.1|10de:22ba
---
Processing 0000:01:00.0 10de:2684
Vendor:Device 10de:2684 found at 0000:01:00.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.0
/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/0000:01:00.1

Binding...
Successfully bound the device 10de:2684 at 0000:01:00.0 to vfio-pci
---
Processing 0000:01:00.1 10de:22ba
Vendor:Device 10de:22ba found at 0000:01:00.1

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.0
/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices/0000:01:00.1

Binding...
0000:01:00.0 already bound to vfio-pci
0000:01:00.1 already bound to vfio-pci
Successfully bound the device 10de:22ba at 0000:01:00.1 to vfio-pci
---
vfio-pci binding complete

Devices listed in /sys/bus/pci/drivers/vfio-pci:
lrwxrwxrwx 1 root root    0 Jun  8 16:51 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root    0 Jun  8 16:51 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1

 

 

VM Log

 

  Hide contents
-device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-2-format","id":"sata0-0-0","bootindex":2}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.229-1.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-1-format","id":"sata0-0-1"}' \
-netdev tap,fd=36,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:0f:ce:cc","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=34,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/1-Games-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,websocket=5700,audiodev=audio1 \
-k de \
-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.4","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring
2023-06-08T17:27:08.650464Z qemu-system-x86_64: terminating on signal 15 from pid 2869 (/usr/sbin/libvirtd)
2023-06-08 17:27:08.879+0000: shutting down, reason=shutdown
2023-06-08 17:28:10.970+0000: Starting external device: TPM Emulator
/usr/bin/swtpm socket --ctrl type=unixio,path=/run/libvirt/qemu/swtpm/2-Games-swtpm.sock,mode=0600 --tpmstate dir=/var/lib/libvirt/swtpm/a2694598-c0a6-e077-916f-47926aa66729/tpm2,mode=0600 --log file=/var/log/swtpm/libvirt/qemu/Games-swtpm.log --terminate --tpm2
2023-06-08 17:28:10.983+0000: starting up libvirt version: 8.7.0, qemu version: 7.1.0, kernel: 5.19.17-Unraid, hostname: ArcticTower
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME=/var/lib/libvirt/qemu/domain-2-Games \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-Games/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-Games/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-Games/.config \
/usr/local/sbin/qemu \
-name guest=Games,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-Games/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/a2694598-c0a6-e077-916f-47926aa66729_VARS-pure-efi-tpm.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-7.1,usb=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu host,migratable=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,host-cache-info=on,l3-cache=off \
-m 16384 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":17179869184}' \
-overcommit mem-lock=off \
-smp 16,sockets=1,dies=1,cores=8,threads=2 \
-uuid a2694598-c0a6-e077-916f-47926aa66729 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=35,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x7.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x7"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x7.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x7.0x2"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Games/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.3","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk2","bootindex":1,"write-cache":"on"}' \
-netdev tap,fd=36,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:0f:ce:cc","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=34,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/2-Games-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:01:00.1","id":"hostdev1","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/002","id":"hostdev2","bus":"usb.0","port":"2"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/003","id":"hostdev3","bus":"usb.0","port":"3"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)

 

 

Unraid Log

 

  Hide contents
Jun  8 10:27:08 ArcticTower  avahi-daemon[2328]: Interface vnet0.IPv6 no longer relevant for mDNS.
Jun  8 10:27:08 ArcticTower  avahi-daemon[2328]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe0f:cecc.
Jun  8 10:27:08 ArcticTower kernel: br0: port 2(vnet0) entered disabled state
Jun  8 10:27:08 ArcticTower kernel: device vnet0 left promiscuous mode
Jun  8 10:27:08 ArcticTower kernel: br0: port 2(vnet0) entered disabled state
Jun  8 10:27:08 ArcticTower  avahi-daemon[2328]: Withdrawing address record for fe80::fc54:ff:fe0f:cecc on vnet0.
Jun  8 10:28:10 ArcticTower kernel: br0: port 2(vnet1) entered blocking state
Jun  8 10:28:10 ArcticTower kernel: br0: port 2(vnet1) entered disabled state
Jun  8 10:28:10 ArcticTower kernel: device vnet1 entered promiscuous mode
Jun  8 10:28:10 ArcticTower kernel: br0: port 2(vnet1) entered blocking state
Jun  8 10:28:10 ArcticTower kernel: br0: port 2(vnet1) entered forwarding state
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x26@0xc1c
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x27@0xd00
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x25@0xe00
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
Jun  8 10:28:12 ArcticTower kernel: vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x25@0x160
Jun  8 10:28:12 ArcticTower  acpid: input device has been disconnected, fd 6
Jun  8 10:28:12 ArcticTower  acpid: input device has been disconnected, fd 7
Jun  8 10:28:12 ArcticTower  acpid: input device has been disconnected, fd 8
Jun  8 10:28:12 ArcticTower  avahi-daemon[2328]: Joining mDNS multicast group on interface vnet1.IPv6 with address fe80::fc54:ff:fe0f:cecc.
Jun  8 10:28:12 ArcticTower  avahi-daemon[2328]: New relevant interface vnet1.IPv6 for mDNS.
Jun  8 10:28:12 ArcticTower  avahi-daemon[2328]: Registering new address record for fe80::fc54:ff:fe0f:cecc on vnet1.*.
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 1/KVM/12762 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 4/KVM/12765 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 3/KVM/12764 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 7/KVM/12768 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 9/KVM/12770 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 8/KVM/12769 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 11/KVM/12772 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 10/KVM/12771 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 13/KVM/12774 took a split_lock trap at address: 0x7fde008c
Jun  8 10:28:13 ArcticTower kernel: x86/split lock detection: #AC: CPU 15/KVM/12776 took a split_lock trap at address: 0x7fde008c
Jun  8 10:29:12 ArcticTower kernel: usb 1-7: USB disconnect, device number 2
Jun  8 10:29:13 ArcticTower kernel: usb 1-7: new low-speed USB device number 14 using xhci_hcd
Jun  8 10:29:13 ArcticTower kernel: input: PixArt USB Optical Mouse as /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/0003:093A:2510.0007/input/input11
Jun  8 10:29:13 ArcticTower kernel: hid-generic 0003:093A:2510.0007: input,hidraw0: USB HID v1.11 Mouse [PixArt USB Optical Mouse] on usb-0000:00:14.0-7/input0

 

 

I really don't know what could be wrong.

 

What settings are clicking on the VM create page, create a new VM template and send the screenshot? You could also try a Ubuntu distro as a test. Are you connecting an actual monitor to it or a headless hdmi fake. Some of the symptoms sound like you need one of these HDMI headless

Edited by Nano
Link to comment
2 hours ago, Nano said:

What settings are clicking on the VM create page, create a new VM template and send the screenshot? You could also try a Ubuntu distro as a test. Are you connecting an actual monitor to it or a headless hdmi fake. Some of the symptoms sound like you need one of these HDMI headless

 

Thanks for the input. I am using a "real" monitor, so no kickstart required to trick the GPU to fire up.

 

Here are my settings.

Screenshot 2023-06-08 225023.png

Screenshot 2023-06-08 225119.png

Link to comment

May be beyond me then, that’s basically what I use and works okay, my understanding was that the PCI device is simply passed through here, I assume you have installed all the drivers in windows?

 

right click the start button on windows, click device manager and make sure you have nothing missing/yellow exclamation mark. Otherwise hopefully someone else can help.

Edited by Nano
Link to comment

It's sadly not a simple driver issue.

 

I tried your suggestion to set up a Linux VM. This also does not work. I used the newest Ubuntu and installed the Nvidia drivers beforehand, but also no luck.

 

After switching the card to the physical one I don't get any output at all.

 

Then I had the idea to try the physical card as a secondary GPU... This leads to this message in both the Linux and Windows VNC viewer:

 

Guest has not initialized the display (yet)

 

So also no luck there. Deleting and recreating the VM also doesn't change anything for the better.

 

VM Log shows:

 

qxl_send_events: spice-server bug: guest stopped, ignoring

 

Which also didn't bring me closer to a solution.

Edited by Kodon
Link to comment
  • Solution

The good new is: I got it working!

The bad news is: I have no idea why.

 

I tested if I can safely reproduce the result and yes, it is consistent! :)

 

What I did:

- Flash BIOS to older version (I updated again later, so far no difference so might not have helped at all, but got me started with fresh, stock settings)

- Completely started from scratch with Unraid, reflash USB, deleted all drive partitions, only kept key file

- Only add one drive to the array at first

- Did not install any apps or plugins

 

The above might or might not have helped, I don't know.

 

- Activate IOMMU group for GPU and respective Audio

- Setup new VM with Q35, vDisk qcow2, Virtual GPU

- Install Windows, install all VirtIO drivers and update OS

- Shutdown VM

- Delete VM

- Setup VM again, choosing the already set up vDisk, Q35 and this time with the physical GPU and respective audio device

- Connected a physical monitor, keyboard and mouse, attaching them to the VM

- Switching to XML view and added

<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

And just before </domain> tag:

  <qemu:commandline>
    <qemu:arg value='-fw_cfg'/>
    <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
  </qemu:commandline>

 

After starting the VM everything worked. I have no idea why but it does.

Deleting the VM after having used a virtual GPU might have had the biggest impact.

 

Important: I need to use a monitor or a HDMI dummy dongle to start up my card!

 

Today I added a second GPU (4060ti) and the same steps brought the same success. But I didn't need the XML part.

 

TL;DR for what I think is absolutely necessary for the 4090:

- Activate IOMMU group

- HDMI Dummy dongle or physical monitor

- XML additions

 

I hope this helps someone who runs into the same trouble!

Link to comment
4 minutes ago, Kodon said:

The good new is: I got it working!

The bad news is: I have no idea why.

 

I tested if I can safely reproduce the result and yes, it is consistent! :)

 

What I did:

- Flash BIOS to older version (I updated again later, so far no difference so might not have helped at all, but got me started with fresh, stock settings)

- Completely started from scratch with Unraid, reflash USB, deleted all drive partitions, only kept key file

- Only add one drive to the array at first

- Did not install any apps or plugins

 

The above might or might not have helped, I don't know.

 

- Activate IOMMU group for GPU and respective Audio

- Setup new VM with Q35, vDisk qcow2, Virtual GPU

- Install Windows, install all VirtIO drivers and update OS

- Shutdown VM

- Delete VM

- Setup VM again, choosing the already set up vDisk, Q35 and this time with the physical GPU and respective audio device

- Connected a physical monitor, keyboard and mouse, attaching them to the VM

- Switching to XML view and added

<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

And just before </domain> tag:

  <qemu:commandline>
    <qemu:arg value='-fw_cfg'/>
    <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
  </qemu:commandline>

 

After starting the VM everything worked. I have no idea why but it does.

Deleting the VM after having used a virtual GPU might have had the biggest impact.

 

Important: I need to use a monitor or a HDMI dummy dongle to start up my card!

 

Today I added a second GPU (4060ti) and the same steps brought the same success. But I didn't need the XML part.

 

TL;DR for what I think is absolutely necessary for the 4090:

- Activate IOMMU group

- HDMI Dummy dongle or physical monitor

- XML additions

 

I hope this helps someone who runs into the same trouble!

XML parts are only required if gpu has large memory i.e. 24g.

 

Would you post the full xml for reference?

Link to comment
21 minutes ago, SimonF said:

XML parts are only required if gpu has large memory i.e. 24g.

 

Would you post the full xml for reference?

 

Sure, I have everything else at default though so nothing interesting to see.

I wish the XML settings would stick and not vanish every time I edit the VM...

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Win11</name>
  <uuid>f7db8eca-8847-8146-dcc3-a645d3c46ea5</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>16</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <vcpupin vcpu='8' cpuset='16'/>
    <vcpupin vcpu='9' cpuset='17'/>
    <vcpupin vcpu='10' cpuset='18'/>
    <vcpupin vcpu='11' cpuset='19'/>
    <vcpupin vcpu='12' cpuset='20'/>
    <vcpupin vcpu='13' cpuset='21'/>
    <vcpupin vcpu='14' cpuset='22'/>
    <vcpupin vcpu='15' cpuset='23'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/f7db8eca-8847-8146-dcc3-a645d3c46ea5_VARS-pure-efi-tpm.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='8' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/Win11/vdisk1.img' index='2'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/Win11/vdisk2.img' index='1'/>
      <backingStore/>
      <target dev='hdd' bus='virtio'/>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:00:58:7c'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Win11/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
      <alias name='tpm0'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
  <qemu:commandline>
    <qemu:arg value='-fw_cfg'/>
    <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
  </qemu:commandline>
</domain>

 

Edited by Kodon
Added Code Tags
Link to comment
3 minutes ago, SimonF said:

Is that the qemu parms bit? Will look to add persistance to gui for it.

 

Both required edits for the RTX 4090 are overwritten when editing the VM in GUI mode

 

On top:

<domain type='kvm' id='1'>

Needs to be changed to / stay at

<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

 

Before

</domain>

This needs to be added / stickied

  <qemu:commandline>
    <qemu:arg value='-fw_cfg'/>
    <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
  </qemu:commandline>

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.