ghost82

--> video=efifb:off

Thanks, I have no solution, everything is correctly, as far as I can see, configured. The only thing I found is: May 23 20:57:30 Tower kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 May 23 20:57:30 Tower kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x900 May 23 20:57:30 Tower kernel: pmd_set_huge: Cannot satisfy [mem 0xe0000000-0xe0200000] with a huge-page mapping due to MTRR override. which may break the gpu passthrough... Some suggest to boot with 'nohugeiomap' kernel argument (in your syslinux configuration), don't know if it can work..

it could be that the application you are running inside windows is not capable of running on multiple threads.

That's because model is not defined in libvirt. I don't think disk model was implemented in libvirt. Try to add a qemu override and see if that works. So, for serial and model do the following: - define the disk block with your disk, set a serial and give it an alias starting with ua-: <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/path/to/disk.img'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' unit='1'/> <alias name='ua-mydisk'/> <serial>YOURSERIALNUMBERHERE</serial> </disk> - add at the bottom before the domain closing tag a qemu override referring to the alias: <qemu:override> <qemu:device alias='ua-mydisk'> <qemu:frontend> <qemu:property name='model' type='string' value='VMware Virtual IDE Hard Drive'/> </qemu:frontend> </qemu:device> </qemu:override> </domain> And make sure you defined the proper legacy schema at the top of your xml: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

May be caused by virtiofs; try to stop using this and use a samba share for example and see if the crashes stop.

Add <serial></serial> to your disk block: something like this in the xml: <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/path/to/disk.img'/> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' unit='1'/> <serial>YOURSERIALNUMBERHERE</serial> </disk>

No further suggestions and I haven't amd cpu to test.

Something changed with ventura on the usb side. If your goal is to have only mouse and keyboard working I think you can have success in emulating a usb 3 controller (xhci) instead of usb 2 (ehci+uhci) and attaching the dongle to that; obviously you need to make changes to ssdt. I think also you wont have any issue by upgrading from big sur to monterey with the actual config. ps: I remember the exciting days of my first gpu passthrough 4-5 years ago in a high sierra vm...I will never forget those days...

I really don't know what's happening here...XH01 is successfully injected but there's still no trace of the fresco controller. No errors in the diagnostics. Unfortunatly I think I have the same fresco card but I cannot test with Ventura, I'm stuck at Monterey, because my cpu doesn't support avx2 and my gpu requires it.

Backup your efi first of all and your actual xml and make sure to be able to restore them in case of not being able to boot. Having said that, I rearranged the topology, try this: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Ventura</name> <uuid>12345678-aaaa-9cc4-bbbb-123456789658</uuid> <title>Ventura</title> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows.png" os="windowstpm"/> </metadata> <memory unit='KiB'>17301504</memory> <currentMemory unit='KiB'>17301504</currentMemory> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='6'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/mnt/user/system/custom_ovmf/OVMF_EDK2_202302_Stable/OVMF_CODE.fd</loader> <nvram>/mnt/user/system/custom_ovmf/OVMF_EDK2_202302_Stable/OVMF_VARS.fd</nvram> <boot dev='hd'/> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/user/domains/Ventura/patched-vnc-ok_vdisk1.img'/> <target dev='hdc' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/user/domains/Ventura/TimeMachine.img'/> <target dev='hdd' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:3e:21:a2'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='************************'/> <qemu:arg value='-smbios'/> <qemu:arg value='type=2'/> <qemu:arg value='-cpu'/> <qemu:arg value='Haswell-noTSX,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/> </qemu:commandline> </domain> Use the attached efi after replacing the xml. If I did things right...I deleted injected devices in DSDT UHCI2 and UHCI3, injected new device XH01 (fresco) --> see this in SSDT-EHCI, injected usb port mapping kext for EH01 (emulated ehci) and for fresco XH01 and applied correct iMac18,3 smbios for that kext; I hope addresses I put in SSDT are right. XH01 should have 'Name (_ADR, 0x00010003)' in SSDT since now in this xml it's attached to bus 4, meaning pcie root port with index 4, which is on bus 0, slot 1, function 3, which should create automatically a 'pci-bridge@1,3' in ioreg, renamed to XH01 with the SSDT; then I added a subdevice S00 under X01 to identify the fresco controller. EFI.zip

The fresco logic is not appearing at all in ioreg... Can you share the whole efi folder?maybe you are injecting some port mapping kext or ssdt that prevents the card to attach and be detected..other than that I cannot explain this behaviour

Try this (replace ****** at the bottom with the proper content), reattach ioreg and also unraid diagnostics: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Ventura</name> <uuid>12345678-aaaa-9cc4-bbbb-123456789658</uuid> <title>Ventura</title> <metadata> <vmtemplate xmlns="unraid" name="Windows 11" icon="windows.png" os="windowstpm"/> </metadata> <memory unit='KiB'>17301504</memory> <currentMemory unit='KiB'>17301504</currentMemory> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='6'/> </cputune> <os> <type arch='x86_64' machine='pc-q35-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/mnt/user/system/custom_ovmf/OVMF_EDK2_202302_Stable/OVMF_CODE.fd</loader> <nvram>/mnt/user/system/custom_ovmf/OVMF_EDK2_202302_Stable/OVMF_VARS.fd</nvram> <boot dev='hd'/> <bootmenu enable='no'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/user/domains/Ventura/patched-vnc-ok_vdisk1.img'/> <target dev='hdc' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/user/domains/Ventura/TimeMachine.img'/> <target dev='hdd' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xc'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:3e:21:a2'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='************************'/> <qemu:arg value='-smbios'/> <qemu:arg value='type=2'/> <qemu:arg value='-cpu'/> <qemu:arg value='Haswell-noTSX,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check'/> </qemu:commandline> </domain>

Provide ioreg txt file saved from inside your mac os vm, with the fresco logic passed to vfio. May be caused by too many usb ports, you currently have 2 emulated uhci controllers + 1 ehci + 1 uhci controller (emulated) + fresco card. usb 1.1 should be useless in Ventura, no support. Usb port mapping with an injected kext may be required too.

Everything seems ok with the configuration and with the roms (using the dumped one is preferred). I could suggest to upgrade the motherboard bios at this point (you currently have older p1.70), since some older agesa versions were bugged. If and when you upgrade the bios your iommu groups may change, so you need to setup again vfio and device passthorugh.

pcie_aspm=off acts globally and it will disable the power management, pci=noaer will only stop advanced error reporting in the logs. Maybe hardware related, one of your PLX Technology switches.

There are a lots of AER errors, I think the crash happens because of the log filling. The source of the issue could be hardware or software. If I were you, first of all, I would clean the slots and check the cables. If it's software related it could be fixed with a new kernel update. In the meantime to not show these, you can try pcie_aspm=off in your syslinux config. Otherwise you can try pci=noaer

Attach diagnostics with passed vbios, attach vbios.

Well, I can say only, revert back and live with it...consider yourself lucky with these gpus since you are able to reboot windows without crashing all. With bugged firmwares you can have all crazy things happening; for example I have a wifi usb dongle that doesn't work in a kali linux vm if the last vm I boot is mac os; I need to start a windows vm and then start the linux one.

0a:00.6 is the hd audio controller of the igpu. You can't pass that hd audio controller without passing through the igpu, and it seems you don't need to do it, so delete the block I pasted in my previous message. You are passing through the radeon rx, both video and audio parts (01:00.0 and 01:00.1), so audio output will come from the radeon gpu (digital audio). Your gpu isn't working for passthrough, to fix this you need to: - add to your syslinux conf: video=efifb:off because currently BAR0 on 01:00.0 is assigned to efifb. Otherwise you could set the igpu (Cezanne) as primary gpu in the bios, efifb should attach to the igpu and the igpu should be flagged as boot vga (check the syslog): by doing this you should not need to pass a vbios (see next bullet point) - pass a vbios for your radeon, because currenly 01:00.0 is flagged as boot vga - attach to vfio at boot iommu groups 9 and 10

What about if you add invtsc? <smbios mode='host'/> + <features> <acpi/> <apic/> <kvm> <hidden state='on'/> </kvm> <ioapic driver='kvm'/> </features> + <feature policy='disable' name='hypervisor'/> <feature policy='require' name='invtsc'/> </cpu>

I think it wont change anything. You say it's disconnected to unraid but in real it's not, because it's always connected to a driver, nvidia, amd or vfio.

Try to force kvm ioapic driver and see if it's better, replace with this: <features> <acpi/> <apic/> <kvm> <hidden state='on'/> </kvm> <ioapic driver='kvm'/> </features>

I think the amd vendor reset plugin is useless, it shouldn't support the 290x, in fact in your log you have: May 9 20:25:31 X299 kernel: vfio-pci 0000:65:00.0: Unsupported reset method 'device_specific' May 9 20:25:31 X299 kernel: vfio-pci 0000:66:00.0: Unsupported reset method 'device_specific' It could be driver specific or hardware specific. If it's driver specific I can suggest to set the gpu as multifunction in the vms: from your description, the windows driver is able to properly reset the gpu even if audio and video are on different buses, but for some reason linux is not able to put the gpu in a state where windows is able to use it after (hoping that linux is expecting the gpu as a multifunction device). So, change to: for pop-os and win11_1: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x66' slot='0x00' function='0x0'/> </source> <rom file='/mnt/Data_Pool/Dokumente/BIOS Files/MSI290XLightningStock.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x66' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> For win11_2: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x65' slot='0x00' function='0x0'/> </source> <rom file='/mnt/Data_Pool/Dokumente/BIOS Files/MSI290XLightningStock.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x65' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev>

Probably because of this: But you didn't attach diagnostics, so we don't know what it is. If you want to test if the issue is because of this, delete this block: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x6'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev>

Try to enable pcie acs override and see if iommu group 11 can be splitted; then attach to vfio at boot 00:1f.3, your audio, and see if it works.