ghost82

Issue is with audio at 00:1f.3 you are trying to passthrough.
You attached nothing to vfio at boot, all devices that you want to passthrough should be attached to vfio at boot.
Currently, audio is in the same iommu group with:
00:1f.0 ISA bridge [0601]: Intel Corporation C236 Chipset LPC/eSPI Controller [8086:a149] (rev 31) Subsystem: Dell Device [1028:07c5] 00:1f.2 Memory controller [0580]: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller [8086:a121] (rev 31) DeviceName: Onboard SATA #1 Subsystem: Dell Device [1028:07c5] 00:1f.3 Audio device [0403]: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller [8086:a170] (rev 31) Subsystem: Dell Device [1028:07c5] 00:1f.4 SMBus [0c05]: Intel Corporation 100 Series/C230 Series Chipset Family SMBus [8086:a123] (rev 31) Subsystem: Dell Device [1028:07c5] Kernel driver in use: i801_smbus Kernel modules: i2c_i801  
You may want to apply the acs override patch in unraid to see if it can break iommu group 7 and separate the audio.
 
Or.....do not passthrough audio at all.

From the logs:
Jun 3 12:07:41 MYDEEDEE kernel: pci 0000:0a:00.0: vgaarb: setting as boot VGA device The 1060 is choosen as boot vga by the host.
However, after this, it is successfully attached to vfio, so it should work.
The vm is correctly configured and also the vfio options.
I would try to add also allow unsafe interrupts in settings --> vm
 
Did you try to install the latest nvidia drivers with the gpu passed through connecting to it through remote desktop installed inside the vm?
If you have old nvidia drivers or no drivers installed there could be no output.
Are you 100% sure the vbios is working?
 
Even if the infamous nvidia error 43 could be caused by nvidia detecting the gpu running in a vm (only with old drivers), you may also change in the xml from this:
<features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='1234567890ab'/> </hyperv> <ioapic driver='kvm'/> </features>  
to this:
<features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='1234567890ab'/> </hyperv> <kvm> <hidden state='on'/> </kvm> <vmport state='off'/> <ioapic driver='kvm'/> </features>  
 

I never used seabios, so I don't know, but it could be that you need to enable bootmenu and add a timeout to your xml.
Anyway, I'm seeing that the vm is using seabios 1.15: I compiled the latest version (1.16.0-4-gdc88f9b), maybe you can try this version (attached).
Extract file bios.bin from the zip and save it somewhere on unraid (i.e. /path/to/bios.bin).
Open the vm settings in xml view and change from:
<os> <type arch='x86_64' machine='pc-i440fx-6.2'>hvm</type> </os>  
to:
<os> <type arch='x86_64' machine='pc-i440fx-6.2'>hvm</type> <loader type='rom'>/path/to/bios.bin</loader> <boot dev='hd'/> <bootmenu enable='yes' timeout='30000'/> </os>  
Try to boot.
As you can see I enabled the bootmenu with a timeout of 30 seconds, so you should have enough time to see what boot options it proposes.
Seabios-1.16.0-4-gdc88f9b.zip

There are kernel panic related to amd gpu.
You are running unraid 6.9.2 and you are talking about "6900", is the 6900xt?That is pretty new and if you're using an old unraid version drivers may not play well, try to upgrade to 6.10.1.
Once upgraded check that gpu (audio, video, etc) is bound to vfio.

@astronax I think I found the culprit for the issue "xml is not saving".
I had a spare usb key with unraid 6.10.1, I had also an additional pendrive, so made the array on that usb pendrive  just to test the qemu/libvirt behavior.
Just a note on my above post: one, before using the virsh command should export nano as default editor with this command (in terminal):
export EDITOR='/usr/bin/nano' then run the virsh command.
--------
However it is not needed to run the virsh command and unraid gui in xml view can be used.
 
The issue is that the domain type line is stripped by unraid.
 
When you view your xml in unraid make 2 changes:
1. on the top you will see a line with this:
<domain type='kvm'> Change it to:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>  
Then to the bottom add:
<qemu:capabilities> <qemu:del capability='usb-host.hostdevice'/> </qemu:capabilities> before the </domain> tag.
 
This time it will save.
 
I found this because using the virsh command failed to validate too, because qemu schemas was not defined.
 
PS: not sure if this will solve your bluetooth/wifi issue, just try...

See if this helps:
https://forums.unraid.net/topic/123419-functionnal-vm-wont-boot-anymore-stuck-on-autoboot/
 

If you google enable gpu acceleration over rdp you will find solutions for discrete gpu by tweaking the windows registry if I remember well..this could work also for the igpu...or not...
I know you are now thinking, if you don't know why you reply?
I would not consider RDP at all for your use, if you passthrough a gpu/igpu you are enabling hardware acceleration, so your apps/games should use hardware acceleration but rdp will render the screen output and it will send it over the network; this is how rdp works, vnc for example works in a different way.
I would consider parsec or any other protocol studied for game streaming and avoid rdp, and obviously with a passed through gpu/igpu.

modify it manually instead of with the gui.
Switch to xml view on your vm settings (top right), find the network block, here an example:
<interface type='bridge'> <mac address='aa:aa:aa:aa:aa:aa'/> <source bridge='br1'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> Change model type from virtio or virtio-net to e1000-82545em, save and boot the vm.
 
Post your diagnostics if you have issues.

https://forums.unraid.net/topic/123300-vm-bios-change/

Since you have a dos partition table I think it was a legacy bios installation and not uefi, which requires gpt.
Moreover the 3 partitions reflect that of a legacy bios installation, system, windows, recovery.
Simply use seabios to boot that disk, instead of ovmf.

Nevermind, it was a placebo effect, it seemed it didn't crash for several reboots, then it crashed again.
Thanks to an app to analyze the memory dumps (WhoCrashed) I was able to identify the culprit: it's a driver power failure of my mellanox infiniband card.
It's a very old card, worked well in windows 7, survived windows 10, but it seems it has issue in windows 11. Luckily I don't use it very much and I can disable the card and unloading the driver (ibbus.sys) before rebooting/shutting down.

Change from this:
<driver name='qemu' type='raw' cache='writeback'/> To this:
<driver name='qemu' type='qcow2' cache='writeback'/>  

I would try to switch from i440fx to q35.
You can create a new q35-5.1 vm (ovmf) pointing to actual vdisk, so no need to reinstall windows.

I would suggest to backup your current unraid usb key and upgrade to the latest 6.10.0 RC3.
Apply the same boot arg and see if it gets solved, I wouldn't see any reason to not being able to solve this.
In case of troubles just restore the backup.

Yes, as you can see:
Mar 3 08:50:30 TheWall kernel: pci 0000:01:00.0: BAR 1: assigned to efifb Mar 3 08:52:02 TheWall kernel: vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0x4000000000-0x43ffffffff 64bit pref] ... ... BAR 1 is already in use by efifb, so you need to not make it to attach, or detach on the fly.
 
append initrd=/bzroot video=efifb:off  
Check also your vbios, best way is to dump from linux, but gpu-z is also fine, the important thing is that you do not download one from internet.
If dumped with gpu-z remove the header.

For example, for this:
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev>  
You have first block with source address 03:00.0
You have second block with source address 03:00.1
You have first block with target address 03:00.0
You have second block with target address 04:00.0
 
This is wrong.
 
To fix it, you add multifunction='on' in the first block, and you change in the second block, bus, slot and function, so that it has the same target bus and slot, but different function:
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </hostdev>  
Yes

In that discussion, some user is reporting it working with the following mod, try to add custom qemu arg at the bottom of your xml, before </domain>.
From your picture your pci passthrough should have the alias hostdev1.
 
... ... <qemu:commandline> <qemu:arg value='-set'/> <qemu:arg value='device.hostdev1.x-msix-relocation=bar2'/> </qemu:commandline> </domain>  

You should have something like this in your bios:
Advanced/AMD PBS/Enumerate all IOMMU in IVRS
 
Change that setting (ENABLE IT) and see if all iommu groups are enumerated
 

(image taken from internet)
 
As you can see, if it's disabled, only devices attached to the primary cpu die will be mapped.
If enabled, devices attached to both dies should be enumerated, in fact now, your iommu groups lack some more devices, not only that gpu.

Yes, exactly what I was asking.
 
Back to the issue, as I suspected, in your log:
0000:0a:00.0: BAR 3: assigned to efifb 0a:00.0 is your gpu:
0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 [GeForce GTX 970] [1462:3160] Kernel driver in use: vfio-pci  
Even if attached to vfio at boot, the host is using it for efifb.
 
In unraid gui go to Main - Boot Device - Flash - Syslinux Configuration, you will see several blocks, one with a green label, that is your predefined boot.
You need to modify in this block the "append line", by adding video=efifb:off
For example, for the "unRAID OS" label it becomes:
append initrd=/bzroot video=efifb:off  
Reboot unraid and try to start the vm.
 
IMPORTANT NOTE: when you reboot unraid, unless you are using another gpu for unraid, if your monitor is attached to the gpu you want to passthrough, the screen will freeze, but unraid is not hanging, connect to unraid from an external device and manage unraid remotely.

First settings are correct: q35+ovmf (you have fs0 on hd).
You need to create a custom entry in ovmf.
Start the vm and press esc to boot into the ovmf bios settings, create a new entry for boot, point it to the efi file, adjust boot order, save and reboot.
 

This is the best choice you have.
 
Probably it would fail, one of the things to consider when you use a different vbios is to use one with the exact same memory.

I understood....You need to be fast enough to press any key when the vm boot.
Once in uefi shell, type exit and press enter, then you will see a message to press any key to boot from the dvd, so press any key and the vm will boot from the iso.

yes inside windows.

12.1 was released on 13 december 2021, so it's available.