ghost82

And what is the error?

Hi Simon, never had a igpu, so not expert, however that line refers to the header of the rom; kernel is expecting the header 0xaa55, but it finds 0xfc44 instead. Are you passing a vbios? If so, first thing to check is the vbios file: open it with a hex editor and check the first 2 bytes: does it start with "55 AA" (0xaa55 is aa55 in little-endian)? If it doesn't start with 55 AA the vbios could not have been dumped correctly. 55 AA are the bytes for standard pci roms. Update: sorry, I read now you are not passing a rom file... Sometimes this happen if the kernel tries to extract and load the vbios, but it is "masked"; most of the times that is because the igpu is in use by something. Is it possible to have a vbios dump? This could not be an "error", most of the time that line refers to expecting 0xaa55, got 0xffff, which can be totally fine, for example if the vbios has been modded to remove the efi image, or not, if the kernel can see nothing (FF FF). However it sounds a little strange the 0xfc44 value...Again, if the extracted rom is ok, even with that signature value, that is only a warning, and you should have the passthrough working. In real I don't think this is an issue, igpu rom should not be a "standard pci rom", so lacking 55 AA.. Looking around it seems only few people had some luck in passing through to linux guests, as far as I know no luck with windows (till now).

I would suggest to backup your current unraid usb key and upgrade to the latest 6.10.0 RC3. Apply the same boot arg and see if it gets solved, I wouldn't see any reason to not being able to solve this. In case of troubles just restore the backup.

I think he means 6.10RC2, if you're on 6.9.2 you need the patch.

I'm afraid but I think it is, your logs point to the rmrr issue: Qemu log: pci,host=0000:07:00.0,id=hostdev0,bus=pci.3,addr=0x0: vfio 0000:07:00.0: failed to setup container for group 11: Failed to set iommu for container: Operation not permitted Libvirt log: qemu-system-x86_64: -device vfio-pci,host=0000:07:00.0,id=hostdev0,bus=pci.3,addr=0x0: vfio 0000:07:00.0: failed to setup container for group 11: Failed to set iommu for container: Operation not permitted Syslog: Tower kernel: vfio-pci 0000:07:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. I read that a patch is included in >=RC2. The patch has some print instructions, "Intel-IOMMU: assuming all RMRRs are relaxable. This can lead to instability or data loss", if it's applied: https://github.com/kiler129/relax-intel-rmrr/blob/master/patches/add-relaxable-rmrr-5_8_and_up.patch but there is no trace in your syslog.

This is HP Proliant, so: vfio-pci 0000:07:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. You have this boot arg: intel_iommu=relax_rmrr But I think you need kernel patch. Here some readings:

No problem man! happy you solved in some way your issue! And happy that I helped a retro game fan!!I'm going for 40s and I miss that cool games played natively

I don't understand this, can you rephrase?

If this is the issue, it could be a bug (?) in the rc3. I think you could always use terminal to edit the xml of the vm with the 'virsh edit' command. If the alias is lacking try to manually add it with terminal: virsh edit HEREVMNAME Then in the editor add the alias in the appropriate device section: <alias name='YOURALIASHERE'/> Save and exit the editor. Double check again with virsh edit that the alias has being added. Close the editor and try.

Allow unsafe interrupts should not prevent the os to load... Maybe you messed with uefi/legacy bios? Check the mode unraid is booting (uefi vs legacy bios) and check the bios options (csm disabled vs csm enabled).

Last command you typed, "genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:07:00.0)) vs. 00000080 (ehci_hcd:usb1)", is not a command, that's why you got the syntax error, it is a line reported in your logs, to explain you what happened. It should be the same, unchecking is fine too! Yes, they have to be added to the vm (vfio), but not attached to vfio at boot. With this config vfio doesn't claim that device at boot, so it attaches to the host driver (if any); when you boot the vm the device detaches from the host driver (if any) and attaches to vfio: that's what I mean for 'on the fly'. Yes, because everytime you boot the bios detects the usb controller (and so, unraid); you may use unraid user scripts to automate it. Otherwise, you may have an option in bios to enable/disable usb controllers (I have it in my bios): by this way you can disable it in bios, so you don't have to type the command on every boot. But take care that disabling devices in bios may re-arrange all irqs, so the capture card may conflict with something else (or not...this must be checked directly). This could be because the "second device" is claiming the same irq as the first one, if this is the case I can't see any option to fix it.

Ok, from your syslog you have: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:07:00.0)) vs. 00000080 (ehci_hcd:usb1) So, a usb 2.0 controller is using irq 16, confirmed by /proc/interrupts, the same irq that the card is trying to use. P.S: I won the bet!! From your lspci output you have 2 usb 2.0 controllers using the ehci driver: 00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 04) Subsystem: Dell 8 Series/C220 Series Chipset Family USB EHCI [1028:05a5] Kernel driver in use: ehci-pci 00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 04) Subsystem: Dell 8 Series/C220 Series Chipset Family USB EHCI [1028:05a5] Kernel driver in use: ehci-pci So try to remove one at a time in unraid terminal; try to start with 00:1d.0, since from your proc/interrupts we read usb1 and the output of lspci is USB EHCI #1 (but not sure if it's related or not..): echo -n 1 > "/sys/devices/pci0000:00/0000:00:1d.0/remove" Try to start the vm. If it still output the error, try with the other one: echo -n 1 > "/sys/devices/pci0000:00/0000:00:1a.0/remove" Try to start the vm. The other device, 07:04.0, is trying to be assigned to the same irq 16, so it should be ok (or not...I'm not sure, if this needs another irq and may conflict with 07:00.0): genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:07:04.0)) vs. 00000080 (ehci_hcd:usb1) Note: take care to where your unraid usb is plugged in: if you want to avoid issue, plug in into a usb 3.0 port, otherwise you have to map the usb ports to know to which controller they belong, and don't plug in into a port belonging to a controller that you are going to disable/remove. Note2: you may need to remove 07:00.0 and 07:04.0 from vfio-pci.cfg so that they don't attach to vfio at boot; that's because at boot you still have the problematic usb controller detected and I think that when they try to attach to vfio you will get the error. If it's the case detach from vfio at boot and let them attach to it 'on the fly' (i.e. when you run the vm, after you disabled the usb controller).

That is something you cannot fix in unraid or linux or windows, at least for me, and I think it cannot be fixed in the bios too, unless you have some settings to manually assign interrupts. Irq interrupts are assigned automatically depending on your motherboard, some irq are shared, there are a lot of cases of irq shared with usb, ethernet, sata controllers. In most cases thia is not an issue, sometimes it is, for example crackling audio could be caused by shared irq between audio and something else (usb, ethernet, etc), and in your specific case, since you can see that your capture card doesn't support INTx disable. Sometimes, for pcie devices, changing the slot changes also the assigned irq. Otherwise your only option is to remove the device sharing its irq with your capture card, from within linux, as I wrote. I suggest you to read something abour irq if you don't know what they are, at least to know the basics.

This is non sense, it expects 0xaa55 and got 0xaa55, so what's the problem? Are you sure you pasted it right? If the gpu is isolated and the vbios extracted correctly, if you open it in a hex editor first hex values should be 'AA 55'. If the gpu is grabbed by something else, part of the vbios will be masked, but the error will be different, something like 'expecting 0xaa55, got 0xffff'. Similar error if the vbios flashed to the card was modded. Btw, this is not an 'error', but an harmless warning. Is the gpu attached to vfio at boot?Are you sure it doesn't attach to efifb? That should be no difference... RX460 needs reset patch, so removing the plugin without having any reset patch is not a good idea.

You can use this script to check if INTx disable is available or not: #!/bin/sh # Usage $0 <PCI device>, ex: 9:00.0 INTX=$(( 0x400 )) ORIG=$(( 0x$(setpci -s $1 4.w) )) if [ $(( $INTX & $ORIG )) -ne 0 ]; then echo "INTx disable supported and enabled on $1" exit 0 fi NEW=$(printf %04x $(( $INTX | $ORIG ))) setpci -s $1 4.w=$NEW NEW=$(( 0x$(setpci -s $1 4.w) )) if [ $(( $INTX & $NEW )) -ne 0 ]; then echo "INTx disable support available on $1" else echo "INTx disable support NOT available on $1" fi NEW=$(printf %04x $ORIG) setpci -s $1 4.w=$NEW Save it into an .sh file, chmod +x, and run it with: ./check.sh 07:04.0

You should check if this device supports 'INTx disable', I think it doesn't. Without 'INTx disable' the device needs its own irq, if another device is using the same irq it won't work: that's probably why you have it working on some builds and not in others, because this depends on the mb layout and shared irqs. You should check what device is using the same irq and disable it, or adjust your device layout so that this device has its own irq. To remove the offending device you can do it from the terminal, something like: echo -n 1 > "/sys/devices/pci0000:00/0000:00:1b.0/remove" 00:1b.0 being the offending device. If I had to bet I would say some usb controller is sharing the same irq....

Did you try to reboot your router?

vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform So I would try to add to syslinux config: vfio_iommu_type1.allow_unsafe_interrupts=1 ---> For unraid without gui it becomes: append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot It is also recommended to attach to vfio at boot iommu groups 14 and 15, gpu video and audio.

You don't need a second pc to dump the rom; if it fails to dump it probably means that you are not able to detach it from the host. You may check your syslog and see if the gpu is in use by efifb (just search for "efifb" and see if it's attached to your amd gpu) and add to your syslinux config video=efifb:off. Using a downloaded rom can work if you know what you are doing, but if the rom is not that corresponding to your gpu (even a rom with changed gpu revision version can fail) it will fail. If most of applications fail to start maybe you haven't installed redistributable packages (visual c++, net frameworks), less probable that you have corrupted filesytem, less probable you have an av blocking some dlls. This isn't very important, on most builds it is fixed with kernel boot-args (see above video=efifb:off); if you are booting unraid in uefi mode, the primary gpu is attached to efifb for console video output, so it gets grabbed by the host, setting it as secondary prevents so without the need to detach the gpu from the host. If the pc bios/firmware is bugged, last option is to change the physical pcie slot, usually pcie priority goes from top to bottom, meaning that a gpu in top slot has priority over a gpu in a slot below the top one.

Hi, your syslog ends with something related to the gnif patch logged. Also your qemu log reports something related to 05:00.1 not being able to reset: 2022-03-08T09:08:25.543430Z qemu-system-x86_64: vfio: Cannot reset device 0000:05:00.1, no available reset mechanism. On gnif github I remember I read some users complaining that on some gpus (5000 series if I remember well) audio reset wasn't working properly**. Since your 05:00.1 is gpu digital audio, can you try to passthrough only the video without the audio (keeping it attached to vfio at boot) to the vm and see if it crashes? From this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </hostdev> To this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> ------ This may be helpful, how do you run it?Any vendor reset patch?Which version?ich777 patch should be version 1.1 **Update: here a link: https://github.com/gnif/vendor-reset/issues/29

That error means that the gpu didn't reset correctly. A full host reboot may fix it. Some nvidia gpus may suffer this if they are not set up as multifunction devices in the vm.

Hi, you have 23 vms configured, telling us what is the "gaming vm" would have speed up things I think you are referring to the "Windows 10" vm. Try to: 1. set gpu as multifunction, change from: <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x2e' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/RX5700XT.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x2e' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> To: <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x2e' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/RX5700XT.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifuntion='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x2e' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> 2. RX5700XT should have been dumped from your gpu, in linux or with gpu-z. 3. Bind to vfio the gpu at boot, iommu groups 37 and 38: actually the amd gpu is set as "boot VGA device", so you will lose unraid video output, be prepared to connect from an external device via network, or if you have options in bios change the boot VGA to the nvidia gpu. 4. Reboot unraid.

Feb 27 13:47:31 IchiTheTower kernel: efifb: probing for efifb Feb 27 13:47:31 IchiTheTower kernel: efifb: framebuffer at 0x4020000000, using 6144k, total 6144k 4020000000-4031ffffff : PCI Bus 0000:01 4020000000-402fffffff : 0000:01:00.0 4030000000-4031ffffff : 0000:01:00.0 From the logs efifb seems to be assigned at 0x4020000000, but in iomem there's no trace of efifb in that range of addresses. I don't think that assigning a vbios could help, since the issue here is that memory is already in use. Logs say that Feb 28 08:39:48 IchiTheTower kernel: vfio-pci 0000:01:00.0: BAR 1: can't reserve [mem 0x4020000000-0x402fffffff 64bit pref] vfio can't reserve memory 0x4020000000-0x402fffffff, which from the logs it's assigned to efifb, but as I said no trace of efifb in memory. efifb should be completely disabled with video=efifb:off, but in this case something odd is happening. Can't help more with this.. As a last resort: 1. backup your unraid usb key so you can revert changes in case of issues 2. since efifb seems to be the issue and since it starts only if unraid is booted in uefi mode, I would try to disable UEFI boot mode for unraid; note: you may need to enable csm in your bios to be able to boot legacy bios oses. By this way efifb should be out of the game. Try to see if you can get a video output, if it still doesn't work you you want to try this, attach new diagnostics after booting in legacy mode instead of uefi.

Inside diagnostics there's no vm configured, so it's impossible to see what is happening.

You need to chroot into the disk, boot from the installer media and chroot into the disk.