ghost82

Change from: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/gtx1660ti.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> to: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/gtx1660ti.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x2'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x3'/> </hostdev> to apply multifunction, re run the command to see if you can list VGA at 04:00.0, not audio, and other gpu devices at 04:00.1, 04:00.2 and 04:00.3 Clean also your sysconfig: from: append vfio-pci.ids=1022:43d5 xen-pciback.hide=(23:00.0)(24:00.0)(25.00.0)(26.00.0) vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot pcie_acs_override=multifunction kvm_amd.nested=1 to append vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=multifunction kvm_amd.nested=1 initrd=/bzroot

same line: append pcie_acs_override=downstream,multifunction video=efifb:off initrd=/bzroot Using another setup and attaching diagnostics is really of no help.

Still something wrong, attach diagnostics related to this output.

Something is wrong here...did you change anything?You have only audio at 04:00.0, no traces of vga, usb and serial bus..and all these things should be at bus 3 not 4..

What's the output of lspci -vv for the gpu inside the vm (rdp server inside the vm + gpu passthrough)?Can you see it?What driver is attached?

🤔

As far as I can see there are no more info in the logs, and all should be setup correctly, so no more ideas.. Check if the vm hangs at boot or if the vm is booting but you have only no video output, maybe an issue inside the vm (drivers)?

You should have something like this in your bios: Advanced/AMD PBS/Enumerate all IOMMU in IVRS Change that setting (ENABLE IT) and see if all iommu groups are enumerated (image taken from internet) As you can see, if it's disabled, only devices attached to the primary cpu die will be mapped. If enabled, devices attached to both dies should be enumerated, in fact now, your iommu groups lack some more devices, not only that gpu.

In your latest diagnostics I cannot see anywhere the video=efifb:off, this is your cmdline: BOOT_IMAGE=/bzimage pcie_acs_override=downstream,multifunction initrd=/bzroot Where did you apply it? Moreover I can see only a gpu passthrough for the 1060 in your windows 11 vm (and settings are wrong because the gpu is not multifunction), not the 1050, what's your goal?

Follow squid comment, attach to vfio at boot the 3080, either set the other gpu as primary and/or be prepared to use an external device to connect via ethernet to unraid to manage it. Once isolated as he said, you should see it in the dropdown menu.

Your diagnostics refer before applying the video=efifb:off kernel parameter. In the logs this is the first issue, boot with video=efifb:off and attach new diagnostics to see if there's something useful.

If you know the source address of the gpu (and you know it: 43:00.0 and 43:00.1) you can always set it manually in the advanced xml view: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x043' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x043' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </hostdev> Note that the target addresses 03:00.0 and 03:00.1 are examples, you need a pci-root-port with 'index=3', and no device attached to bus 3.

Booting unraid in uefi mode should not be an issue to run non uefi gpu with seabios. It may depend on the gpu itself (?). But you say that in the past it worked, so... If you want to try, I see that some uefi bioses for the gtx 670 are available in internet; without flashing the card, you can try and pass that bios to the vm and try ovmf. You could also dump your own vbios (preferred) and add support for uefi gop, with a windows tool called GOP Updater. Your gpu is gk104 (gpu architecture=4): Your modified vbios will be in the same folder of the tool

Which configuration? maybe windows is replacing the drivers when you update? If you got at least once the video output from the gpu, there's no need to manually modify the vm settings

It doesn't seem an issue related to passthrough, but to nvidia drivers and ubuntu. I would suggest to write directly to the ubuntu forum, having a manjaro build I often read about black screens issues with nvidia after updates. Remember that when you passthrough the gpu there's little or no difference at all compared to bare metal, so issues related to bare metal installations can be replicated in or vms. The only thing I would check is the setting of the gpu in the vm, make sure you passthrough all the components of the gpu in a multifunction device.

It should not..but who knows..from your screenshot only the 4 intel emulated controllers are using irq, all the other devices are using msi, so it's not an issue with irq. However msi for some devices can be problematic. Not much to say unfortunately, but if I were you I: 1. disable all msi fixes and see if it changes something 2. if it doesn't change, remove all the passthroughs, except for the nvme (you have the os on it), so remove gpu and usb, and see if it changes something 3. if it's ok add again one at a time the other passthrough, first gpu, then usb, and see if it works 4. only if you have issues in the vm for specific devices (usually gpu and audio, sometimes with network), enable msi, you don't need to enable msi for all devices, having shared irq can be perfectly fine, as it is also in several bare metal builds

Yes, exactly what I was asking. Back to the issue, as I suspected, in your log: 0000:0a:00.0: BAR 3: assigned to efifb 0a:00.0 is your gpu: 0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 [GeForce GTX 970] [1462:3160] Kernel driver in use: vfio-pci Even if attached to vfio at boot, the host is using it for efifb. In unraid gui go to Main - Boot Device - Flash - Syslinux Configuration, you will see several blocks, one with a green label, that is your predefined boot. You need to modify in this block the "append line", by adding video=efifb:off For example, for the "unRAID OS" label it becomes: append initrd=/bzroot video=efifb:off Reboot unraid and try to start the vm. IMPORTANT NOTE: when you reboot unraid, unless you are using another gpu for unraid, if your monitor is attached to the gpu you want to passthrough, the screen will freeze, but unraid is not hanging, connect to unraid from an external device and manage unraid remotely.

diagnostics will help, no diagnostics, no or little help..without looking at them it seems an issue with efifb using your gpu by the host. Attach also the output of cat /proc/iomem from unraid terminal

vfio: Cannot reset device 0000:0b:00.1, depends on group 31 which is not owned. Even if in different iommu groups, maybe you need to attach to vfio also the usb controller at 0b:00.2 and the serial bus controller at 0b:00.3 of the gpu: 0b:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1) Subsystem: NVIDIA Corporation Device [10de:2182] Kernel driver in use: xhci_hcd 0b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1) Subsystem: NVIDIA Corporation Device [10de:2182] Attach them to vfio and change in your ubuntu vm from this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/gtx1660ti.rom'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> to this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/vbios/gtx1660ti.rom'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x2'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0b' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x3'/> </hostdev> Reboot before running the ubuntu vm.

Yes, they are the same. As far as the NvVars file you don't need it, EFI variables are written into OVMF_VARS file. NvVars is needed when the VARS file is not specified and you have only a big read only file for bios (OVMF): in this case ovmf is already splitted into CODE and VARS, the last one being writable.

vga port output never existed in real macs, so it wont work out of the box; it may work by patching connectors with whatevergreen (and I think it's only possible for igpus).

I would suggest to not passthrough the nvme controller, but save a vdisk file into the nvme attached to a virtio controller; it wont be as fast as passing through the controller, but I think you wont notice any particular performance issue. Pass your gpu, fix the multifunction as described above, and attach the new diagnostics if it doesn't boot.

This line has nothing to do with your nvme address. That line specifies the pci address inside the vm (it's the target address, not the source!). You are using a vdisk saved in /mnt/user/domains/Windows 11 - Test/ attached to a virtio controller, at address 03:00.0 in the vm. You can change that address in the xml to something else, not in use by something else, for example 04:00.0 or 05:00.0, or what you want, obviously providing also a pcie-root-port to attach to it. To passthrough the nvme, you simply delete this: <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='writeback'/> <source file='/mnt/user/domains/Windows 11 - Test/vdisk1.img'/> <target dev='hdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </disk> and add this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> Better to isolate at boot the nvme 03:00.0

Make sure the gpu is not used by something else in the host. Update drivers in the vm, latest version is R510 U2 (511.65). If it doesn't work you may need to hide the hypervisor in the xml, quadros are not consumer gpus.

the whole zip.