v7.3.3 Windows 11 VMs GPU pass through audio only (black screen) - SOLVED

On ALL prior versions on Unraid I did NOT need to pass-through a vBIOS

on THIS verison of Unraid I MUST pass-through a vBIOS

This is for a Windows 11 VM passing through a 5060 Ti 16 Gb card

Here are the hoops i jumped through...

System Devices:

IOMMU group 17:      (confirm that both are bound)

[10de:2d04] 01:00.0 VGA compatible controller: NVIDIA Corporation GB206 [GeForce RTX 5060 Ti] (rev a1)

[10de:22eb] 01:00.1 Audio device: NVIDIA Corporation GB206 High Definition Audio Controller (rev a1)     

VM Edit:  Graphics Card, Confirm that Multifunction ON

1 – Remote into VM and update to latest NVIDIA Driver (no change).

2 – Reboot Server

3 – Unplug TV Power, Disconnect HDMI cable (from Quick disconnect behind PC), Reconnect HDMI cable, plug TV power back in

4 - BEFORE running VM after clean start

dmesg | grep -i vfio

[    2.957939] VFIO - User Level meta-driver version: 0.3

[   82.960880] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[   82.985634] vfio-pci 0000:01:00.0: Enabling HDA controller

[   83.090072] vfio-pci 0000:02:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[   85.325520] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[   85.325523] vfio-pci 0000:02:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

AFTER running VM (and letting Windows run for a full 10 minutes)

dmesg | grep -i vfio

[    2.957939] VFIO - User Level meta-driver version: 0.3

[   82.960880] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[   82.985634] vfio-pci 0000:01:00.0: Enabling HDA controller

[   83.090072] vfio-pci 0000:02:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[   85.325520] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[   85.325523] vfio-pci 0000:02:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none

[  729.058594] vfio-pci 0000:01:00.0: Enabling HDA controller

[  729.058650] vfio-pci 0000:01:00.0: resetting

[  729.186673] vfio-pci 0000:01:00.0: reset done

[  729.250212] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

[  729.274881] vfio-pci 0000:07:00.0: resetting

[  729.298608] vfio-pci 0000:07:00.0: reset done

[  729.308493] vfio-pci 0000:01:00.0: resetting

[  729.308532] vfio-pci 0000:01:00.1: resetting

[  729.660745] vfio-pci 0000:01:00.0: reset done

[  729.660774] vfio-pci 0000:01:00.1: reset done

[  729.661357] vfio-pci 0000:07:00.0: resetting

[  729.788911] vfio-pci 0000:07:00.0: reset done

[  729.789758] vfio-pci 0000:01:00.0: resetting

[  729.892765] vfio-pci 0000:01:00.0: reset done

dmesg | grep -i vfio

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GB206 [GeForce RTX 5060 Ti] [10de:2d04] (rev a1)

        Subsystem: Gigabyte Technology Co., Ltd Device [1458:4191]

        Kernel driver in use: vfio-pci

        Kernel modules: nouveau

01:00.1 Audio device [0403]: NVIDIA Corporation GB206 High Definition Audio Controller [10de:22eb] (rev a1)

        Subsystem: NVIDIA Corporation Device [10de:0000]

        Kernel driver in use: vfio-pci

        Kernel modules: snd_hda_intel

5 – Remote into VM (Using RustDesk)

·       Device Manager – This device is working properly

·       Nvidia App / System / Display Settings / Display Properties:  Recognizes my TV (LG TV SSCR2)

6 – With TV on, Disconnect HDMI cable (from backside of TV), Reconnect HDMI cable, Select source

7 – Hook up a keyboard to the USB-Hub (passed through).  Confirm it works in Notepad

Force a Graphics Driver Reset

• press Win + Ctrl + Shift + B simultaneously.

• hear a beep as Windows attempt to re-detect and re-handshake with the LG TV.

8 – Under System/Display Turn HDR On / Off (note I see HDR come on Screen, but no video image passes through)

9 - Disconnect HDMI cable (directly from PC), Reconnect HDMI cable, Select source

10 – Run DDU (not in safe mode).  Note that 5060 Ti is listed as GPU #1 and GPU#3.   GPU #2 is the MSFT Basic Display Adapter

Select GPU, NVIDIA, Clean & Restart

…

Restart

Run DDU (not in safe mode).  Note that MSFT Basic Display Adapter is listed as GPU #1 and GPU#2.  

Install latest NVIDA Game Ready Driver (v617.42)

….

Power Down VM

Turn On VM

11 – VM / Edit – Remove the Soundcard, Start VM… didn’t work… FORCE Stop… add sound card back

12 – VM / Edit / GPU / Set Multifunction to OFF… Start VM… didn’t work so powered off VM and set Multifunction to ON

13 – Run Dump_GPU_vBios using SpaceInvaders script as UserScript in the background… which failed

First Update the Script to point to the 5060_Ti

14 – Pass through vBIOS --- SOLVED

GIGABYTE Gaming GeForce RTX 5060 Ti 16GB GDDR7 PCI Express 5.0 x8 ATX Graphics Card GV-N506TGAMING OC-16GD

https://www.techpowerup.com/vgabios/281750/281750

Strip the header with the Portable HxD20 Hex Editor v2.5.0.0

https://mh-nexus.de/en/downloads.php?product=HxD20

Delete everything that starts above the line beginning with U(little 2) wI VIDEO  IBM VGA