i had a similar problem, but could fix it.
I changed my case and the new one was shorter than the old one. My old RX580 was too long and i had to exchange it.
So i decived to go for the MSI Radeon RX 5500 XT MECH OC (8GB). I had no trouble with my old card ASUS ROG STRIX RX580 (8GB) and had even no problems with the reset bug, i was able to force stop the VM and start it again with no problem. I thought it would be the same with the new card, but i was wrong^^
My System:
M/B: ASUS ROG STRIX B450-F GAMING Version Rev 1.xx
BIOS: American Megatrends Inc. Version 3003. Dated: 12/09/2019
CPU: AMD Ryzen 5 1600X Six-Core @ 3600 MHz
Memory: 48 GiB DDR4 (max. installable capacity 128 GiB)
GPU1: MSI Radeon RX 5500 XT MECH OC (8GB) -> Main GPU
GPU2: ASUS PH-GTX1060-6G (6GB)
UnRaid: 6.8.3 nvidia build
Kernel: Linux 4.19.107-Unraid x86_64
So what did i do:
1. doublecheck VM settings
Form view:
Machine: Q35-4.2
BIOS: OVMF
Graphics Card: AMD Radeon RX 5500/5500M (09:00.0)
Sound Card: AMD Navi 10 HDMI Audio (09:00.1)
XML view:
check that the GPU is set to multifunction='on' and Audio is on the same bus as the GPU with function='0x1'
2. vbios
I had to use the vbios for the new card, which wasn't neccessary for the RX580. So i downloaded the vbios for this card from powertechup (https://www.techpowerup.com/vgabios/217516/msi-rx5500xt-8192-191111).
3. Reboot
Changed the VM to autostart and rebooted the whole system. The system booted into unraid, i entered the encryption key and after a few seconds the screen turned black and a few seconds later the windows desktop of the VM appeard with a bad solution^^.
4. Driver
I did know that there was an issue with new radeon drivers and only the version 18.9.3 would work, but this driver was not available anymore. So i thought i give it a try with the last stable driver (20.2.2). Installation worked, but the screen turned every few seconds black an back to normal again, sometimes with artifacts over the whole screen. So this didn't work.
I tried the newest driver (20.4.2), that was worse, but i managed to uninstall the driver again.
Then i found the driver 18.9.3, but i was not able to install it because it did not match witch my system.
Okay so i tried the oldest available driver for the RX5500XT, which was 19.12.2 and i found it here (https://www.amd.com/en/support/previous-drivers/graphics/amd-radeon-5500-series/amd-radeon-rx-5500-series/amd-radeon-rx-5500-xt)
After the installation i had to reboot and here we go, the driver is working, my screens are working with their best solution and i can say that it works so far (running since 3 hours). I did not try any benchmark or stress test but i am optimistic.
P.S. if you want to install the driver 19.12.2 and you use folder redirection of your personal folder like (documents, pictures, etc) you have to change it back to a local path like c:\users\username\... otherwise the installation will fail with error 212, but the error does not specify which folder redirection should be local, so i ended up changing every path back to its original path.
I hope you found a way for you, but maybe this helps others.