Good evening, 

 

I am trying to run a Windows 10 VM but anytime I restart or shutdown the VM it will not launch again unless I restart Unraid entirely. I can successfully create a Windows 10 VM and it runs fine.  But as soon as I restart the VM or shutdown the VM (gracefully) and attempt to start it again I just receive a black screen and nothing more. I have splashtop and Steam downloaded so if windows was actually booting I would see it. 

 

Troubleshooting steps taken so far. 

1. I have tried removing the VM with and without disks and recreating it. 

2. I have swapped to my Nvidia GTX 1060 and have no issues 

3. I have tried OVMF and Seabios. 

4. I have tried with Q35 and i440fx

5. I am aware that the reset bug should not effect 6000 series AMD GPUs but I ran Spaceinvader script anyway just to check and it didn't fix my issue. 

6. I have tried with and without Vbios. I attempted with Techpowerup Vbios, Self dumped one from Unraid and self dump from plugging GPU into separate working computer and dumping it through GPU-z

7. Tested the GPU on a regular WIndows 10 computer and it works fine. 

8. Tried installing AMD GPU drivers. having them or not causes no difference. 

9. I waited for Windows to install updates before trying to restart. 

10. Switching to VNC allows me to boot no problem

11. Verified IOMMU groups had GPU in own group. Bound GPU/audio to VFIO at boot. Even tried every form of ACS override (Downstream, multifunction, both, etc) 

12. Verified XML has Video/audio on same virtual PCIE slot. 

 

I just don't know where else to go from here. Any help would be appreciated. 

 

Thanks! 

 

server-diagnostics-20220526-2238.zip VM Manager Log.txt

Hi, in your syslog you have this:

May 26 16:26:11 server kernel: vfio-pci 0000:0b:00.0: can't change power state from D0 to D3hot (config space inaccessible)
May 26 16:26:11 server kernel: vfio-pci 0000:0b:00.1: can't change power state from D0 to D3hot (config space inaccessible)

So it cannot be resetted.

 

Since it seems you configured all well, this could be a bios issue.

You are using F50 bios which is quite old, latest bios should be F63b (which should not be a final release?) or F62.

Depending on your motherboard revision:

Rev. 1.0

https://www.gigabyte.com/Motherboard/X470-AORUS-GAMING-7-WIFI-rev-10/support#support-dl-bios

 

Rev 1.1

https://www.gigabyte.com/Motherboard/X470-AORUS-GAMING-7-WIFI-rev-11/support#support-dl-bios

 

Worth a try in my opinion.

Hey ghost82,

 

Thanks for the advice, I went ahead and gave the update a shot. I am now on the latest bios. Unfortunately things only got worse from there. I can no longer boot the VM at all with the GPU. Now when I boot I get a distorted image (Attached)

 

Troubleshooting steps taken since Bios Update:

1. Ensured HVM and IUMMU was still enabled

2. Ensured XML was still correct with GPU Video/Audio still set to multi-function and on the same virtual slot.

3. Ensure IOMMU groups were still bound to VFIO

4. Removed Win10VM with discs and recreated it.

5. Swapped monitors and Displayport cables just in case.

 

Do you have any additional advice for me?

 

 

server-diagnostics-20220527-0854.zip VM Manager Logs 27 May.txt

Edited May 27, 20224 yr by Anthadeas

1 hour ago, Anthadeas said:

Unfortunately things only got worse from there

 

Trust me it should be better now

Now you have this:

May 27 05:50:50 server kernel: vfio-pci 0000:0b:00.0: BAR 0: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]

...and yes because:

May 27 05:28:41 server kernel: pci 0000:0b:00.0: BAR 0: assigned to efifb

 

Go to Main --> flash --> syslinux configuration and edit the "green label":

add to the 'append' line:

video=efifb:off

 

such that it's like:

append blablabl blablalba video=efifb:off

[without blablabl blablalba ]

 

Reboot the server and try

Edited May 27, 20224 yr by ghost82

ghost82,

 

Your right! My day did get better!

 

Your secret sauce code worked. Unfortunately it only allowed me to boot. In fact it even let me shut down and  boot again once. But as soon I installed AMD GPU Drivers I am back to my original issue. If I restart or Shutdown/Boot I just get a black screen again. I think we are back to what you said about the GPU not resetting.

 

Would you happen to have any other special secrets you could share? 😁

 

server-diagnostics-20220527-1220.zip VM Logs.txt

1 hour ago, Anthadeas said:

Would you happen to have any other special secrets you could share?

unfortunately it really sounds a bios bug to me..6000 series cards do not suffer the reset bug, but you're not alone, other users with aorus motherboards are experiencing this issue with 6000 series gpus.

It seems that you can use a reset script to properly reset the gpu, you can read more here:

https://forum.level1techs.com/t/6700xt-reset-bug/181814/36

 

You could also contact asus and report your issue, maybe one day they will release a new bios for your motherboard.

Edited May 27, 20224 yr by ghost82

Well I do appreciate all of your help. I have learned a lot trying all of these steps.

 

In case the next person runs into this post with the same issue, here are a few additional notes I will add.

 

Update 2: GTX 1060 and GTX 780 work without issue again. I made a change to the VM on accident and didn't catch it. Back to the problem going the 6700xt.

Update: I am wrong. My VM with GTX 1060 is also not booting after restart. Looks like you are totally right ghost82. Well at least I know where to start. troubleshooting again.

 

 

1.  I don't think this is a motherboard issue, I installed a GTX 1060 on the same motherboard with and without the 6700xt installed as well and I do not have these same issues. The VM with the 1060 runs fine and I can restart it freely. I could be wrong and the motherboard just isn't playing friendly with AMD GPUs but after following all of the fixes from the above forums post https://forum.level1techs.com/t/6700xt-reset-bug/181814/36 and still not resolving the issue I just don't think it is the motherboard.
2.  I attempted the reset script from the forum post above. The script did not successfully run and it crashed Unraid. So exercise caution if you try that.
3. I do believe after all of the attempted fixes that there is in fact still a reset bug that is effecting some 6700xt models at the least. After removing all of the other variables the 6700xt is the only remaining factor that I can see. Since I am not the only one with these exact issues drives me to this conclusion.

Thanks again for your help ghost82!

Edited May 28, 20224 yr by Anthadeas