Windows VM only works once

[yakboyslim]

I have a windows 10 VM set up on a passed through NVME drive. I am able to start this VM once with GPU passthrough, and it works fine. However, once I shut the VM down it will not start again. No errors, nothing in the VM log that stands out to me. It just won't start again.

 

I think I was able to delete the VM and make a new one and get it back once, but the most recent time this happened required me deleting the libvrt image and rebooting unraid.

 

Diagnostics attached. (My apologies for all the SSH errors in there, I thought I had fixed the issues with my failover pihole's gravitysync, but it appears I did not.)

codraid-diagnostics-20211130-1506.zip

[ghost82]

On 12/1/2021 at 12:10 AM, yakboyslim said:

However, once I shut the VM down it will not start again

Some things to check:

1. fix the topology of the gpu in the xml vm settings.

Change from:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

 

To:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>

 

2. Disable fast startup in windows: https://help.uaudio.com/hc/en-us/articles/213195423-How-To-Disable-Fast-Startup-in-Windows-10

 

3. May be related to the nvidia plugin, but I'm not familiar with this, @ich777 may provide some more info if this 'issue' could be related to this.

He clearly wrote:

Quote

If you only want to use your Nvidia graphics card for a VM then don't install this Plugin!

Since you have 2 nvidia gpus I think you want to use the nvidia plugin for the non passed through gpu, but I don't know how and if it works if you have 2 nvidia gpus.

Edited December 2, 2021 by ghost82

[ich777]

51 minutes ago, ghost82 said:

3. May be related to the nvidia plugin, but I'm not familiar with this, @ich777 may provide some more info if this 'issue' could be related to this.

From what I see this should not affect the functionality from the VM how it's configured on this system, since the GPU that is used in the VM is bound to VFIO so the Nvidia Plugin/Driver can't see it and should be ready for use in a VM.

 

Everything fine from my side.

 

May I ask if you need the GVT-g plugin if you are using it like that?

[yakboyslim]

On 12/2/2021 at 3:24 AM, ghost82 said:

Some things to check:

Thanks for the help. I just moved so this has been on a backburner, but I just tried again with these changes.

 

Same results. Unfortunately now I can't boot into windows at all to disable fast startup.

 

During windows boot I get the Tiano splash for the VM, then I get the spinning wheel for windows. It may go to the recovery options start screen, but no matter what option I choose there it freezes. Usually the spinning wheel reappears and then stops spinning. When this happens if I look at the CPU utilization on the Unraid dashboard at least one thread is pegged at 100%. If I wait long enough other threads will get stuck at 100% too. I have waited over an hour, and this condition never improves, only gets worse with more threads sticking.

 

I have also tried changing the CPU pinning, but this does not improve it. In fact, I have seen where the stuck threads are occasionally not even pinned for the VM. The stuck threads are fairly random, just when I think a certain thread is the problem a different thread will stick first.

 

As one last aside I found a forum post on here about disabling WSD  for samba shares due to random stuck threads. Figured it was worth a shot, but it didn't help.

 

Any help is appreciated, I am so frustrated I am about to buy another computer, when this VM was supposed to be the plan all along.

[ghost82]

I would suggest to not passthrough the nvme controller, but save a vdisk file into the nvme attached to a virtio controller; it wont be as fast as passing through the controller, but I think you wont notice any particular performance issue.

Pass your gpu, fix the multifunction as described above, and attach the new diagnostics if it doesn't boot.

[yakboyslim]

Diagnostics attached.

 

I'll look into not passing through the NVME, but I will need to figure out moving that image to a vdisk. Would this be the cause of the problem? I should add that I had occasional success with restarting this VM, and only recently has it completely given up. So it has worked with the nvme passed through, just not most of the time.

codraid-diagnostics-20220209-1746.zip

[yakboyslim]

So I have narrowed the problem down to hyper threaded pairs. It starts up and runs fine as long as I do not select both pins in a thread pair (i.e. only select 0 OR 1, but not 0 and 1) .

 

This works for me, for now, but is definitely leaving a lot of performance on the table, and especially since the purpose of the VM was for video editing for my wife any CPU tradeoff is unfortunate.

 

I have tried selecting and deselecting the "Hyper-V" setting in the VM. I have not started messing with the BIOS of the Unraid machine itself, but everything was set so hyperthreading should be supported, and definitely is for everything else running on the machine. I am also trying to work a plan to try moving the disk to a vdisk, but with this new revelation I'm not sure if that is my problem any more.

 

Any ideas what could cause this issue? Any settings to try?

[yakboyslim]

Looks like I got it solved. Saw a thread of someone with similar issue who reinstalled unRAID and that fixed it. That didn't work for me, but switching to the next release (6.10 rc) did.

 

Works great now, and the tips above got me audio out too.