Issues: NVME passthrough to Win10 VM

[ne10g]

Hi all!

 

Strange issue occurring here. Looking for advice on what info would help diagnose it if you don't mind?

 

System:

 

ASUS X299 Tuf MK 2 Motherboard

128GB DDR4 RAM

1 x 2 TB NVME (connected direct to motherboard)

2 x 2TB NVME drives connected to an ASUS Hyper M.2 X16 PCIE card (2 ports used - 2 free) (connected to x16 port 2 and set to DATA in BIOS) 

1 x NVIDIA 3090 (connected to x16 Port 1)

1 x PCIe 10Gbps Intel LAN card

1 x PCIe WiFi/Bluetooth card 

 

Symptoms:

 

Windows 10 Pro is installed bare metal on the 2TB NVME that is connected direct to the MB. All drivers installed, all functions perfect when booted direct.

 

In UNRAID I stub the 3090 and the 2TB NVME which is connected direct to the MB.  I pass them both through to the Windows 10 VM.

I amend the XML and add multifunction to the GPU device and change the bus to be the same for Audio and Video.

I pass through the BIOS which I dumped with GPUz and then stripped out the headers.

 

All works perfect, most of the time.  However, the odd time (when writing to the disk/downloading at 7Gbps+ speeds...speed maybe unrelated) windows will freeze and give me a bog standard BSOD.

 

If I restart the VM, gets to the bios screen and is super slow...eventually lets me go into the settings where I would usually select a boot device....but....it is not there!)

 

As soon as it crashes, without fail, the NVME cannot be seen anymore by UNRAID.  If I check the hardware profile which would usually show it as stubbed...not there.

 

If I reboot UNRAID and boot straight back into UNRAID...device is not there.  As if it does not exist.

 

BUT....if I reboot and I select boot device from POST, and select the NVME drive with Windows installed...it will boot into Windows.

 

THEN...if reboot and boot into UNRAID, it is back!

 

Reproducible every single time.  Anyone ever seen anything like this? Any ideas what to check?

 

Nothing obvious in either the VM log or the UNRAID log.

[ne10g]

EDIT: I was wrong above.

 

To get it back I need to POWER OFF the UNRAID server, then power back on.  Simple restarting does not even give me access to the NVME with windows on.

 

So if it crashes, just done it right now, I need to power OFF the server, then back ON and then, and only then, can I access the NVME drive again.