Still Broken :-( Restarting/Shutting Down a VM w/ GPU passthrough causes the entire system to crash?

[jlficken]

I tried using a 1050 Ti back in 6.8.3 and this happened as well.  I had hoped that moving to 6.9.1 would fix it but I guess not 😞

 

I decided to try it again now that I'm on 6.9.1 and it did it again.  Every time I have to run a Parity Check which takes 25+ hours.

 

 

I've tried both machine types to no avail.  It never does this when I don't have GPU passthrough enabled.  Any thoughts as I'm at the end of my rope with this?

 

 

 

fsnas1u-diagnostics-20210401-1053.zip

Edited May 23, 2021 by jlficken

[jlficken]

So I take it I'm the only one having this problem?

 

Since disabling GPU passthrough I've restarted the VM multiple times as well as shut it down with no issues so I have no idea what the problem is.

 

The ROM file came from my GPU using the SpaceInvaderOne script so I'm hoping that's not the problem.  I've tried the TechPowerUp one for my card as well.

[jlficken]

I give up as I have no idea what to try.  UnRAID is just going to be my NAS from now on and I'll just keep a gaming/work PC around as having the entire server crash when simply shutting down or rebooting a VM is unacceptable.

[jonp]

Hi there,

 

This could be due to the brand of device you're trying to use.  Never heard of Inno3D before.  They may have bad device firmware that doesn't like being assigned to a VM.  I'm assuming that without the device assigned, the VM can be spun up and shut down as many times as you want.  A simple test would be to replace the GPU with an alternate from EVGA or another quality brand (EVGA is recommended) and see if you can reproduce the same result.  If so, there may be more to investigate, but if not, you'll at least know the issue is specific to the hardware you're using.

 

If you want to try and gather more information on what is causing this issue, you'll need to connect a monitor and keyboard to the system's onboard graphics and boot into console mode.  Login to the console with your root account and then type the following command:

 

tail /var/log/syslog -f

 

This will begin printing log events directly to your monitor.  Now recreate the VM crashing issue and I bet you'll see some critical information posted to the screen that you can then take a picture of and post back here for us to analyze.

[jlficken]

Yeah I can't really replace the card as there's no 8-pin/6-pin supplemental power option in the Supermicro CSE-846 which is why I used the Inno3D card as it doesn't require supplemental power and no other company makes a single slot 1050 Ti.  Finding another card really isn't an option right now with the global GPU shortage either.

 

Inno3D is a huge company though and has been around since the 90's.  I remember using their GPU's back when Packard Bell was still around.

https://www.inno3d.com/static.php?refid=1

 

I have IPMI on the server so I can view the console and run that command to see what happens if I can get it to crash.

 

ETA: I also Isolated the CPU cores that are dedicated to the VM yesterday and stopped passing through a USB DVD drive.  I have ordered a PCI-E USB 3.0 card and will pass the entire card through instead when it arrives on Tuesday.

 

Edited April 8, 2021 by jlficken

[lnxd]

7 hours ago, jlficken said:

Inno3D is a huge company though and has been around since the 90's.  I remember using their GPU's back when Packard Bell was still around.

https://www.inno3d.com/static.php?refid=1

😂 Inno3D's heyday was well before me, I only know about them from some LTT videos where they've reviewed "weird" cards. The thermals on that card must be insane, how do you manage the heat?

 

The diagnostics.zip you provided doesn't seem to have the answer in it, having the server crash + needing to reboot means the logs are gone. All that's in there is the logs that show the system booting and data that otherwise suggests your server should be working fine.

 

@jonp's answer should give you the information you need, I wish my "server" had IPMI. That or configuring a remote Syslog Server in Unraid WebUI > Settings > Syslog Server. But it looks like you've found yourself a possible solution.

Edited April 9, 2021 by lnxd

[jlficken]

It happened again.  Any thoughts?

 

[jlficken]

That device is a Mellanox ConnectX-2.  I have a ConnectX-3 as well that's not being used so I swapped them out.

 

I also installed a USB3 card to test hot swap.

 

I still have my fingers crossed I'll get this figured out but I'm getting burnt out on it.

Edited April 21, 2021 by jlficken

[jlficken]

Windows decided to reboot last night to install and update and it seems to have went fine as everything was back up this morning so I'm hopeful that the ConnectX-2 was the issue.  I have no idea why it would have been an issue though.

[jlficken]

It's still broken.  It's totally killed the server twice now when rebooting the VM.

 

How can I get the Syslog file when I can't even get IPMI to respond and it's not on the Flash drive anywhere?

 

IPMI console viewer lets me enter root for the user and then locks up.  All drive activity ceases on the server.  It's just dead....

 

I'm getting so utterly frustrated with unRAID right now that I'm not sure I even want to mess with it anymore.

[Sinister]

i am also having this problem with an evga gtx 960

[jlficken]

It rebooted once after about a week sucessfully.

 

Now after 30 days I get this:

 

It can send an email but otherwise goes unresponsive on anything but the Dockers and Settings tabs.

 

Trying a "shutdown -r now" it just goes away completely and won't even respond to a ping but it never reboots.

 

Would a recording of everything that happens next time be helpful?