Unraid random crashes / hardware check errors

[workermaster]

Hi all, 

 

Earlier this year I got my Unraid system running just fine. It was running several Dockers and 2 virtual machines. After getting some trouble with the system (one VM crashing and losing the audio passthrough wich could only be fixed by a full system restart) and wanting to split the system up in 2 systems, I took it all apart and build one gaming system using the hardware that Unraid previously ran on, and a new Unraid server wich now ran on my old gaming hardware. This setup worked just fine. 

 

A few weeks ago, I got a new GPU and decided to try and see if this new GPU and Windows 11 would run smoothly on Unraid. Since it ran perfectly, I decided to consolidate both systems into one. The hardware that I use now is the same hardware as earlier this year. But now it has problems. 

 

I get random crashing. When the system crashes, it locks up. Nothing works and I can't get into Unraid at all. It requires a hard reset. This can have multiple causes but in my case it could be the Ryzen bug. I have followed the guide to disabeling global C states and have turned the PSU mode in the BIOS to "idle power" or something like that. I have also used the fix that Spaceinvaderone posted on Youtube:

 

 

But my system still acts weird. I haven't had a full lockup since disabeling the C states but have had the system randomly reboot. It looked like a clean reboot. I did however, just get this error message:

 

I downloaded the nerdpack, installed mcelog (I have no idea if that also means that it runs because I can't figure out if I need to start that logging thing somewhere) and downloaded the diagnostics. 

 

Does anyone know what the hardware error is?

tower-diagnostics-20210830-2208.zip

[workermaster]

Last night I got this email:

 

I didn't know that mcelog sends emails so it was quite scary to see this. I am wondering how it got my email address and what this error means. 

[trurl]

50 minutes ago, workermaster said:

wondering how it got my email address

Everything on your server that sends emails uses the same process, and it uses the email address you have configured in Notifications. It is good that you get email Notifications, though that one looks like there is something wrong with it. Probably it comes from Fix Common Problems.

 

Those previous Diagnostics were taken just after you installed mcelog.

 

Post new Diagnostics.

 

 

[workermaster]

1 hour ago, trurl said:

Everything on your server that sends emails uses the same process, and it uses the email address you have configured in Notifications. It is good that you get email Notifications, though that one looks like there is something wrong with it. Probably it comes from Fix Common Problems.

 

Those previous Diagnostics were taken just after you installed mcelog.

 

Post new Diagnostics.

 

 

That sounds good. This means that the email system works as it should. 

Here are the new diagnostics. 

 

 

tower-diagnostics-20210831-1503.zip

[trurl]

The MCE are just after boot and don't continue so they can be ignored.

 

Also noticed this in syslog

Aug 31 09:15:08 Tower kernel: traps: Radarr[10112] general protection fault ip:14c1f0b8a6f3 sp:14c1eb2876e0 error:0 in libcoreclr.so[14c1f0849000+36a000]

GPF is an excuse to run memtest.

 

Real question is what is wrong with your cron setup causing the email.

 

What do you get from the command line with this?

 

[trurl]

What do you get from the command line with this?

ls /etc/cron.daily

 

[workermaster]

16 minutes ago, trurl said:

What do you get from the command line with this?

ls /etc/cron.daily

 

Not that much:

 

The server still hasn't crashed but the gaming VM has. It suddenly freezes and after a few seconds, the screen goes black. I get the feeling that I am going to run into every single problem that has ever existed. 

[trurl]

That suggests the user scripts plugin. Do you have any scripts used by that plugin?

 

See if @Squid has an idea.

[workermaster]

8 minutes ago, trurl said:

That suggests the user scripts plugin. Do you have any scripts used by that plugin?

 

See if @Squid has an idea.

I have 3 scripts in there. One of them was for resetting my RX 6800 XT GPU. I have deleted that one since I do not need it. 

There is also another script for removing the 3 transcode limit on the GTX 1050 Ti for Plex. But that one doesn't work I think because I am still limited. 

And there is a script to dump the vbios of my GPU. 

 

At this moment the only script running is the one for removing the limiter of the 1050 Ti and that one only runs at the array startup. 

[workermaster]

My gaming VM with 6 cores and a RX 6800 XT has crashed 3 times today. Unraid itself keeps working just fine. The problem is that when I reboot the VM, the onboard motherboard audio is not working. The VM does not see it. How would I need to go about figuring out why the VM crashes during gaming?

[Squid]

5 hours ago, workermaster said:

I didn't know that mcelog sends emails

This is a known issue with 6.10-rc1 and I believe is fixed on the next release.

 

[workermaster]

21 hours ago, Squid said:

This is a known issue with 6.10-rc1 and I believe is fixed on the next release.

 

I did not get any new mail last night or the last few reboots. 

 

I did get another VM crash and last night Unraid locked up again. Very annoying that I do not have any logfiles. There appears to not be any errors logged. It all just suddenly stops. 

Currently I suspect that the Unraid hard lock / crash is caused by Binhex Syncthing container. The crashes did not happen before that. 

I also think that the VM crashes are mostly caused by buggy games. It happens when I play older assassins creed games but not when running Furmark or Prime95 (or both).

 

I am going to leave the Syncthing container off for a few days to test that. And I am also going to try playing a newer game to see if that solves the crashing. 

 

The last few days have been rough and I am almost at the point of giving up. I have spent over 2 weeks debugging stuff to get it all running and I feel like I am close to being done but I am also very close to giving up and throwing everything in the garbage. 

[workermaster]

Since I have shutdown the Synthing Docker from Binhex, my system has not crashed again last night. So far so good. Will test this for a few more days before I contact Binhex in order to find out what causes this. 

 

I did get another email telling me the same thing as before. Something about the size of a VM not being correct. 

Edited September 2, 2021 by workermaster