I have random crashes of unraid and the fix common problems tool is reporting me hardware fault.
Not sure whats causing the problem. The only update I made recently was upgrading from an older AMD 5700XT to a AMD 6900XT.
I attached the diagnostics and it would be nice, if someone could have an eye on it.
 
I guess this error from syslog is not relevant?
 

Jul 19 08:11:46 Tower kernel: mce: [Hardware Error]: Machine check events logged
Jul 19 08:11:46 Tower kernel: registered taskstats version 1
Jul 19 08:11:46 Tower kernel: mce: [Hardware Error]: CPU 19: Machine Check: 0 Bank 0: baa00000000c0135
Jul 19 08:11:46 Tower kernel: mce: [Hardware Error]: TSC 0 MISC d012000100000000 SYND 4d000014 IPID 1000b000000000 
Jul 19 08:11:46 Tower kernel: Btrfs loaded, crc32c=crc32c-generic, zoned=no, fsverity=no
Jul 19 08:11:46 Tower kernel: mce: [Hardware Error]: PROCESSOR 2:a20f10 TIME 1721369470 SOCKET 0 APIC 7 microcode a20102b

 

Thx in advance!

 

tower-diagnostics-20240719-0842.zip

Server rebooting by itself is almost always hardware issue, and those MCE errors point to the same, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

I already cycled thru my 4 RAM Sticks. But that did not seem to solve the problem. Also MEMTEST did not find any faulty RAM. I know its always apita to figure out which hardware is the problem.

Does the problem continue if you put back the old GPU?

Will try that over the weekend, as I would have to setup all my VMs. I also did a Bios Update, but if so I would blame more the GPU causing the problems.

Esp. as I bought it used and for a fairly low price, marked as working like new. 😕

I have to come up with the problem again.

First I thougt, the problem was due to a defective Power Supply, but I still have randomly crashes of my VMs while working on them.
I occurs in Windows and Mac VMs. If I boot Windows nativly in a windows, I can work for days, without any crashes, so I conclude, it is not a hardware problem.

Is it possible, that a defective USB bootstick leads into power down crashes?
The Problem is, the system turns off, so I do not see any log entries, causing the problem.

Any ideas? Worth to jump on an new USB stick?

By the way, I am on latest 6.12.11, but had the problems with 6.12.10 too.
 

Edited August 12, 20241 yr by derpuma

40 minutes ago, derpuma said:

Is it possible, that a defective USB bootstick leads into power down crashes?

Seems very unlikely to me.

As said, nothing really changed in my system. I also put back the old GPU and used the old VM XML, but I also have crashes there with the old GPU.
From hardware side, this was my only change.
I also checke the NVME disks, where the VMs run from. They seem fine too.
First I suspected a faulty Motherboard, RAM oder CPU. But I have as said, no crashes under native Windows.
I also tested different RAM, so RAM should also not be the problem.
Any idea where to start?
Installing a new unraid and set it up setup by step from scratch with a demo licence and see if that helps?
 
 

19 minutes ago, derpuma said:

Installing a new unraid and set it up setup by step from scratch with a demo licence and see if that helps?

It's worth a try.

So I did. A new fresh install of Undraid, with only VMs, noch dockers or other plugins also leads into fatal power off the machine and reboot.
The funny thing is, I do not need a heavy load or something. Sometime I only use the Browser, surfing the internet, staying up to get a coffee and come back and the machine turned of and rebooted into unraid after a powerloss.

And as said, the system runs pretty stable while I am on native Windows. No crashes or poweroff there, I ran a Cinebench over night and had no hardware problems there at all.

 

I have a last suspect: my NVME M2 Pcie Disk. I run my VMs from that NVME, my native Windows runs from a regular sata SSD.
Whats wondering me, is that there is a complete power shutdown and that the machine starts and immediatly boots up again after this powerloss.
Could this be due to a broken NVME?

The only other thing I have in  mind is, that it could it be a Bios related setting, which leads into crashes while I am using KVM? This happens regardless of windows, mac or linux VM.
So I doubt that it is the XML configuration of the virtual machines, esp. because I run them since a couple of years now with this hardware and zero problems until this mess started.


 

Has this been taken care of?

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

Hi Jorge, thx for the hint.

I am on a Ryzen 5 5950X and Gigaybyte Aorus X570 Pro.
I already tried the Bios setting from my screenshot on "enable".
I can try to disable c-states globally. 
I am wondering, that I never had issues from 2019 to 2023 but I did some bios updates.
Do you think it is worth to go back to an older bios? I already went back from latest beta bios to latest stable release with no changes.
And fyi, I use 4 sticks of 16GB RAM in the right slots. So no messing up with dual channel mode (as dsaid also tried with single RAM stick).
Even XMP on or off shows no difference regarding to the crahes.
 

Edited August 16, 20241 yr by derpuma

On 8/14/2024 at 1:32 PM, derpuma said:

ran a Cinebench over night and had no hardware problems there at all.

It may not fully load CPU, suggest check CPU temp in memtest, if sit at 85c+, you need check the cooling.

CPU temp is not a problem. I use an 360 AOI and Idle around 40°C. Under full load barely over 60°C. A long Cinebench run not nearly up to 80°C.
And as said, the crashes even happen, when the VM sits and does nothing. For ex. I go for a lunch brake, come back and Unraid with VM crashed and restarted into shell.

Edited August 17, 20241 yr by derpuma

Does crash only happen if VM running ?

Seems you have passthrough many device, could you temporary disable GPU passthrough  ?

Also try disable Resizable BAR ( Smart Access Memory ) if BIOS have that.

Edited August 18, 20241 yr by Vr2Io