TL;DR:

Feb 18 22:47:04 Tower kernel: mce: [Hardware Error]: Machine check events logged
Feb 18 22:47:04 Tower kernel: [Hardware Error]: Corrected error, no action required.
Feb 18 22:47:04 Tower kernel: [Hardware Error]: CPU:0 (19:21:2) MC18_STATUS[Over|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|Scrub]: 0xdc2041000000011b
Feb 18 22:47:04 Tower kernel: [Hardware Error]: Error Addr: 0x0000000ebb89e200
Feb 18 22:47:04 Tower kernel: [Hardware Error]: IPID: 0x0000009600150f00, Syndrome: 0x22ae00020a800d02
Feb 18 22:47:04 Tower kernel: [Hardware Error]: Unified Memory Controller Ext. Error Code: 0, DRAM ECC error.
Feb 18 22:47:04 Tower kernel: EDAC MC0: 1 CE Cannot decode normalized address on mc#0csrow#2channel#1 (csrow:2 channel:1 page:0x0 offset:0x0 grain:64 syndrome:0x2)
Feb 18 22:47:04 Tower kernel: [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD

 

Hardware:

Motherboard: Asus Pro WS X570-ACE

CPU: AMD Ryzen 9 5900X

RAM: 4x32GB DDR4 ECC 3200mhz (Had down-clocked most of the time to 2400mhz as it was an early testing step)

HBA: LSI 9305-16i

NIC: Intel X520-DA2 (Dual 10G SFP)

GPU: NVIDIA GTX 1650

 

I've been fighting random crashing for about a year. Generally it would crash within 2 weeks of a reboot, but then progressively get worse. One thing to note on the crashes, it would NEVER crash during a parity check. At the worst times, it would crash within 12 hours of the parity check finishing. My parity checks take just over 24 hours. When the server was good, it would usually be just after I had to manually power it down (power outage, etc.) that it would start acting up again. I would fuss with it and then it would become "stable" again.

 

History (I'm trying to be as detailed as possible):

I bought my first home back in March and installed Home Assistant. Was using Samsung SmartThings and wanted to expand with the house. I had added a Z-Wave and Zigbee radio USB devices. This is around the time it first started crashing "regularly". As part of troubleshooting I reset the BIOS and set it up again. Ran MemTest64 on the system and no errors. Then I down-clocked the RAM as I remembered I had done that before and the BIOS reset would have set it back to default. Seemed to work better for about a month.

 

My UPS died and when I got everything back up and running it started crashing a new way. When it would crash, I would have an error on the screen saying it couldn't write to the flash drive and I would have to "press any key" and then the system would reboot. The USB drive would not show up in the BIOS after reboot and I would have to completely power off the system or unplug the drive. Thought this might have been because my USB drive was failing as it was around 10 years old at this point. Once replaced, the system started working alright again, but after about 3 weeks, it started crashing again.

 

It went back to just rebooting again. Thought it might have been the new USB drive causing issues, even though it was a brand new one. Replaced with a second drive and things seemed to improve once again. No crashes until... it started again.

 

My next thought was it might have be a faulty USB controller on the motherboard. With HA and the two USB devices, I moved HA to a dedicated machine. This seemed to be a really good improvement. My system was stable for nearly two months. With it being stable I decided I might go ahead and make some improvements to grow.

 

Last weekend, I moved everything over to a new case, and now the issues have returned. Random reboots after the parity check finished. One other thing I had done was upgrade to a 16i HBA instead of my old 4i4e HBA. Started the internal logging server to try to catch errors and found a bunch of the error messages under the TL;DR as shown above. Seeing that, I removed two sticks of RAM bringing my system from 128GB ECC to 64GB ECC to see if that helps. If not, I plan on swapping out the two sticks I removed. I have not ran a MemTest since transferring hardware. One other thing that I had done "recently" was install Immich on the server, so this could also be increasing the memory load on the system.

 

Thank you for reading and helping if you are able.

That error suggests a cache issue with the CPU, this time it was corrected, but if there's an uncorrected error it will crash the server.

 

Also make sure this has been taken care of:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-819173

 

I remember coming across that post and messing with the C-States and DRAM Timings. I definitely remember using that to set the speed. I'm not sure if I did the C-States after doing the BIOS reset a few months ago. I also don't remember the part about the increasing CPU voltage, so I will give that a try as well.

 

Thanks!

did you end up resolving your issue and if so did it involve any voltage adjustments? i have the same CPU suffering random crashes (confirmed is due to CPU).

I had to completely disable the C-States and then everything started working again. I'm back up to 128GB ECC and have been stable. I believe I do have the RAM back at 3200mhz instead of 2667mhz as well.

thanks i feel like i resolved my issue now by adding a voltage offset. i had c-states disabled from the very beginning.