Random server reboots/crashes

[HynesJeff]

Let me start off by saying I went through https://forums.unraid.net/topic/37579-need-help-read-me-first/

Because my server is crashing i cant retrieve a diagnostics file from the looks of it. Fix Common Problems is reporting this:

Your server has detected hardware errors. You should install mcelog via the NerdPack plugin, post your diagnostics and ask for assistance on the unRaid forums. The output of mcelog (if installed) has been logged

however I cannot run it as when I do:

mcelog: ERROR: AMD Processor family 23: mcelog does not support this processor. Please use the edac_mce_amd module instead. CPU is unsupported

Google turns up nothing but dead ends for the above to be able to run edac_mce_amd .... 

The post mentions a Troubleshooting Mode in Fix Common Problems but I don't see it and there's no trace of config/logs/syslog.txt on the flash drive.

 

I've been getting these crashes quite often (about every other week or so) all my hardware in my server is basically brand new besides some older drives in the array.

 

Anyone have any suggestions for how I could troubleshoot this?

[HynesJeff]

I found 'Mirror syslog to flash' but i guess that doesn't help me - till the next time it crashes.

[HynesJeff]

https://www.reddit.com/r/archlinux/comments/e33nyg/hard_reboots_with_ryzen_3600x/

I have a similar 3600 and Asrock x570 pro4

[JorgeB]

Start here.

[comet424]

same here my one unraid  it randomly reboots on its own.. as i loose access to the docker.. i also find dockers turn themselves off on there own  on 2 of my unraids

but no system logs and i looked at the start here but that didnt explain anything.. since there is no parity drive on the unraid thats rebooting

it started in the past week that i noticed

[HynesJeff]

On 7/11/2020 at 3:26 AM, johnnie.black said:

Start here.

Thanks! I'm in dual channel 2/4 3200 (I plan on switching to 4 kits of 2666 ecc ram though in the future) so that shouldn't be the problem...

 

I have disabled global c-state control... we'll see if that fixes it.

[HynesJeff]

It's been up for 5 and a half days so far. Much better! The global c-state setting seems to have resolved it. 

[HynesJeff]

Spoke too soon.... Just crashed

[JonathanM]

57 minutes ago, HynesJeff said:

Spoke too soon.... Just crashed

Have you tried slowing down the memory speed?

[HynesJeff]

On 8/21/2020 at 10:00 PM, jonathanm said:

Have you tried slowing down the memory speed?

No, but its only crashed now once since I made the bios change, in the ~2 weeks since ive made it. It made a huge difference.

[donovan7]

HynesJeff,

 

Check your RAM, run MEMTEST64.

 

Explain;

I had a random hardware crash, reboot issue for about 6 months, I looked all over for the issue, but could not figure it out. I ended up running MEMTEST64 and found some bad RAM, it was easy to find out which DIMM it was with isolation. Once I pulled the bad stick out, it was solid for 3 months, RMA'd the ram and all is good again.

[HynesJeff]

its been almost a year but I finally found the root cause to this (Been dealing with random crashes since)

 

All my docker containers used /mnt/user/appdata - i have a lot of containers, and plex being one of them

 

Changing them all to /mnt/cache/appdata not only hugely improved the performance of unraid and my docker containers (and plex from locking up/crashing), but it stopped unraid from crashing!!!!

 

So in conclusion, unraid was crashing unraid. Nothing hardware related at all.

 

I suspect the speed of my SSD (sabrent rocket 4.0), the number of files i have in appdata, my plex directory, and sheer size of my media library overloaded it somehow.

Edited August 3, 2021 by HynesJeff

[HynesJeff]

its been almost a year but I finally found the root cause to this (Been dealing with random crashes since)

 

All my docker containers used /mnt/user/appdata - i have a lot of containers, and plex being one of them

 

Changing them all to /mnt/cache/appdata not only hugely improved the performance of unraid and my docker containers (and plex from locking up/crashing), but it stopped unraid from crashing!!!!

 

So in conclusion, unraid was crashing unraid. Nothing hardware related at all.

 

 

And for anyone that is skeptical:
 

# time find /mnt/user/appdata | wc -l
458909

real    0m33.611s
user    0m0.842s
sys     0m7.161s

vs

# time find /mnt/cache/appdata | wc -l
458910

real    0m0.770s
user    0m0.178s
sys     0m0.635s

 

33.6 seconds vs 0.77 seconds... the difference is HUGE

[itimpi]

Have you checked that none of the ‘appdata’ share is on the main array?   If it is that would explain such a huge difference.

[comet424]

@HynesJeff

 

have you also noticed  that Docker and VMs are completely slow on the top tabs   its like the SSDs goto sleep after a given amount of time and you gotta wait for them to spin up even though they are a SSD  they act like a spinner

so clicking a Docker or a VM  takes about 10 seconds to populate give or take.  on a 3500 or a 7000 nvme like it was a spinner drive but once running   its quick

no one in the forums could answer it. its still and un answered question i asked multiple times..so i wrote the tech support unraid to ask...  about the slowness and the slowness of 3500 and 7000mb/s nvmes  unraid  cant reach those speeds seems to be capped around 2500 and may spike to 3000 at the end of a Disk speed test..  so i asked them as no one could answer it in the forums

i wonder if that would solve the issue..   if i change it from /mnt/user/appdata  to /mnt/cache/appdata    as all my appdata is on the cache but it will go to sleep and i asked this question for a couple years now

 

i think i may have to try.. see if this would help mine..

Edited August 4, 2021 by comet424

[HynesJeff]

On 8/4/2021 at 10:36 AM, itimpi said:

Have you checked that none of the ‘appdata’ share is on the main array?   If it is that would explain such a huge difference.

my appdata is set to cache only but i just confirmed there are no appdata directories on any of my array drives, only on my ssd cache drive.

 

On 8/4/2021 at 10:44 AM, comet424 said:

@HynesJeff

 

have you also noticed  that Docker and VMs are completely slow on the top tabs   its like the SSDs goto sleep after a given amount of time and you gotta wait for them to spin up even though they are a SSD  they act like a spinner

so clicking a Docker or a VM  takes about 10 seconds to populate give or take.  on a 3500 or a 7000 nvme like it was a spinner drive but once running   its quick

no one in the forums could answer it. its still and un answered question i asked multiple times..so i wrote the tech support unraid to ask...  about the slowness and the slowness of 3500 and 7000mb/s nvmes  unraid  cant reach those speeds seems to be capped around 2500 and may spike to 3000 at the end of a Disk speed test..  so i asked them as no one could answer it in the forums

i wonder if that would solve the issue..   if i change it from /mnt/user/appdata  to /mnt/cache/appdata    as all my appdata is on the cache but it will go to sleep and i asked this question for a couple years now

 

i think i may have to try.. see if this would help mine..

What do you mean by "top tabs"? Are you referring to the unraid UI?

[comet424]

@HynesJeff  ya Top Tabs of the Gui

Docker Plugins  VMs, etc

i find it slow  choosing Docker.. i took video how i stop choose a different option after ..  it takes like anywhere from 2 seconds or 10-30 to 1 min  to load up the Docker Page