Random freezes for a month+ and forced shutdowns!

[piratx]

Hello,

 

My unraid is acting really weird for a month now (maybe more). I am having many freezes during the week and I cannot still find the reason of those.

In the meantime I have:

• changed my USB drive to a new one (from Sandisk to Verbatim)
• changed my Parity disk from an old 8TB Red WD to a new 16TB Exos Seagate (SAS)
• changed one my oldest 8TB drive (6y old) to a new 14TB Gold WD

 

After all these and during these changes as well, I have freezes too. Like when I changed my parity drive in the parity building process I had to force reboot the machine to access it again. Same thing for the 14TB that at 55-60% of the rebuild my unraid froze again.

 

Freeze = When I cannot login via webGUI or ssh, but I can still ping it and it's responding to that all the time. I have tried and waited to hit the power button once and wait for it to go to a clean shutdown but a) that might take from 2-5 hours and b) it stucks with timeouts in the process. So mostly my shutdowns was forced..

Also I have some weird errors on the boot, which ends with, those can't automatically be fixed. https://cdn.discordapp.com/attachments/1192048906087968798/1200928304916607096/IMG_0963.MOV?ex=65c7f6a4&is=65b581a4&hm=acf94c3b6eb32c00a2d80276424cabdf2f6520289e8648b9bff04992b4c29d3e&

 

Also attaching my latest log (which I currently save on my flash USB drive)

Thank you so much for reading this and for helping!

 

syslog-previous

[JorgeB]

Nothing obvious that I can see, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. 

[piratx]

I have already tried the safe mode, but it crashed on that one too.

Hardware problem like? One of the disks or worse?

 

Thanks in advance!

[JorgeB]

If it's hardware it can be be anything, RAM, board, CPU, PSU, etc, start by running memtest.

[piratx]

I have tried memtest for 2 hours and all Passed.

Should I do it for a day or something? Thank in advance!

[JorgeB]

Assuming you have at least two RAM sticks, try with just one, if the same try a different one, that will basically rule out a RAM issue, since memtest is only definitive if it finds errors.

[piratx]

I got a new crash last night. I pushed myself to click the power button onces.. it maybe took 4 hours to completely shutdown the machine but I've done it safely (i hope).

Can I upload here the last log? Maybe there's something new on it? 

 

Thanks in advance!

[JorgeB]

17 minutes ago, piratx said:

Can I upload here the last log?

Of course

[piratx]

Here's the -previous log.

 

syslog-previous-2

[JorgeB]

Unraid is running out of memory, if it keeps happening try limiting more the RAM for VMs and/or docker containers, the problem is usually not just about not enough RAM but more about fragmented RAM, alternatively a small swap file on disk might help, you can use the swapfile plugin:

 

https://forums.unraid.net/topic/109342-plugin-swapfile-for-691/

[piratx]

I've used swapfile plugin. I'll leave it run until I change one of the 8tb drives I have with a new 10tb, maybe later tonight. 

I need to do 2 disk changes (2x8TB with 2x10TB). After that I dont have any reason to poweroff/restart it again.

 

Thank you!