In the past month or so, my Unraid has crashed about every 4-6 days.

It has been in use for about 6ish months, and only had these crashes in the past month or so.

 

WebUI does not respond to anything, and the only way to reboot it is by pulling the plug...

It has always happened while I am at work, so I don't know a specific time it happens.

 

I am running 1 VM. Windows Server 2019, with 2GB Ram, 1 CPU core

 

I am also running the following Dockers: Jackett. Krusader. Radarr. Sonarr. CAdvisor. Bazarr. Syncthing. Netdata. Pihole-DoT-DoH. Plex. Unifi Controller (Limited to 2 GB ram). Tubesync

 

Attached is a Diagnostics export.

 

Hope someone has a solution or is able to help.

goga-unraid-diagnostics-20210818-1657.zip

Enable syslog mirror to flash then post the log after a crash.

I've now enabled Syslog Mirror to flash. Let's see when it crashes again. If it crashes again...

 

Server has not crashed yet, but today I got a notification from Fix Common Problems, telling me "Out Of Memory errors detected on your server"

 

Server has 32GB Ram installed.

 

VM Running with 2GB

All running docker containers's memory load added together equals about 61003 mb ram used.

 

According to https://www.linuxatemyram.com, then my system should be fine (I think).

root@GOGA-UNRAID:~# free -m

                   total        used        free      shared  buff/cache   available

Mem:          31987        9016         835         900       22135       21905

Swap:             0           0           0

 

And running dmesg | grep oom-killer shows the following output.

root@GOGA-UNRAID:~# dmesg | grep oom-killer

[129937.393367] cluster-Cluster invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

[256112.858195] mongod invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

 

 

Attached is a new Diagnostics, also containing the syslog.

 

Hope anyone is able to help solve this problem

goga-unraid-diagnostics-20210822-1832.zip

4 hours ago, GoGa_M said:

61003 mb

Where are you getting that number?

 

610003 MB is about 610GB so I assume there is a typo.

I am definitely not good with numbers....

All dockers combined are using about 6GB ram

Today it finally crashed again...

 

Attached is dianostics, and also the syslog that have been running since last time it crashed.

 

Is anyone able to analyze the syslog to see what makes it crash?

 

 

Dont mind the log spam from 192.168.2.13. That is just a Lansweeper server scanning my Unraid.

goga-unraid-diagnostics-20210903-1759.zip syslog

Unfortunately there's nothing relevant logged before the crash, this usually indicates a hardware problem, there are some unrelated ATA errors you should also check, likely a power/connection issue.

Before replacing hardware try downgrading to 6.8.3. I was having similar issues but 6.8.3 has been rock solid now for 35 days

I made a post similar to this a few days ago. I have had issues for quite a while. I updated to rc1 the other day but it also crashed. I'm going to go back to 6.8.3 and see what happens.

Sent from my Mi 9T using Tapatalk

So I have been quite quiet in this thread for the past month(s), but I have been testing some things. I just forget to post my findings... So here we go:

 

It seemed that the server would only crash, when I was taking a backup (using Duplicati or IperiusBackup). But ONLY (as far as I know) when I was backing up data on my cache drives in raid1. And it only crashed about every 2-3 backups. So some backups were running fine.

So I stopped doing backup of the cache for a while, and it ran for 24+ days without crashing. But then it randomly crashed again sometime after... 

 

I tried to replace one of the SSD's in the Raid1. It did have a SMART error, but only due to old age. It was a old SSD anyways.

After the new SSD was working, I tried running a backup, and it crashed again within 2 days...

 

 

Now, I only have 6 SATA ports in my motherboard, but I need 7 in total (5 for HDD Array, 2 for SSD Cache). When I setup the raid1 cache, I purchased a "PCIe to 2x SATA port adapter" (StarTech.com 2 Port SATA 6 Gbps PCI Express SATA Controller Card), and connected one of the HDD drives to the PCIe card. both SSD's were connected to the motherboard.

Yesterday I removed the PCIe card, and now only using one SSD Cache. To see if the PCIe card caused the crash somehow.

I am not sure, but I THINK it all slowly started after I started using Raid1 cache.

 

Ran a full backup of the cache during the night, and the server is still running fine. Ill give ti a couple of days, and run backup every night to see if it crashes again.

 

Lets see what happens in a few days 

Welp, it crashed again today.... So all the things I did, did not help at all

It crashed while I was not home, and I don't know the exact time of the crash.

 

Last thing I'm gonna try is to downgrade it. See if that helps.

Downgrading to 6.9.1. I know that version ran for 60+ days without problems.

 

 

Anyone got any other ideas to try?

Edited November 26, 20214 yr by GoGa_M

Found any solution? 

 

I have the exact same problem...

Every 4-6 days it crashes, im going mental here and cannot find the issue..

 

18 minutes ago, Matt3ra said:

Every 4-6 days it crashes,

Post your diagnostics to see the hardware/config used.

12 minutes ago, JorgeB said:

Post your diagnostics to see the hardware/config used.

Thanks for quick resonse.

 

 

theark-diagnostics-20220210-0853.zip

syslog-192.168.1.39.log1.txt

Edited February 10, 20224 yr by Matt3ra

Nothing jumps out, enable the syslog server and post that after a crash.

7 minutes ago, JorgeB said:

Nothing jumps out, enable the syslog server and post that after a crash.

The attatched file syslog is from that.

The crash was tonight between 9/2-10/2

 

The syslog server doesent seem to give any useble info..  

syslog-192.168.1.39.log1.txt

Edited February 10, 20224 yr by Matt3ra

57 minutes ago, Matt3ra said:

The syslog server doesent seem to give any useble info.

That usually points to a hardware issue, one thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

11 minutes ago, JorgeB said:

That usually points to a hardware issue, one thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

i will try that, thanks alot for the help.

 

But actually i did have the same issue before i moved to another server..

The red thread should then be the USB-stick or one of the drives? Could an faulty drive cause this kind of issue? 

Edited February 10, 20224 yr by Matt3ra

41 minutes ago, Matt3ra said:

The red thread should then be the USB-stick or one of the drives? Could an faulty drive cause this kind of issue? 

Unlikely for that symptom.

Hi All,

 

Sorry I always forget to post my results...

 

Downgrading to 6.9.1 seems to have fixed all the crashing problems.

Nothing more I can tell or have done to fix the problem

Great info.
I will try that next, im up and running for 6 days now so im on a new record  

@Matt3ra - Any update on the crashing? Curious to hear your findings...