GoGa_M Posted August 18, 2021 Share Posted August 18, 2021 In the past month or so, my Unraid has crashed about every 4-6 days. It has been in use for about 6ish months, and only had these crashes in the past month or so. WebUI does not respond to anything, and the only way to reboot it is by pulling the plug... It has always happened while I am at work, so I don't know a specific time it happens. I am running 1 VM. Windows Server 2019, with 2GB Ram, 1 CPU core I am also running the following Dockers: Jackett. Krusader. Radarr. Sonarr. CAdvisor. Bazarr. Syncthing. Netdata. Pihole-DoT-DoH. Plex. Unifi Controller (Limited to 2 GB ram). Tubesync Attached is a Diagnostics export. Hope someone has a solution or is able to help. goga-unraid-diagnostics-20210818-1657.zip Quote Link to comment
JorgeB Posted August 18, 2021 Share Posted August 18, 2021 Enable syslog mirror to flash then post the log after a crash. Quote Link to comment
GoGa_M Posted August 18, 2021 Author Share Posted August 18, 2021 I've now enabled Syslog Mirror to flash. Let's see when it crashes again. If it crashes again... Quote Link to comment
GoGa_M Posted August 22, 2021 Author Share Posted August 22, 2021 Server has not crashed yet, but today I got a notification from Fix Common Problems, telling me "Out Of Memory errors detected on your server" Server has 32GB Ram installed. VM Running with 2GB All running docker containers's memory load added together equals about 61003 mb ram used. According to https://www.linuxatemyram.com, then my system should be fine (I think). root@GOGA-UNRAID:~# free -m total used free shared buff/cache available Mem: 31987 9016 835 900 22135 21905 Swap: 0 0 0 And running dmesg | grep oom-killer shows the following output. root@GOGA-UNRAID:~# dmesg | grep oom-killer [129937.393367] cluster-Cluster invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [256112.858195] mongod invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Attached is a new Diagnostics, also containing the syslog. Hope anyone is able to help solve this problem goga-unraid-diagnostics-20210822-1832.zip Quote Link to comment
trurl Posted August 22, 2021 Share Posted August 22, 2021 4 hours ago, GoGa_M said: 61003 mb Where are you getting that number? 610003 MB is about 610GB so I assume there is a typo. Quote Link to comment
GoGa_M Posted August 23, 2021 Author Share Posted August 23, 2021 I am definitely not good with numbers.... All dockers combined are using about 6GB ram Quote Link to comment
GoGa_M Posted September 3, 2021 Author Share Posted September 3, 2021 Today it finally crashed again... Attached is dianostics, and also the syslog that have been running since last time it crashed. Is anyone able to analyze the syslog to see what makes it crash? Dont mind the log spam from 192.168.2.13. That is just a Lansweeper server scanning my Unraid. goga-unraid-diagnostics-20210903-1759.zip syslog Quote Link to comment
JorgeB Posted September 3, 2021 Share Posted September 3, 2021 Unfortunately there's nothing relevant logged before the crash, this usually indicates a hardware problem, there are some unrelated ATA errors you should also check, likely a power/connection issue. Quote Link to comment
Tristankin Posted September 4, 2021 Share Posted September 4, 2021 Before replacing hardware try downgrading to 6.8.3. I was having similar issues but 6.8.3 has been rock solid now for 35 days Quote Link to comment
mkono87 Posted September 5, 2021 Share Posted September 5, 2021 I made a post similar to this a few days ago. I have had issues for quite a while. I updated to rc1 the other day but it also crashed. I'm going to go back to 6.8.3 and see what happens. Sent from my Mi 9T using Tapatalk Quote Link to comment
GoGa_M Posted November 24, 2021 Author Share Posted November 24, 2021 So I have been quite quiet in this thread for the past month(s), but I have been testing some things. I just forget to post my findings... So here we go: It seemed that the server would only crash, when I was taking a backup (using Duplicati or IperiusBackup). But ONLY (as far as I know) when I was backing up data on my cache drives in raid1. And it only crashed about every 2-3 backups. So some backups were running fine. So I stopped doing backup of the cache for a while, and it ran for 24+ days without crashing. But then it randomly crashed again sometime after... I tried to replace one of the SSD's in the Raid1. It did have a SMART error, but only due to old age. It was a old SSD anyways. After the new SSD was working, I tried running a backup, and it crashed again within 2 days... Now, I only have 6 SATA ports in my motherboard, but I need 7 in total (5 for HDD Array, 2 for SSD Cache). When I setup the raid1 cache, I purchased a "PCIe to 2x SATA port adapter" (StarTech.com 2 Port SATA 6 Gbps PCI Express SATA Controller Card), and connected one of the HDD drives to the PCIe card. both SSD's were connected to the motherboard. Yesterday I removed the PCIe card, and now only using one SSD Cache. To see if the PCIe card caused the crash somehow. I am not sure, but I THINK it all slowly started after I started using Raid1 cache. Ran a full backup of the cache during the night, and the server is still running fine. Ill give ti a couple of days, and run backup every night to see if it crashes again. Lets see what happens in a few days 1 Quote Link to comment
GoGa_M Posted November 26, 2021 Author Share Posted November 26, 2021 (edited) Welp, it crashed again today.... So all the things I did, did not help at all It crashed while I was not home, and I don't know the exact time of the crash. Last thing I'm gonna try is to downgrade it. See if that helps. Downgrading to 6.9.1. I know that version ran for 60+ days without problems. Anyone got any other ideas to try? Edited November 26, 2021 by GoGa_M Quote Link to comment
Matt3ra Posted February 10, 2022 Share Posted February 10, 2022 Found any solution? I have the exact same problem... Every 4-6 days it crashes, im going mental here and cannot find the issue.. Quote Link to comment
JorgeB Posted February 10, 2022 Share Posted February 10, 2022 18 minutes ago, Matt3ra said: Every 4-6 days it crashes, Post your diagnostics to see the hardware/config used. Quote Link to comment
Matt3ra Posted February 10, 2022 Share Posted February 10, 2022 (edited) 12 minutes ago, JorgeB said: Post your diagnostics to see the hardware/config used. Thanks for quick resonse. theark-diagnostics-20220210-0853.zip syslog-192.168.1.39.log1.txt Edited February 10, 2022 by Matt3ra Quote Link to comment
JorgeB Posted February 10, 2022 Share Posted February 10, 2022 Nothing jumps out, enable the syslog server and post that after a crash. Quote Link to comment
Matt3ra Posted February 10, 2022 Share Posted February 10, 2022 (edited) 7 minutes ago, JorgeB said: Nothing jumps out, enable the syslog server and post that after a crash. The attatched file syslog is from that. The crash was tonight between 9/2-10/2 The syslog server doesent seem to give any useble info.. syslog-192.168.1.39.log1.txt Edited February 10, 2022 by Matt3ra Quote Link to comment
JorgeB Posted February 10, 2022 Share Posted February 10, 2022 57 minutes ago, Matt3ra said: The syslog server doesent seem to give any useble info. That usually points to a hardware issue, one thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. Quote Link to comment
Matt3ra Posted February 10, 2022 Share Posted February 10, 2022 (edited) 11 minutes ago, JorgeB said: That usually points to a hardware issue, one thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one. i will try that, thanks alot for the help. But actually i did have the same issue before i moved to another server.. The red thread should then be the USB-stick or one of the drives? Could an faulty drive cause this kind of issue? Edited February 10, 2022 by Matt3ra Quote Link to comment
JorgeB Posted February 10, 2022 Share Posted February 10, 2022 41 minutes ago, Matt3ra said: The red thread should then be the USB-stick or one of the drives? Could an faulty drive cause this kind of issue? Unlikely for that symptom. Quote Link to comment
GoGa_M Posted February 13, 2022 Author Share Posted February 13, 2022 Hi All, Sorry I always forget to post my results... Downgrading to 6.9.1 seems to have fixed all the crashing problems. Nothing more I can tell or have done to fix the problem Quote Link to comment
Matt3ra Posted February 16, 2022 Share Posted February 16, 2022 Great info. I will try that next, im up and running for 6 days now so im on a new record Quote Link to comment
thestraycat Posted February 24, 2022 Share Posted February 24, 2022 @Matt3ra - Any update on the crashing? Curious to hear your findings... Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.