unRAID Lock-ups/Freezes - 6.8.3


Recommended Posts

I've reported this issue in another thread a couple of weeks ago but for some reason I can no longer find that in my activity history. No one had replied to my issue and it's now becoming more frequent so I'll try again:

 

I am experiencing temporary lock-ups/freezes of my media unRAID server that I'm unable to resolve so far. The freezes occur randomly, as many as 10+ times a day. As shown on the included picture, many of the cores/threads are pegged at 100% utilization, yet TOP shows very little CPU usage. The freeze-ups (everything halts - unRAID, Dockers, VMs) are anywhere from 20 to 120 seconds long.

 

unRAIDLockups.thumb.jpg.ee3eac0ef8fa134473b6ed91560f287f.jpg

 

I've checked the system logs whenever a freeze-up occurs, but nothing other than occasionally seeing disk spin-ups or spin-downs. It's just not consistent enough to lead me to a cause. Since it may be related to drive spin-ups and spin-downs, perhaps it's something to do with the Dynamix Cache Dirs plugin? I didn't see this issue on my i7-6700k w/32GB RAM when I ran unRAID on it, but it started when I migrated this unRAID build to a Supermicro CSE-847 (36 drive storage chassis; specs in my signature below).

 

As mentioned above, these freeze-ups seem to be random in occurrence. No specific time or app/docker/vm that appears to cause it. As my CSE-847 has only 24 drives installed, I tried migrating all my disks to the front 24 bays - I previously had 20 front bays filled and 4 (out of 12) rear drive bays. Taking the rear SATA backplane out of the loop hasn't helped. I've even attached both miniSAS connections from my LSI SAS2008 controller to the front backplane, which in some cases is supposed to improve disk I/O performance. Alas that hasn't been the case for me.

 

Regardless, the slow disk I/O is more likely as many of my drives are at 97% or higher for capacity usage. As most of the disks have filled slowly over time, I still had over 5TB of total free space with the 20 array drives (+ 2 x parity, 1 x cache, 1 x Unassigned Device). I do realize it takes longer to read/write as the drive fills due the increase in bit density as the inner tracks are shorter in length.

 

The speed I can learn to tolerate, but the freeze-ups are what I really want to resolve. I'm planning to replace the Supermicro x8DTN+ motherboard by using a DAS conversion (the Supermicro CSE-PTJBOD-CB3). I'm saving up to build a new host server that's either Threadripper or next-gen Ryzen based, but I'm also looking at some Epyc and Intel Xeon options. I'll be acquiring a new SATA controller with external ports to connect to the CSE-847 once converted to a DAS.

 

I hope that the newer hardware might help resolve this issue, but it would be nice if there was a fix/workaround until then (saving your coins is definitely harder because of our current pandemic-economic woes). If anyone has any thoughts on a possible cause, please share them with me. Thanks!

 

Link to comment
9 hours ago, johnnie.black said:

Start by booting in safe mode and running the server as a basic NAS without dockers/VMs for a couple of days, if it still crashes likely that it's likely a hardware problem, if it doesn't start turning the services on one by one.

That was on my list of things to try. I have previously done an extended memory test for 48 hrs with no errors reported. I'll have to move some of my Docker containers to my backup unRAID - living for a couple of days without them while doing the memtest was a little tough. At least I can manage without the VMs.

 

Note that my 2nd unRAID system has also experienced a similar issue, but it's running on an old i7-980 hexacore setup. It doesn't happen daily or even weekly on the 2nd system - maybe once every couple of weeks. We'll see if it increases in frequency with my 'daily use' Docker containers added.

 

Thanks!

Link to comment
  • 2 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.