Server randomly locks up

[Treytor]

Hey all,

 

Linux newbie here, apologies in advance for my ignorance.

 

I've been having trouble recently with my server (unraid 6.1.8 ). I wish I could identify exactly when this started, but I can't. It seems to happen randomly or when I'm doing a lot of file transfers. But I can't pinpoint specifically what might cause it to happen. It has happened while actively copying something, during a move procedure from the cache disk, or in the middle of the night without anything going on (that I know of).

 

I only have a plex docker running on it, and nothing else. The strange thing is the plex docker still runs, as long as I don't actually try to play anything.

 

I lose access to shares (windows), and the web GUI becomes unresponsive. I'm unable to gracefully shutdown the server via telnet as well. Disks 3 and 6 report as being busy when I try to unmount them.

 

Running "fuser -mv /mnt/disk* /mnt/user/*" locks up and doesn't report anything.

 

Running "top" reports 100% cpu activity with the "shfs" command.

 

I tried running a reiserfsck on disks 3 and 6. One reported no problems, another had two minor errors that were resolved, but didn't seem to help.

 

If I try to run "diagnostics" from telnet it get stuck and never completes and I have to do a hard reset, which I did just now. Here is the diagnostics information after it reboots and comes back online. Any ideas?

 

Thanks in advance!

cooper-diagnostics-20160305-1017.zip

[Treytor]

Just noticed there's a new version of unraid. Updated to 6.1.9 now and will post a new diagnostics in case that changes anything.

 

Aaand here you go

cooper-diagnostics-20160305-1026.zip

[Treytor]

It just happened again after being up for a couple days. It was while I was copying a lot of data off the server (over 1tb).

 

Same thing, everything unresponsive, but still able to telnet in. First thing I tried is a 'diagnostics' this time, and it's stuck "Starting diagnostics collection"

 

I'm able to log in with another telnet session and re-confirm all my findings in my original post. Unable to unmount disks 3 and 6.

 

I ran a memtest earlier today and no errors came up.

[ashman70]

Tell us about your system, the hardware, what dockers and plugins you have? When was the last time you did a parity sync?

[Treytor]

My setup is pretty standard:

 

Supermicro X8SIL-F-O

4GB ECC ram (soon to be upgraded to 16GB)

Intel i7 CPU 860 @ 2.80GHz

20 mixed size drives (2-5TB) totaling 60TB

250GB SSD cache drive

750W power supply

 

The only dockers I'm running are plex and plexpy. Their data is stored on the SSD.

 

The last time I ran a parity check was maybe a little over a month ago with no errors.

 

I assume a lot of the info you might need would be in the logs I posted? If I'm missing anything let me know, thanks!

[trurl]

What is the exact model of your power supply?

[Treytor]

CORSAIR TX Series CMPSU-750TX

[Treytor]

Just happened again. Exact same situation as before. Can still telnet in, shfs is using 100% cpu. Not able to collect logs.

 

The only difference this time I was able to unmount disk 6. Disk 3 was still busy.

[roland]

Hi,

 

I am having the same problem every few days.  There are a few more posts around the forum.  Look for shfs 100%.

I tried most of the recommendation in these threads, except changing the file system. (new disk is on its way) But so far no luck.

 

Any extra ideas would be appreciated.

[JonathanM]

I had the same symptoms until I converted over to XFS.

[Treytor]

How would I go about converting to XFS when my array is full (8.5 TB free in a 60TB array) and have no more open disk slots available?

[JonathanM]

How would I go about converting to XFS when my array is full (8.5 TB free in a 60TB array) and have no more open disk slots available?

Assuming your largest data drive is less than 8TB, you should be able to juggle data to completely empty your largest drive, change its format to XFS, copy the contents of your next largest drive to the newly formatted drive, change format to XFS, lather rinse repeat. PITA, but doable.

 

The unbalance plugin plus the thread dedicated to XFS conversion should help.

[Treytor]

That does sound like a pain in the ass, but doable. Interestingly my largest drive is disk3 (the one that I'm never able to unmount when this happens) so I'll give it a shot.

 

Thanks!