February 11, 201610 yr I had been running a version of v5 for ages without issues, and upgraded straight to v6.1.6 a couple weeks ago following the instructions to reformat the USB key. Since then the system has been 'freezing up' most nights: no GUI access, no share access, but it pings and I can SSH in and grab stuff via WinSCP (though some commands don't work). I upgraded to 6.1.8 yesterday, but it still froze up again this morning. I tried to capture diagnostics but the command did not fully complete; I've attached what it did generate. When it's frozen up like this, I can't run the powerdown command (just doesn't actually powerdown anything). Specs: Gigabyte GA-MA74GM-S2H 2x1GB 800Mhz DDR2 AMD Athlon 64 X2 Dual Core 6000+ 3.00GHz 4 data drives, 1 cache, 1 parity, all ReiserFS SNMP plugin and the Powerdown plugin installed. So far I have tried: Disabling the cache drive on all disks Disabling Docker Checked SMART status of all disks (the quick test) and all look OK. One disk has some current pending sectors but this has not increased in a long time. Last night I wrote a script that SSH's in and grabs the top 5 processes by CPU usage every 2 minutes to see if that would turn up anything useful (date; ps aux | sort -rk 3,3 | head -n 5). It shows this process grabbing a bit more CPU % over the course of the night: /usr/local/sbin/shfs /mnt/user -disks 31 2048000000 -o noatime,big_writes,allow_other -o remember=0 Gonna be honest, don't know what that actually is indicating but it doesn't seem right. Attached the output as processes.txt. Last night I believe the shares stopped working between about 10pm and 2:30am, but there's nothing in the syslog that shows anything funky happening. I've attached some logs. The server is currently in it's half responsive state, so if anyone has any troubleshooting suggestions that I can run over SSH, or logs and files I can grab, I'll leave it like this for a few hours. Thanks in advance! towerzip.zip
February 11, 201610 yr Author It's still responding to pings but I've now lost SSH access Off to press the power button...
February 11, 201610 yr Community Expert The correct answer for pending sectors in zero. You must have been thinking of reallocated sectors, which is OK if small and not increasing, but pending sectors need to be dealt with.
February 11, 201610 yr Author The correct answer for pending sectors in zero. You must have been thinking of reallocated sectors, which is OK if small and not increasing, but pending sectors need to be dealt with. Thanks trurl. Would pending sectors cause the symptoms I'm seeing? I'd rather have a stable array before I start swapping disks out. I don't think the array would stay useable long enough to rebuild the disk anyways.
February 11, 201610 yr Community Expert 2GB RAM is probably going to be a little tight to do much of the new v6 stuff. Have you done a memtest recently? My guess is filesystem corruption so you might try checking filesystems. Lots of this going on in your syslog in pretty rapid succession. Is that all you or someone hacking? Feb 10 17:29:55 Tower emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Feb 10 17:35:51 Tower sshd[16159]: Accepted password for root from 192.168.2.23 port 58159 ssh2 Feb 10 17:36:08 Tower sshd[16159]: Received disconnect from 192.168.2.23: 11: disconnected by user Feb 10 17:36:08 Tower sshd[16159]: Disconnected from 192.168.2.23 Feb 10 17:42:52 Tower sshd[17220]: Accepted password for root from 192.168.2.28 port 43326 ssh2 Feb 10 17:44:34 Tower sshd[17517]: Connection closed by 192.168.2.90 [preauth] Feb 10 17:44:34 Tower sshd[17531]: Connection closed by 192.168.2.90 [preauth] Feb 10 17:44:42 Tower sshd[17544]: Accepted password for root from 192.168.2.90 port 59052 ssh2 Feb 10 17:44:42 Tower sshd[17544]: Received disconnect from 192.168.2.90: 11: disconnected by user Feb 10 17:44:42 Tower sshd[17544]: Disconnected from 192.168.2.90 Feb 10 17:44:59 Tower sshd[17606]: Accepted publickey for root from 192.168.2.90 port 59053 ssh2: RSA SHA256:nFGy5dzfcnBw7S55LhvRt8tZrQLdQAvxFAmye3qB20U Feb 10 17:57:47 Tower sshd[17606]: Received disconnect from 192.168.2.90: 11: disconnected by user Feb 10 17:57:47 Tower sshd[17606]: Disconnected from 192.168.2.90 Feb 10 18:00:01 Tower sshd[19824]: Accepted publickey for root from 192.168.2.90 port 59069 ssh2: RSA SHA256:nFGy5dzfcnBw7S55LhvRt8tZrQLdQAvxFAmye3qB20U Feb 10 18:00:01 Tower sshd[19824]: Received disconnect from 192.168.2.90: 11: disconnected by user Feb 10 18:00:01 Tower sshd[19824]: Disconnected from 192.168.2.90 Feb 10 18:00:18 Tower sshd[19891]: Accepted publickey for root from 192.168.2.90 port 59071 ssh2: RSA SHA256:nFGy5dzfcnBw7S55LhvRt8tZrQLdQAvxFAmye3qB20U Feb 10 18:00:18 Tower sshd[19891]: Received disconnect from 192.168.2.90: 11: disconnected by user Feb 10 18:00:18 Tower sshd[19891]: Disconnected from 192.168.2.90 Feb 10 18:00:37 Tower sshd[17220]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:00:37 Tower sshd[17220]: Disconnected from 192.168.2.28 Feb 10 18:01:33 Tower sshd[20065]: Accepted password for root from 192.168.2.28 port 43396 ssh2 Feb 10 18:01:33 Tower sshd[20065]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:01:33 Tower sshd[20065]: Disconnected from 192.168.2.28 Feb 10 18:01:43 Tower sshd[20096]: Accepted password for root from 192.168.2.28 port 43399 ssh2 Feb 10 18:01:43 Tower sshd[20096]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:01:43 Tower sshd[20096]: Disconnected from 192.168.2.28 Feb 10 18:02:28 Tower sshd[20217]: Accepted password for root from 192.168.2.28 port 43400 ssh2 Feb 10 18:02:32 Tower sshd[20217]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:02:32 Tower sshd[20217]: Disconnected from 192.168.2.28 Feb 10 18:03:15 Tower sshd[20347]: Accepted password for root from 192.168.2.28 port 43404 ssh2 Feb 10 18:03:15 Tower sshd[20347]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:03:15 Tower sshd[20347]: Disconnected from 192.168.2.28 Feb 10 18:07:53 Tower sshd[21023]: Accepted password for root from 192.168.2.28 port 43418 ssh2 Feb 10 18:08:01 Tower sshd[21023]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:08:01 Tower sshd[21023]: Disconnected from 192.168.2.28 Feb 10 18:09:14 Tower sshd[21251]: Accepted password for root from 192.168.2.28 port 43421 ssh2 Feb 10 18:09:14 Tower sshd[21251]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:09:14 Tower sshd[21251]: Disconnected from 192.168.2.28 Feb 10 18:11:52 Tower sshd[21651]: Accepted password for root from 192.168.2.28 port 43431 ssh2 Feb 10 18:11:52 Tower sshd[21651]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:11:52 Tower sshd[21651]: Disconnected from 192.168.2.28 Feb 10 18:13:25 Tower sshd[21890]: Accepted password for root from 192.168.2.28 port 43437 ssh2 Feb 10 18:13:25 Tower sshd[21890]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:13:25 Tower sshd[21890]: Disconnected from 192.168.2.28 Feb 10 18:24:05 Tower sshd[23460]: Accepted password for root from 192.168.2.28 port 43469 ssh2 Feb 10 18:24:05 Tower sshd[23460]: Received disconnect from 192.168.2.28: 11: disconnected by user
February 11, 201610 yr Author 2GB RAM is probably going to be a little tight to do much of the new v6 stuff. Have you done a memtest recently? My guess is filesystem corruption so you might try checking filesystems. Lots of this going on in your syslog in pretty rapid succession. Is that all you or someone hacking? Feb 10 17:29:55 Tower emhttp: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Feb 10 17:35:51 Tower sshd[16159]: Accepted password for root from 192.168.2.23 port 58159 ssh2 Feb 10 17:36:08 Tower sshd[16159]: Received disconnect from 192.168.2.23: 11: disconnected by user Feb 10 17:36:08 Tower sshd[16159]: Disconnected from 192.168.2.23 Feb 10 17:42:52 Tower sshd[17220]: Accepted password for root from 192.168.2.28 port 43326 ssh2 Feb 10 17:44:34 Tower sshd[17517]: Connection closed by 192.168.2.90 [preauth] Feb 10 17:44:34 Tower sshd[17531]: Connection closed by 192.168.2.90 [preauth] Feb 10 17:44:42 Tower sshd[17544]: Accepted password for root from 192.168.2.90 port 59052 ssh2 Feb 10 17:44:42 Tower sshd[17544]: Received disconnect from 192.168.2.90: 11: disconnected by user Feb 10 17:44:42 Tower sshd[17544]: Disconnected from 192.168.2.90 Feb 10 17:44:59 Tower sshd[17606]: Accepted publickey for root from 192.168.2.90 port 59053 ssh2: RSA SHA256:nFGy5dzfcnBw7S55LhvRt8tZrQLdQAvxFAmye3qB20U Feb 10 17:57:47 Tower sshd[17606]: Received disconnect from 192.168.2.90: 11: disconnected by user Feb 10 17:57:47 Tower sshd[17606]: Disconnected from 192.168.2.90 Feb 10 18:00:01 Tower sshd[19824]: Accepted publickey for root from 192.168.2.90 port 59069 ssh2: RSA SHA256:nFGy5dzfcnBw7S55LhvRt8tZrQLdQAvxFAmye3qB20U Feb 10 18:00:01 Tower sshd[19824]: Received disconnect from 192.168.2.90: 11: disconnected by user Feb 10 18:00:01 Tower sshd[19824]: Disconnected from 192.168.2.90 Feb 10 18:00:18 Tower sshd[19891]: Accepted publickey for root from 192.168.2.90 port 59071 ssh2: RSA SHA256:nFGy5dzfcnBw7S55LhvRt8tZrQLdQAvxFAmye3qB20U Feb 10 18:00:18 Tower sshd[19891]: Received disconnect from 192.168.2.90: 11: disconnected by user Feb 10 18:00:18 Tower sshd[19891]: Disconnected from 192.168.2.90 Feb 10 18:00:37 Tower sshd[17220]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:00:37 Tower sshd[17220]: Disconnected from 192.168.2.28 Feb 10 18:01:33 Tower sshd[20065]: Accepted password for root from 192.168.2.28 port 43396 ssh2 Feb 10 18:01:33 Tower sshd[20065]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:01:33 Tower sshd[20065]: Disconnected from 192.168.2.28 Feb 10 18:01:43 Tower sshd[20096]: Accepted password for root from 192.168.2.28 port 43399 ssh2 Feb 10 18:01:43 Tower sshd[20096]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:01:43 Tower sshd[20096]: Disconnected from 192.168.2.28 Feb 10 18:02:28 Tower sshd[20217]: Accepted password for root from 192.168.2.28 port 43400 ssh2 Feb 10 18:02:32 Tower sshd[20217]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:02:32 Tower sshd[20217]: Disconnected from 192.168.2.28 Feb 10 18:03:15 Tower sshd[20347]: Accepted password for root from 192.168.2.28 port 43404 ssh2 Feb 10 18:03:15 Tower sshd[20347]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:03:15 Tower sshd[20347]: Disconnected from 192.168.2.28 Feb 10 18:07:53 Tower sshd[21023]: Accepted password for root from 192.168.2.28 port 43418 ssh2 Feb 10 18:08:01 Tower sshd[21023]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:08:01 Tower sshd[21023]: Disconnected from 192.168.2.28 Feb 10 18:09:14 Tower sshd[21251]: Accepted password for root from 192.168.2.28 port 43421 ssh2 Feb 10 18:09:14 Tower sshd[21251]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:09:14 Tower sshd[21251]: Disconnected from 192.168.2.28 Feb 10 18:11:52 Tower sshd[21651]: Accepted password for root from 192.168.2.28 port 43431 ssh2 Feb 10 18:11:52 Tower sshd[21651]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:11:52 Tower sshd[21651]: Disconnected from 192.168.2.28 Feb 10 18:13:25 Tower sshd[21890]: Accepted password for root from 192.168.2.28 port 43437 ssh2 Feb 10 18:13:25 Tower sshd[21890]: Received disconnect from 192.168.2.28: 11: disconnected by user Feb 10 18:13:25 Tower sshd[21890]: Disconnected from 192.168.2.28 Feb 10 18:24:05 Tower sshd[23460]: Accepted password for root from 192.168.2.28 port 43469 ssh2 Feb 10 18:24:05 Tower sshd[23460]: Received disconnect from 192.168.2.28: 11: disconnected by user That was me - had a script running to see what process was hogging the CPU, the output of which is attached in the original post. I'm not doing any of the new V6 stuff and don't need to, just thought it was time to upgrade. I hadn't had any issues in V5, it was stable for a couple years. Beginning to regret the upgrade decision now. I actually tried to check the filesystems. Got through one disk OK, but while checking the second disk the bit in the gui where it tells you the status of the check just disappeared. Tried to restart the box with the gui but the array wouldn't stop, so I tried to restart with the powerdown command but that didn't work, and I had to hard reset it. Sigh. I might give it another go tomorrow.
February 11, 201610 yr Author Forgot to mention, I've not done a memtest recently. I'll add that to the list! Thanks trurl.
February 12, 201610 yr Author I have checked the file system on all the data disks and they all came back clear, and I ran 2 passes of memtest with no errors. Anyone have any other suggestions? Any guesses as to what would cause the shfs process to use more and more CPU until the system becomes gradually unusable?
February 12, 201610 yr Community Expert The major thing missing from your zip are any SMART attributes. If you can't get them from the GUI or running diagnostics from command line maybe you can get them the old fashioned way. See v5 help in my sig.
February 12, 201610 yr Author The server is kindly behaving itself today so I've grabbed the full diagnostics from the gui, attached. tower-diagnostics-20160212-1437.zip
February 12, 201610 yr Community Expert Actually 2 drives with pending sectors Serial Number: WD-WMAVU1296241 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 16 Serial Number: WD-WCC070339040 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 29 Probably not really the cause of your main problems though. The only thing I can think of is add memory. One of my servers only has 4GB RAM but I only use it as a NAS to backup some shares from my main server. And it is only running an Atom 330. The GUI can be a little more sluggish than on my main server but it is working fine with v6 for its limited duties. Since all your drives are still ReiserFS you could also try going back to v5 just to see if your problem persists and to clear your pending sectors.
November 23, 20169 yr Author Because I hate it when I google a problem and only get hits without actual fixes, I thought I'd update this to say that reformatting all disks to XFS fixed the problem. I tried upgraded the memory and removed all disks with pending sectors and it didn't make a difference. Running on XFS now for a few months without a hiccup.
Archived
This topic is now archived and is closed to further replies.