Hi All, this is starting to drive me mad, so i'd really appreciate any help you can provide.

 

I'm going to go into detail of issues here, so the short version is:

TL;DR: was stable for over a year, never had a single issue. upgraded parity drive and 6.1.9->6.3.3 on same day, now keeps becoming unresponsive in a variety of ways, every 12-48 hours.

 

24th - 26th April

So, my array had 1p+3d 6TB WD Reds in, and I had a brand new 8TB Red drive that was in the server, precleared and ready to go when i got round to it. So on Monday 24th Apr, I did decided to both upgrade my server from 6.1.9 to 6.3.3, and swap out the parity drive. I did the upgrade first, there was a bit of an issue with docker, had to delete and recreate the image, no biggie, but then it all seemed to work ok. Then i swapped out the drive, and the 8TB parity rebuild started. I also decided to add a container, Krusader.

 

I think the parity rebuild was complete at this point, but on the evening of the Weds 26th Apr, my wife said she couldn't access plex while I was away. I tried to VPN on to my openVPN container, it wouldn't connect. i vpn'd on to my router, and tried to browse to the webUI, nothing! i can't remember which, but a couple of containers still seemed to respond, but ultimately, the server was down. When I got home, I connected a monitor, and saw this mess:

13sec video of monitor

SlowMo - Poor quality though

I couldn't ssh, I tried to hit the power button, nothing happened. Had to force it off. once back on, unproper shutdown detected, parity check...etc

 

27th/28th April

Plex not responding, i try and restart the container, the web gui stops responding, theres nothing I can do, but most other dockers seem fine. i use ssh to powerdown, it says something about a system halt, and then thats it. i cant even ssh anymore, i think only one docker app is now working. i put a keyboard direct, do powerdown again, same thing happens on there, system halt but everythings frozen, hdd's still going full pelt in the parity check, i have to force it off again.

 

29/30th April

I'm abroad, and notice on the 29th that a random core of my 8 cores shows 100% usage. every 30 seconds or so, it switches to a different core, but at any one time, its 100% active on one of them. cadvisor tells me that the process using 92-100% CPU is kswapd. I think maybe it's down to parity check.

On the 30th, parity check is over (parity now valid with 1 error!?!?!?) but the cpu usage is still there. I dont know why its happening, the only change i've made that would affect CPU time is the Krusador container, which i dont really need. I try and stop krusader from the dashboard (im abroad and vpn'd to a container btw). instead of the page reloading with it stopped in say 1second, it just loads forever. the whole webui is now not responding, but containers still are. about 5 mins later, it comes back with "Code execution exception" or something like that, with a big red X. I try again a few hours later, no joy. I decide to do it when im back home.

 

1st May

I'm back home, i cannot stop that krusader container, webgui just hangs for 5 mins every tiem and fails. i see something in the logs from a few hours ago about "TCP: request_sock_TCP: Possible SYN flooding on port 8181. Sending cookies. Check SNMP". That's PlexPy's port. I try to stop that, same thing, will not stop. I try to update plexpi, because randomly pretty much every container now has an update. I get some errored out plexpy container in my 'dockers' view and an orphaned plexpy container. cant start it, logo has gone. weird.

I decide, that if i can't stop krusader, which may or may not be the culprit of this 100% kswapd issue (i know its a kernal thing, but krusador was my only software change! (other than unraid upgrade, but can't undo that!)), then i need to restart.

There i am, with my finally valid parity. i hit stop array, and the bottom right breadcrum trail just hangs forever on "stopping services". nothing works, webgui is not responding, some containers did stop, others haven't yet. I ssh on successfully, copy the syslog to a user share and have a look. it had as far as "spinning up disks" i think. try to "powerdown", doesnt work. tap the power button, doesnt work. I have to force it off AGAIN.

Boot it up, start the array, parity check starts. i delete krusader before starting any dockers. i delete both the F-ed up plexpy and the orphaned plexpy image, and re add it. start all containers, everything is looking smooth, no kswapd issues, parity rebuild is running very smoothly and quickly, cpu /ram usage looks good. finally, everything is back to normal.

 

Today (2nd may)

last time i checked on the server, parity was on 73%, all good. i'd watched a couple things on plex earlier, everything is running smoothly!

about 30-60 mins later, i fire up my tv's plex app, it seems to take a while to connect (all on ethernet), but eventually shows my libraries, on the left, but no thumbnails. i expect a buggy samsung tizen app, so i exit it, and try again. this time, i get some thumbnails but not all. oh well, i've got the one i want. click it, hit play, and it just loads constantly (direct play, not using cpu, usually an instant load). I exit again, and reopen the app, this time it wont connect at all. i try on my phone, nothing. plexpy sends me a notification, "Plex Server down!". i try every other docker and the web ui, nothing. plexpy is the only docker working, and it's complaining that it cant connect to plex.

I ssh, it opens with an empty terminal.

I exit, reconnect with ssh, i get a login. i issue a syslog copy command to a share, it accepted the command, but never returned anything. it's just a cursor on an empty new line. i check the share, i have a zero-byte syslog file there.

try to ssh again, nothing.

try again, it works, i try and view log, nothing.

try to ssh again, nothing.

keyboard to server, login to root, i see the "last login at..." but it doesnt give me a prompt to type anything, it's hung. 

ssh again, manage to get in, and issue a powerdown, no response.

i hit power button, nothing.

all shares still working, but that is it. hdd still going mad doing it's parity check. 

I've had to force it off again!

 

 

now it's rebuilding, i'm at 10%.

everything seems to be running fine atm, but i doubt it will last long.

i installed the common problems plugin, it detected a sonar port path mapping that deviated from the template, no biggie, sonar is fine.

it also told me i have an ssd with no trim plugin ( i assumed TRIM support was built in to unraid), i've installed it an scheduled it for a couple hours after mover's schedule.

I'm running the troubleshooting mode now, awaiting the inevitable!

 

Please let me know anything that i should do to help diagnose the issue, or any dumps i can provide. Please, this is really stressing me out.

 

specs wise: Xeon E3-1230v2 / ASRock Z77Pro3 / hyperx 2x8GB DDR3 1600 / samsung 830 256gb cache and WDRed8TB on mobo's sata3 / 4xWDRed6TB's on mobo's SATA2 / GT210 for head, aerocool ap-pro 450w PSU (pulls 60-65 idle / 100-120 load from the wall)

 

Also, i doubt it's going to be this, but the only known 'issues' i have:

• cache drive is btrfs (it was default when i installed for some reason, i found out it can have issues late. main desire to upgrade to 6.3.3 was for 'cache prefered' so that i can unload data, reformat to xfs, reload data at some point, but i'll do that when things are stable.
• parity drive is running a little warm: the 8tb red is in a makeshift stand in a 5.25 bay, with poor cooling. it runs at 42'C normally, idle or with normal load, but parity checks bring this to 44-46'C. it has never gone above 46'C. other HDD's sit on 30-32'C. SSD is 28 idle, 33 during mover, rockets to 44 on heavy write load. I'm currently looking for a new case to accommodate all HDD's with better cooling.

 

Thank you for your time, i know this was a long read

 

 

Edited May 2, 20179 yr by newerNan
2 words

Try to get a diagnostics file.  From the GUI:   'Tools'   >>>   'Diagnostics'     that will download the file to your download directory/folder that you browser uses on your PC.   From the Command Line, type    diagnostics    that will save the file to the  logs  folder on your Flash Drive.

 

You can also start the server in the Safe Mode during the boot process.  If it boots successfully, I would allow the parity rebuild to finish at that point.  Get a diagnostics file after that is done and upload it in a new post.  Change your startup settings so that everything doesn't start at bootup.  That way you can turn things back one thing at a time when you do a normal boot. 

 

By the way, a good practice is to change one thing at a time and make sure that everything is working properly for a couple of days before moving on the next item. 

 
 

Hi @Frank1940, Thanks for your reply.

 

I've attached a diagnostics dump from right now.

As a parity check is already 15% through, i was going to leave it to complete (if it gets that far) for now, rather than stopping it early and rebooting to safe mode. Is it still worth uploading a diagnostics dump after this completes, from it's current normal boot?

 

At the moment, nothing starts automatically from boot, I've always manually started array and each docker container.

 

I agree, I should have given it a week after upgrade, before parity drive change, to ensure it was running ok. At the time, one of my drives had hit 85%, and i was getting anxious. I thought my 10 minute smoke test would have sufficed. Lesson learnt i guess.

 

 

harmzserv-diagnostics-20170502-2227.zip

First thing to do is to read the first two posts in the release thread for 6.2.0:

 

       

Fix anything that applies to you. 

 

Next take a look at the release thread for 6.3.0.   While there were not as many big changes, the fix for XSS and CSRF can result in a lot of error messages in the syslog file if everything is has not been updated meet the requirement for a proper security token. 

One thing to do is to check to see that all of your Dockers, plugins and VM's are up to date.  Making a big jump in version numbers usually means that there may well be some incompatibility issues from some of these.