In the last week I have been having some issues with my server. Out of nowhere the system will just hang. All dockers become unresponsive, cannot reach the server though any WebGUI or via Putty. Single power button push appears to not do anything and a forced shutdown is the only option. 

Now I have done it back to back days. Yesterday when it rebooted I had a file system error, that I was able to recover from via google searches and repair. Today I woke up to the same frozen server. I have rebooted and no obvious drive issues listed on main page, but in the logs I am getting errors. It looks like (sde) drive could have something going on with it. But I am not sure how to read the logs to know exactly the cause, or what to start ruling out. 

Its running a parity check now, VERY slowly (4.3 mb/sec) which will never finish. But I am hesitant to cancel because I did yesterday and this is obviously something failing/failed that needs to be addressed.

Before I start pulling things apart, wanted to see if someone more versed in reading the logs could point me in a direction.

Appreciate any help.

matrix-diagnostics-20231015-0224.zip

Edited October 15, 20232 yr by DipNFalls

There are what look like power/connections issues with disk5, replace cables and post new diags after starting a new parity check.

1 hour ago, JorgeB said:

There are what look like power/connections issues with disk5, replace cables and post new diags after starting a new parity check.

Thank you. I suspected power or some type of connection issue. I use 2 sets of drive cages, and every little thing that has been happening the past couple weeks has been associated with drives in the top cage. I took out all the drives, disconnected all the cabling and then hooked everything up outside the cage. And alot less red in my boot up.

If its power or cable issues, I can start playing around and narrowing it down what and where. Just wanted to rule out other things first.

matrix-diagnostics-20231015-0909.zip

Looks good so fat but start a parity check and see if no more errors come up.

Looking good. Had to rebuild the one drive that had issues and ran into nothing during the 36 hours it took. Rebooted and logs look clear to me. 

Which means sometime this weekend the tower will burst into glorious flames, because...........why not?

Anyway I appreciate the experienced eyes and knowledge in pointing me to the issue. I do think one of my power cables is bad. One of the hard drives I had outside the cages wasn't powering on when using it after putting everything back together. Redid all the cabling and everything working now.

matrix-diagnostics-20231016-2307.zip