I suspect this is a hardware issue as I recently replaced the PSU and never had any problems previously (this is the "little black" server w/hardware listed in my sig below).  A couple nights ago I ran the first parity check since installing the new PSU and it was going fine until I woke up the next morning and could no longer access the web UI or flash drive folder via the network and no longer saw the server as an attached device on my router config page either.  The server itself was still powered on but with no way to access it I had no choice but to hard power down.  Upon re-starting the array I was prompted to run another parity check and did.  When it was still running fine at >60% completed before I went to bed I figured the previous night might have been caused to some unfamiliar commands I was trying through telnet mucking something up and probably there wasn't any real issue. Then I wake up this morning and once again the server had dropped completely off the network and become unresponsive (attaching VGA to check at the console shows a blank screen that won't respond to any keyboard presses).

 

I've attached a syslog but since it's from after reboot this morning I'm not sure it's going to show anything about what cause the server to hang before the power down.  Would appreciate any advice on troubleshooting steps from here.

 

BTW, parity and data drives are all connected to the motherboard's SATA ports and since the new PSU only had 4 SATA power connectors I had to connect the other 2 drives through MOLEX to SATA power adapters.  If it's a PSU related issue could it even be one of those cables being bad instead of the PSU itself?  There hasn't been any problems during normal operation.  Writing to the server and watching movies off it haven't been any issue since I swapped the PSU a couple weeks ago, only these parity checks.  Thanks.

syslog.txt

I suspect this is a hardware issue as I recently replaced the PSU and never had any problems previously (this is the "little black" server w/hardware listed in my sig below).  A couple nights ago I ran the first parity check since installing the new PSU and it was going fine until I woke up the next morning and could no longer access the web UI or flash drive folder via the network and no longer saw the server as an attached device on my router config page either.  The server itself was still powered on but with no way to access it I had no choice but to hard power down.  Upon re-starting the array I was prompted to run another parity check and did.  When it was still running fine at >60% completed before I went to bed I figured the previous night might have been caused to some unfamiliar commands I was trying through telnet mucking something up and probably there wasn't any real issue. Then I wake up this morning and once again the server had dropped completely off the network and become unresponsive (attaching VGA to check at the console shows a blank screen that won't respond to any keyboard presses).

 

I've attached a syslog but since it's from after reboot this morning I'm not sure it's going to show anything about what cause the server to hang before the power down.  Would appreciate any advice on troubleshooting steps from here.

 

BTW, parity and data drives are all connected to the motherboard's SATA ports and since the new PSU only had 4 SATA power connectors I had to connect the other 2 drives through MOLEX to SATA power adapters.  If it's a PSU related issue could it even be one of those cables being bad instead of the PSU itself?  There hasn't been any problems during normal operation.  Writing to the server and watching movies off it haven't been any issue since I swapped the PSU a couple weeks ago, only these parity checks.  Thanks.

Your syslog is clean (as you suspected).  It could be the cabling.

 

What you really want to do is at the console, enter in the following:

 

tail -f /var/log/syslog > /boot/syslog.txt

 

the console won't show you anything, and will appear to hang.  What its doing is copying every line from the log as it appears to the flash drive.

 

Then, run a parity check and if the system fails upload the syslog.

I notice that the power supply is not a single-rail variant.  I wonder if insufficient power is the problem - a parity check is when the power supply would be most heavily loaded.  A bad connection on a power splitter might also caused issues.

Your syslog is clean (as you suspected).  It could be the cabling.

 

What you really want to do is at the console, enter in the following:

 

tail -f /var/log/syslog > /boot/syslog.txt

 

the console won't show you anything, and will appear to hang.  What its doing is copying every line from the log as it appears to the flash drive.

 

Then, run a parity check and if the system fails upload the syslog.

 

Thanks Squid.  Really appreciate your help on this.  Just ran the command and started a new parity check.  Just so I understand though, if it does the same thing then do I pull the flash after the system hang but before hard powering down and the log info should be current up to that point?  If I hard power down first (or if it crashes this time instead of hanging) then the log would be wiped, correct?

Thanks Squid.  Really appreciate your help on this.  Just ran the command and started a new parity check.  Just so I understand though, if it does the same thing then do I pull the flash after the system hang but before hard powering down and the log info should be current up to that point?  If I hard power down first (or if it crashes this time instead of hanging) then the log would be wiped, correct?

No - the file is being written as you go so should be there even if you do a hard powerdown.  It is possible that a hard reset may require Windows to 'fix the drive as a file was open on it at the time of the crash but it should do that OK.

I notice that the power supply is not a single-rail variant.  I wonder if insufficient power is the problem - a parity check is when the power supply would be most heavily loaded.  A bad connection on a power splitter might also caused issues.

 

Hmm.  I don't understand PSU single vs. multi-rail but if insufficient power is the problem then I think it would have to be a defective unit.  Prior to changing the PSU, kill-a-watt was reading only a 44w draw from the wall (the C-60 processor is a 9w TDP and aside from the case fans the hard drives are the only things the unit has to power).  So even at the full load of a parity check, if a modern ATX power supply can't drive that then it can't drive anything.  Heck a Pico should probably be able to power it.

 

If it's a bad cable then hopefully the syslog will give me some indication of which drive is causing the problem so I can isolate that.  I do wonder if that's it though, only because the hanging is only happening way deep into the parity check.  In the previous posts I'd found of people having the same problem (though not necessarily PSU related -- in one case it was a bad SATA controller) the system would hang very early in the parity check at like 3-5% completed.  I'd think if the PSU itself was putting out insufficent power it would be the same thing.  In a parity check does it read though the parity data on each drive 1 at a time, or all at once?  If it's one at a time and the problem is a connection to a specific drive that's one of the last to be checked then that would make more sense, wouldn't it?  Just spitballing since I don't really know how parity checks work.

No - the file is being written as you go so should be there even if you do a hard powerdown.  It is possible that a hard reset may require Windows to 'fix the drive as a file was open on it at the time of the crash but it should do that OK.

 

Got it.  Thanks for clarifying!

Your syslog is clean (as you suspected).  It could be the cabling.

 

What you really want to do is at the console, enter in the following:

 

tail -f /var/log/syslog > /boot/syslog.txt

 

the console won't show you anything, and will appear to hang.  What its doing is copying every line from the log as it appears to the flash drive.

 

Then, run a parity check and if the system fails upload the syslog.

 

Okay, this time the parity check completed.  No errors found and nothing that jumps out at me from the syslog either -- although since I really have no idea how to read these things I've attached it here in case there are any red flags that jump out to anyone else.  If not then I guess I'll just keep my fingers crossed it doesn't happen again.

syslog.zip

Your syslog is clean (as you suspected).  It could be the cabling.

 

What you really want to do is at the console, enter in the following:

 

tail -f /var/log/syslog > /boot/syslog.txt

 

the console won't show you anything, and will appear to hang.  What its doing is copying every line from the log as it appears to the flash drive.

 

Then, run a parity check and if the system fails upload the syslog.

 

Okay, this time the parity check completed.  No errors found and nothing that jumps out at me from the syslog either -- although since I really have no idea how to read these things I've attached it here in case there are any red flags that jump out to anyone else.  If not then I guess I'll just keep my fingers crossed it doesn't happen again.

You do have a ton of rcu_shed issues in your syslog.

 

This issue *should* be fixed in the next beta http://lime-technology.com/forum/index.php?topic=38896.0 and is due to be released sometime in the coming week as per jonP

You do have a ton of rcu_shed issues in your syslog.

 

This issue *should* be fixed in the next beta http://lime-technology.com/forum/index.php?topic=38896.0 and is due to be released sometime in the coming week as per jonP

 

Thanks Squid.  I always get a lot of rcu_shed flags in my parity check syslogs.  I posted a log with a bunch of those back when I first ran unRAID on this server in 2013 but I found another thread back then that said they were just harmless warnings which could be disregarded so I've never bothered about it since.  It would be nice if it was fixed in the next release but my bigger concern is not knowing what caused these system hangups and if there's still possibly a hardware issue even though it made it through the parity check this time.

 

I'll make a point though to run that tail command every time I do a parity check so just in case the problem re-surfaces I'll have it captured in a log.  Thanks again for your help!