This must be the week for my server to start acting up because I have had all sorts of weird things start happening.  I will attach a syslog and the u_notify email I get to the post so that it can be looked through for more information.

 

The story as it happened which should be able to be seen in the syslog:

I believe disk5 went down first and that is a Seagate 1TB drive.  It went down so I did the whole shutdown, unplug, startup, shutdown, plug in, let it rebuild onto the disk.  This was working fine except that the rebuild failed and I ended up with a report of 288 errors on the drive.  I could not seem to get to the drive but I did manage to run a smart test on it and it passed... just fine.  There were a few reallocated sectors but nothing had changed in a while so that looked fine.  Then disk6, a Seagate 1.5TB drive, reported 6 errors sometime last night and when I got up this morning it along with disk4, a Samsung 750GB drive, was not responding either.  I am not quite sure why these drives decided to be a pain in the arse now as they have been working fine. I got a message about disk4 and disk6 being unassigned so I stopped the array and went to the management page and attempted to assign them to the proper place.  The Samsung came back online no problem (there were no errors with this disk) but the Seagate 1.5TB drive would not come back, which I kind of expected because of the errors.

 

disk5 and disk6 do not currently have any data on them only disk4 has data on it.

 

I am leaning towards a cable problem at this point but I need to grab some extra cables when I am out today.  Any guidance in the mean time is appreciated and if you need any more info please let me know.

 

pastebin of syslog

Sorry to hear about all this trouble.  Starting with the good news, I did not see any problems related to the physical drives themselves, just their connection to the system.  In fact, I'm rather sure that a reboot and the Trust My Array procedure would have fully restored the array and all drives, BUT it would have immediately repeated the same set of issues.  There were FAR too many errors for me to check them all, but I looked at a lot and they seemed to be consistently communications related, with the SATA links going up and down, and lots of communications errors.  Disk 5 (sdi) was definitely the worst, was the first to be lost, and then was recovered and lost, over and over again, reappearing as sdj a number of times, and sdk at least once.  Disk 6 (sdg) was the next to go, and as far as I saw, never reappeared.  Then Disk 4 (sdh) decided to join the rebellion, and was recovered as sdj at least once.

 

Disk 5 was probably fine, but now that you started a rebuild of it, which was quickly aborted when the drive was dropped, I'm not as sure it is still fine.  Once the connection issues are corrected, I would run reiserfsck on it, and if it indicates errors, consider fully rebuilding it again, if you feel you can fully trust your parity and the system.

 

The evidence:

* The problems seem to be similar with all 3 drives

* There are no apparent physical drive errors - probably rules out the drives

* There are NO ICRC or BadCRC flags raised, which are typical of the ordinary bad cable issue - does not completely rule out the cables, but they are probably OK (but could be loose)

* The errors appear like the drives are briefly disconnecting and reconnecting

 

Some speculation as to what might be wrong:

* If these are connected to a backplane, then that becomes the obvious suspect.  Check for drives not firmly seated, perhaps vibrating loose.  Perhaps faulty backplane

* Check for loose cables, all connectors, for both SATA and power cables including any splitters

* It could possibly be a PSU problem, either it may be failing, or too many drives on one rail (this is not an expert opinion)

* I would especially suspect anything that is common to just these 3 drives, while keeping in mind that Disk 5 is the worst

Sorry to hear about all this trouble.  Starting with the good news, I did not see any problems related to the physical drives themselves, just their connection to the system.  In fact, I'm rather sure that a reboot and the Trust My Array procedure would have fully restored the array and all drives, BUT it would have immediately repeated the same set of issues.  There were FAR too many errors for me to check them all, but I looked at a lot and they seemed to be consistently communications related, with the SATA links going up and down, and lots of communications errors.  Disk 5 (sdi) was definitely the worst, was the first to be lost, and then was recovered and lost, over and over again, reappearing as sdj a number of times, and sdk at least once.  Disk 6 (sdg) was the next to go, and as far as I saw, never reappeared.  Then Disk 4 (sdh) decided to join the rebellion, and was recovered as sdj at least once.

 

Disk 5 was probably fine, but now that you started a rebuild of it, which was quickly aborted when the drive was dropped, I'm not as sure it is still fine.  Once the connection issues are corrected, I would run reiserfsck on it, and if it indicates errors, consider fully rebuilding it again, if you feel you can fully trust your parity and the system.

 

The evidence:

* The problems seem to be similar with all 3 drives

* There are no apparent physical drive errors - probably rules out the drives

* There are NO ICRC or BadCRC flags raised, which are typical of the ordinary bad cable issue - does not completely rule out the cables, but they are probably OK (but could be loose)

* The errors appear like the drives are briefly disconnecting and reconnecting

 

Some speculation as to what might be wrong:

* If these are connected to a backplane, then that becomes the obvious suspect.  Check for drives not firmly seated, perhaps vibrating loose.  Perhaps faulty backplane

* Check for loose cables, all connectors, for both SATA and power cables including any splitters

* It could possibly be a PSU problem, either it may be failing, or too many drives on one rail (this is not an expert opinion)

* I would especially suspect anything that is common to just these 3 drives, while keeping in mind that Disk 5 is the worst

 

At this point it is the trusting of my system that is the question.  I am not ruling anything out so i will check all the connections when i get home after work.

 

The PSU is a Corsair 550W so i should be fine there.  I have been running on this PSU with these drives for probably 3 or more months now.  It is gone through several parity checks without a problem. There are no backplanes involved so that can be eliminated from the equation.  The cables are of the locking type so them being loose is not likely the problem.

 

disk5 and disk6 are both plugged into the second controller on my Abit AB9 Pro board, but disk4 is on the main SATA ports.  I think the only thing that might be in common between the 3 disks is that they might all be on one of these.  I have another one at the house so that will probably be the first thing i check/replace.  If this does turn out to be a problem this will have been the first time i have ever had a problem with a cable from Monoprice, luckily they guarantee their cables so i can send it back if i like.  I am leaning towards this adapter being the problem now and that will be the first thing i check when i get home.