Hi all,

 

Somwhere in a parity check (which had some sync errors), I started getting errors on disk12.  I don't believe parity check completed.  When I rebooted, disk12 is now blue (all else green).  The array is "Stopped - Disabled disk replaced."

 

I am not sure what to do, because of the following:

1) I am not sure Parity is good

2) Disk12, although now disabled, IS good - I mounted it in a PC and copied off all contents using rfsTool

 

Any suggestions?  "Start" will rebuild disk12, which I don't believe is required

 

Thanks!

 

When a disk loses connection with unRAID, unRAID kicks that disk from the array.  When the disk connection is restored, it does not automatically rejoin.  unRAID will simulate the contents of the disconnected drive using parity and the other disks until the owner takes some action.

 

UnRaid will quickly forget about the physical disk altogether, and when a new disk is detected (even if is the same one), unRAID will quickly assume that the disk is meant to replace the failed one - and automatically assign it to the slot and show it with a blue ball.  unRAID wants to rebuild the contents of the simulated disk.

 

The first question you need to answer is why the disk was disconnected.  The best place to start is with a syslog.  It is important that the syslog be taken before a reboot, so your opportunity may have been lost.

 

Your next step should be to get a smart report from the disk to verify that the disk is not failing (see the wki for instructions on running smartctl).  If it is showing lots of reallocated sectors or the overall health of the drive is bad, the disk is failing and you need to replace it.

 

If the smart report looks okay, my suspicion is that your disk lost power - possibly a bad power splitter, but could also have just come loose.  It could also be a bad power supply or something more serious, but no reason to suspect that at this point.

 

If you've solved that mystery and know that the drive has good power and signal, you have two choices:

1 - Let unRAID rebuild the physical disk.

2 – Follow the “trust my parity” procedure (search in the wiki)

 

#1 is the best option if you have written data to the simulated disk.  Otherwise, anything written since the disk failed will be lost.

 

#2 is the best option is you are sure that the contents of the disk are correct/up-to-date, and don’t want to rely on accurate parity.  After this process, let unRAID’s automatic parity check run to completion to ensure that any disk descrepancies are addressed.  If sync errors occur, I’d suggest running a second one to confirm you have zero sync errors.

 

Thank you for the detailed answer... exactly what I needed.  Syslog is gone.  I physically was able to copy all contents of the disk onto another machine (500G) without error, so I am assuming the drive is not failing.  I do not trust parity to rebuild it since I had some sync errors during my last incomplete parity check.

 

I am pretty sure I have a power issue... I am always needing to jiggle cords, etc. to get the box to boot up, but haven't been able to isolate it.

 

I believe option #2 is probably the way to go.

 

Thanks again!

If you've solved that mystery and know that the drive has good power and signal, you have two choices:

1 - Let unRAID rebuild the physical disk.

2 – Follow the “trust my parity” procedure (search in the wiki)

 

#1 is the best option if you have written data to the simulated disk.  Otherwise, anything written since the disk failed will be lost.

 

#2 is the best option is you are sure that the contents of the disk are correct/up-to-date, and don’t want to rely on accurate parity.  After this process, let unRAID’s automatic parity check run to completion to ensure that any disk descrepancies are addressed.  If sync errors occur, I’d suggest running a second one to confirm you have zero sync errors.

I just wanted to emphasize what was said.

 

Fact #1.  The drive was taken out of service because a "write" to it failed.

 

Assumption...  The data that was going to be written to the drive NEVER was written to the drive.

Assumption#2. Therefore, the disk is NOT going to contain the correct contents of whatever you last wrote to that drive.

 

Even if you argue the "write" was only housekeeping bits.... they might make the difference between a corrupted file-system and a sane one.  (those housekeeping bits matter)

 

Unless you purposely wish to not have that "failed write" (and EVERYTHING WRITTEN SINCE THEN TO THAT DRIVE) then option #2, to force it back into service using the "trust" procedure is the not your correct action.  If you do force a drive back into service I think you must ALSO perform a file-system check on it, to be sure the reiserfs was not damaged by whatever was not written.

 

Your best option in almost all cases is to let unRAID re-construct the drive, option #1.

 

Joe L.

Well then this is not good... I already started option #2, which will be done in about 14 hours.

 

Regarding a corrupt file system (at least on Disk12), I don't think this is the case.  I did successfully mount the drive on another system and pulled all the data of it.  No errors, and from a random sampling of movies I tried, all is well with disk 12.

 

But considering my parity check failed to complete, the only assumption I could make is that parity was not reliable enough to rebuild Disk 12..... which I knew had good data on it.

 

Not sure what to do other than wait it out.

 

How does on perform a file-system check on it, to be sure the reiserfs was not damaged??

 

 

Well then this is not good... I already started option #2, which will be done in about 14 hours.

At least you are informed.

Regarding a corrupt file system (at least on Disk12), I don't think this is the case.  I did successfully mount the drive on another system and pulled all the data of it.  No errors, and from a random sampling of movies I tried, all is well with disk 12.

Using a different driver to "read" the disk is not exactly a definitive test for damage.  Good that it is readable, but would you know if it is un-dammaged?   Hard to tell.  It might just be one file that is missing... lots of possibilities

But considering my parity check failed to complete, the only assumption I could make is that parity was not reliable enough to rebuild Disk 12..... which I knew had good data on it.

Parity and the other disks are just "bits" when used to re-construct a drive.  It could be a freshly cleared and un-formatted drive being re-constructed...  The re-construction process has absolutely no idea of the contents of the file-system or what is in it, nor does it care.

 

for the process to fail a parity check something else happened... probably at the hardware level, but who knows...

Not sure what to do other than wait it out.

Yes.

How does on perform a file-system check on it, to be sure the reiserfs was not damaged??

Look in the wiki here: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

I'd like to make a general suggestion. If you have power splitters and you have some kind of intermittent problem it's time to just buy all new splitters. If you are converting some molex to SATA power then maybe try SATA to SATA splitters in case a power supply molex is bad.

 

If the SATA data cables are questionable then replace them too with new locking cables.

 

The server should just work with no issues day to day if you actually expect to recover from a real hard drive problem. Otherwise, with disks being flakey you'll likely have an issue right when you least want one.

 

Peter

 

Fact #1.  The drive was taken out of service because a "write" to it failed.

Although I agree that unRAID takes a drive out of service on a failed write, I also think it takes it out of service if the drive is simply unresponsive.  (My experience is that a drive is much more likely to become unresponsive over the hours, days, or sometimes weeks of sleeping than in the middle of a period of activity.)

 

Assumption...  The data that was going to be written to the drive NEVER was written to the drive.

Assumption#2. Therefore, the disk is NOT going to contain the correct contents of whatever you last wrote to that drive.

Even if you argue the "write" was only housekeeping bits.... they might make the difference between a corrupted file-system and a sane one.  (those housekeeping bits matter)

Everything that I have read about ReiserFS is that it handles crashes smoothly.  And based on my experience with the few hard reboots I’ve had to do, the filesystem has held up well.  In general, I’d rather rely on a disk that had not been cleanly shut down, than risk a disk failure in the middle of a lengthy rebuild procedure.

 

If I did want to do a disk restore in this situation, I wouldn't do it directly back onto the same disk if I could help it.  I would do the drive restore onto a new /spare disk and save the other disk as a spare until the procedure was successfully completed and the array fully protected again.  With that backup drive I'd have a fighting chance of restoring data even if I had a drive failure while it was rebuilding.

 

Unless you purposely wish to not have that "failed write" (and EVERYTHING WRITTEN SINCE THEN TO THAT DRIVE) then option #2, to force it back into service using the "trust" procedure is the not your correct action.  If you do force a drive back into service I think you must ALSO perform a file-system check on it, to be sure the reiserfs was not damaged by whatever was not written.

 

Your best option in almost all cases is to let unRAID re-construct the drive, option #1.

 

Joe L.

 

My preference would be to use the trust procedure.  It minimizes the amount of time that the array is operating without parity protection, minimizing risk of losing an entire disk (or 2 disks?) worth of data.  The cost is the small chance that some of the more recent file written to the disk might be incomplete (if the disk went offline right in the middle of a file write), or that you might need reiserfsck to repair some disk formatting issues.