Alright, so you guys are going to be my last hope.  I had a single drive failure which upon replacing the failed drive, corrupted nine of my data drives.  I decided to leave unraid because this was a bug I had experienced twice prior in 4.7.  So now I'm running 5.0.5 and trying to see if I can pull any data off my drives that I still have lying around.  I've read the wiki extensively and tried everything logical to recover it using the built in ReiserFS repair methods.  I know data is on my drives as I used a paid recovery program that was able to see the files, but want to see if I can repair the partitions before I shell out more money for a program that may or may not recover all my files.

 

I have a working unRAID server with no issues, I am just using it in Maintenance mode to repair my drives.  The drive I'm currently trying to recover is /dev/sdc  I've ran a SMART report and it is well within standard tolerances with no bad/reallocated sectors.

 

Right now, the best results I've gotten have been following the steps listed in this thread, http://lime-technology.com/forum/index.php?topic=5072.0

 

I ran the script Joe L. built since I never touched the partition table prior to loss of data.  I am now running into "No reiserfs metadata found." after running "reiserfsck --scan-whole-partition --rebuild-tree /dev/sdc"

 

Any idea where to go from here?  My Log is attached from the previous steps.  Many thanks in advance

reiserfs.txt

Alright, so you guys are going to be my last hope.  I had a single drive failure which upon replacing the failed drive, corrupted nine of my data drives.

 

I am having a problem understanding the genesis of your problem. How could a single drive failure corrupt 9 drives? When a drive is replaced, only that drive would be written to by unRAID.  Something else must have happened. Maybe you had a hardware failure. But there is nothing I can think of that would have simultaneously corrupted every drive in your system.

 

I do not have the technical expertise to help you. I am sorry. I once ran reiserfsck with Tom's help, and was able to recover my data after the beginning of the disk was zeroed out.

It's great to see users trying to fix their own systems without help, but sometimes it really is best to use the help of others, especially when it's Linux!  It does require more tech knowledge than you may be used to, and typing the wrong command or the right command imperfectly can quickly get you into deep trouble.  In this case, it's easy to get confused as to what commands apply to drives and which commands apply to partitions.  It looks like you typed the reiserfsck command to work on the drive /dev/sdc, but reiserfsck only works on partitions, and can really botch a drive up if wrongly applied.  The correct syntax should have been something like reiserfsck --fix-fixable /dev/sdc1, note the very important one at the end.  Your reiserfsck command may still work, but it absolutely requires sdc1 (the first partition), not sdc (the drive).

 

See Check Disk File systems.

 

I don't know your situation, so cannot comment very definitively, but I've never heard of a bug that could corrupt 9 drives.  It also seems extremely unlikely that reiserfsck  should be needed on more than one or 2 drives.  I really would like to see a syslog from that time.  Many errors that look like disk failures are not, just issues with power or cabling or controllers.

I am having a problem understanding the genesis of your problem. How could a single drive failure corrupt 9 drives? When a drive is replaced, only that drive would be written to by unRAID.  Something else must have happened. Maybe you had a hardware failure. But there is nothing I can think of that would have simultaneously corrupted every drive in your system.

 

My initial problem is documented here.  http://lime-technology.com/forum/index.php?topic=23723.msg208947#msg208947  Everyone including Tom threw their hands up and couldn't figure out how to fix it.  As you can see eventually my drives began showing up as unformatted and became unreadable inside Knoppix as well.  I can assure you there is and are no hardware errors except the drive that was causing a Kernal Panic.  I've been using the exact same hardware (with the exception of drives) over a year now with no issues.

 

The correct syntax should have been something like reiserfsck --fix-fixable /dev/sdc1, note the very important one at the end.

 

Problem, When running that Syntax on any of my drives I get the below results.  Also non unRAID Linux forums have consistently recommended repairing the partitioning tables before touching the partition.  Which logically makes sense, especially since my drives are reported as "not optimized for unRAID" by Joe L's partition script.

 

root@Tower:~# reiserfsck --fix-fixable /dev/sdc1
reiserfsck 3.6.24

Will check consistency of the filesystem on /dev/sdc1
and will fix what can be fixed without --rebuild-tree
Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

reiserfs_open: the reiserfs superblock cannot be found on /dev/sdc1.
Failed to open the filesystem.

If the partition table has not been changed, and the partition is
valid  and  it really  contains  a reiserfs  partition,  then the
superblock  is corrupted and you need to run this utility with
--rebuild-sb.

Unfortunately, I'm out of time at the moment, but I do feel bad for you.  I've just spent some time going back over the 3 threads and your syslog and it's perhaps the worst horror story I've seen here.  It looks to me like you got some bad advice, as all but one of your drives were probably completely fine, but may have been damaged by some of the advice.  I believe they were misled by the mistaken "Unformatted" indicators.  I need a little more time to think about it, after I'm off work tomorrow.

 

Just a thought, the Linux live CD "Test Disk" may be the best bet to fix the MBRs, properly locate the partitions.

Thanks Rob.  I'll look into the Linux Live disk and give it a shot.

Some random thoughts, probably not particularly helpful:

 

* On November 16 2012, 6:30pm, the syslog and image show a system that appears to me to be fundamentally corrupted.  Of 12 drives in the system, 6 of them show as 'Unformatted', the first and last 3, disks 1, 2, 3, 10, 11, and 12.  There is no way that 6 working disks, in full production, should suddenly be actually unformatted, so at that point we cannot trust anything the system tries to tell us.  The best thing that can be done is to shut the system off completely, no trying to fix anything, not even trying to diagnose anything, because it is not trustworthy, and because there is a huge risk that writing will be performed to one or more of the disks and/or to the flash drive.  Do not try to shut down, and if the power button normally starts a clean shutdown, don't use it either.  Find the power switch on the back or better yet, pull the plug.  We don't want the system to try and 'update' the flash or any of the disk file systems with misinformation.  Hopefully after it cools down a bit, a reboot will load a clean OS, and everything will work as it is supposed to.

 

* In earlier versions of UnRAID, there were conditions where upon starting the array, if for whatever reason a drive could not mount, it would be marked as 'Unformatted', and formatting options presented, an obviously very dangerous state that could cause data loss.  Most of these were fixed in the v4.5 series of releases (eg. see the Release Notes).  But it appears that in v4.7 (and later?), another rare condition must still exist.  In this case, it resulted in 6 drives showing as Unformatted.

 

* Syslog shows more ACPI issues than normal, a little concerning, needs a BIOS update.  This normally does not cause any problems, but since the syslog otherwise is rather clean, we're grasping at straws, looking for any reason at all why the system behaved so badly on Nov 12.

 

* Syslog shows this is running under v4.7, and there is a SAS 4 port card using the mvsas driver and the associated SCST subsystem.  Both of these modules were especially immature in that version, the early days of SAS support, and I really don't trust either of them.  And in fact, Tom didn't either and dropped usage of the scst subsystem, that's the last version it appeared in.  Current SAS support is much more mature and reliable.  I really don't know if this caused any of the problems then, but it did not help.

 

* There is no obvious issue at all in the syslog, until the drives attempt to be mounted.  Then 6 of them fail to be mounted, with exactly the same error message, an ambiguous one that does not tell us exactly what the problem was.  It does appear as if the system tried to read at the wrong location on each of the 6 drives, as if it wrongly used alignment or jumper or other info to locate the correct sector for the starting partition.  If they were WDEARS drives (and they aren't), I'd strongly suspect someone had just switched the alignment jumpers on the drives, but as far as I know, none of these drives have the jumpers.  If the alignment option had just been changed (eg. MBR-unaligned to 4K-aligned), that could possibly confuse the system, but it's not supposed to use that once a drive is online, already formatted and in use.  It should have just ignored it.  Somewhere it mentions that one of the drives was not set up 'optimally', but that does not matter once online.  Once a drive is a part of the array, 'optimal' or 'optimized' does not matter, jumper does not matter, alignment does not matter.  You don't try to change any of them, you live with whatever it is, or things can go disastrously wrong.  Peter, I believe, can correct me if I'm wrong on this.  I think the rule is, once it is in production, you don't mess with it, you live with it.

 

* The reiserfsck command, used on a drive rather than a partition, can seriously damage the file system.  This seems fundamentally wrong to me that that is possible.  I know that it wants to assume you know what you are doing, and that it can be directed against *any* device, but when requested to work on a disk device, it really should strongly warn the user that that is probably wrong, that it should probably be working on a partition of that disk instead (eg. sdc1 instead of sdc).  I'm not a Linux expert, but I cannot agree with the developer of reiserfsck on this.  I've seen too many file systems damaged even further than they already were.  I *will* take partial responsibility here, in that our instructions on the Check Disk File systems wiki page do not have sufficient warnings about this.  We already have a lot of red there, but a little more seems warranted.

 

* It looks to me as if some of the very well-intentioned users wanting to help you believed the screen showing so many unformatted drives, then tried to fix them.  I don't know for sure, but that seems to me how most of them were damaged. 

 

* There are 3 drives with HPA's, but since they are smaller drives, I don't think they are of any concern.

 

* After the system started the array and 6 drives were discovered appearing unformatted, you stopped the array and discovered that Disk 11 was showing as Missing, clearly wrong but then I don't trust anything this system was saying at that point.  Nothing about Disk 11 appears different in any way, than any of the other 5 disks that could not be mounted, except that the system on boot immediately tried to start the system AND rebuild Disk 11!  Seems to me there should have been a pause for confirmation here, not an autostart.  I cannot tell if Disk 11 was red-balled before this (syslogs do not reveal red ball status), but it seems bazaar that the system tried to physically mount Disk 11.  If it was trying to mount a reconstructed Disk 11, then that's even more bazaar that it could not mount it.  But then how can you mount a reconstructed drive when half of your other drives cannot mount, and later cannot even provide a temp.

 

Enough thoughts for now.  I really hope "Test Disk" helps you.

Wow RobJ! Excellent postmortem. This is the most catastrophic failure I have ever seen, and your ability to reconstruct the play by play is amazing. Many lessons in there about what to do and not do when things start to go wrong. I found the part about hitting the big red switch particularly interesting.

 

tggg - I feel for you! Good luck with your attempts to recover data. I am somewhat optimistic that you will have some successes, but fear it will be far from complete.

[me=DaleWilliams], feeling unworthy, bows low in the presence of RobJ, the master.[/me]

Rob I can't thank you enough for going through all that history and doing some good post analysis as a learning point for all.  I'll definitly report back if I can pull off some data recovery.

 

bjp, thanks for the feels.  I originally wasn't overly concerned with the data I lost as it mas mainly CD/DVD/BD rips.  Unfortunatly the personal data I lost during this failure was stored on a hard drive that got destroyed a few weeks after I gave up on trying to repair the server.