Ever had one of those moments when you clicked “ok” and knew an instant later you had made a HUGE mistake?  I had one of those moments with unRAID.  The purpose of this post is to let others know how easy it is to make this big blunder – and to provide information as to how to recover.

 

I was running my array unprotected doing the initial load.  All data was backed up.  Loading the array had gone flawlessly, and after using it for several days my confidence was high.  I was ready to do the parity sync.  I decided to remove 2 750G drives from my desktop machine first.  It would make formatting them go so much quicker, and would only have to build parity once.  These drives contained the lion’s share of my backup data.  I added them without a problem, and formatted them.  5.3Tb of data.  Sweet!  (Lesson 1 – don’t delete your backups, until the array is parity checked.) 

 

I started to run parity sync, and then realized that one of my drives was stuck “mounting”.  Things quickly went from bad to worse, and after bouncing the array to try to get the drive to mount, one of the other drives showed “unformatted”.  One more bounce and ALL the drives were stuck mounting.  Crap!

 

Searching the forums yielded a suggestion.  Delete the superio.dat file and reset the disk.cfg files.  After doing this, unRAID came up fresh, with no drives in the array.  I decided to assign each drive – one at a time - to the disk1 slot to see if it was okay.  So I went to assign my first disk to the disk1 slot and start the array.  Here is where I made the big blunder.  I accidentally assigned the drive NOT to disk1, but to PARITY.  This is a remarkably easy mistake to make, as parity is at the top of the disk slots.  (Lesson 2 – Carefully review the unRAID status screen BEFORE starting the array).  I started the array and then realized instantly what I had done, and in a panic hit the big red switch on the server.  Crap!!!  (I realize now there was a button on the GUI to stop parity sync, but I wasn’t thinking so clearly.  Killing power was what came to mind first!)

 

I again reset disk.cfg (to prevent it from coming up and continuing its parity building) and booted unRAID.  I tried to mount disk1 and it showed “unformatted”.  This was a disaster!  But as bad as losing that one disk was, I was still worried about all the other drives.  I put this disk aside and continued my investigation, assigning each disk in turn to disk1 (NOT PARITY) in the array.  What I found is that both drives on one of my IDE channels were not functioning.  I heard “pinging” coming from the drives during unRAID booting, and saw nasty I/O errors in the syslog.  I decided to try replacing the IDE cable – and got the first good news of the day.  The drives were fine, the cable was bad.  Lesson 3 – DO NOT USE the round IDE cables.  I had seen this posted by several users, but ignored it since my array was working fine.  I repeat, DO NOT use the round IDE cables.

 

So back to the first disk that had been in the parity slot for a short time.  Luckily Tom was on-line and helped me figure out what to do.  I ran “reiserfsck /dev/md1” (UPDATE 8/8/11: If you ever run on on an OS level device rather than an unRAID device, make sure to use the first partition - ie., “reiserfsck /dev/sdc1", and not the disk level device. UPDATE 7/24/15 - unRAID now supports multiple drive formats, including RFS, XFS and BTRFS. reiserfsck is ONLY for RFS (reiserfs) formatted drives!)  (and it questioned whether this was really a reiserfs formatted drive.  It told me to run with the “--rebuild-sb option” if I was sure.  If ever you have to run with this option, it asks you a bunch of questions.  Here are the correct responses (many thanks to Tom for helping me through these):

 

1. The version of reiserfs is 3.6.x.  This is for unRAID 4.2.1.  (This is NOT the default, so be careful)

2. Block size is 4096 (default)

3. “No journal device was specified.  (If journal is not available, re-run with --no-journal-available option specified)”  Is journal default?”  (Answer Y)

4. “Do you use resizer?” (Answer N)

5. It tells you that a new uuid has been generated. 

6. “rebuild-sb: You either have a corrupted journal or have just changed the start of the partition with some partition table editor. If you are sure that the start of the partition is ok, rebuild the journal header. Do you want to rebuild the journal header?”  Answer Y

7. The following info is displayed:

 

Reiserfs super block in block 16 on 0x901 of format 3.6 with standard journal

Count of blocks on the device: 73264320

Number of bitmaps: 2236

Blocksize: 4096

Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 0

Root block: 0

Filesystem is NOT clean

Tree height: 0

Hash function used to sort names: not set

Objectid map size 0, max 972

Journal parameters:

        Device [0x0]

        Magic [0x0]

        Size 8193 blocks (including 1 for journal header) (first block 18)

        Max transaction length 1024 blocks

        Max batch size 900 blocks

        Max commit age 30

Blocks reserved by journal: 0

Fs state field: 0x1:

        some corruptions exist.

sb_version: 2

inode generation number: 0

UUID: <my UUID removed>

LABEL:

Set flags in SB:

Is this ok ? (y/n)[n]:

 

8. Answer “Y”. 

9. With amazing speed the program does its thing and ends

 

Although a small victory, problems were still ahead.

 

I then needed to get reiserfsck to search the entire drive to try and rebuild the directory structure.  I ran the program

      “reiserfsck –scan-whole-partition –rebuild-tree /dev/md1”.  (UPDATE 8/8/11: If you ever run on on an OS level device rather than an unRAID device, make sure to use the first partition - ie., “reiserfsck /dev/sdc1", and not the disk level device. UPDATE 7/24/15 - unRAID now supports multiple drive formats, including RFS, XFS and BTRFS. reiserfsck is ONLY for RFS (reiserfs) formatted drives!)

 

It started to work but quickly got a nasty physical I/O error and halted.  The disk had a bad sector!  This was unreal!

 

So … I ran SpinRite.  In only about 30 seconds (blink of an eye in SpinRite), SpinRite encountered a bad sector and went into a TWO HOUR recovery process, until it finally remapped the bad sector.  The entire rest of the disk was fine.  I ran SpinRite a second time – no errors.  My guess is that when I hit the power switch, that the disk was in the middle of a write operation and somehow I physically damaged that spot on the disk.  This is consistent with the error being so close to the front of the drive.

 

Back to reiserfsck.  I ran it with the above options and it ran overnight.  In the morning it had finished.  It had spit out a ton of informational message of anomalies it had encountered.  I had to bring down the array and restart it to see the results.  It was good news.  Although the root of the drive had dozens of generated directories and garbage files, most of the data was organized in 2nd level subdirectories which were intact!  I was shocked (so was Tom).

 

I copied all the data to another drive and all is well.

 

I had to do a big case mod this weekend so that I could mount the drives and use the frustratingly short flat IDE cables.  After completed my case mod, I was able to load all the disks without difficulty.  It’s running very cool too – temps in the mid-20C’s on all drives.

 

Now that all the data was safe, I wanted to reformat the drive before running parity sync.  I assigned it to the parity slot for a few seconds and magically it was unformatted.  I then let unRAID format it.  I started parity sync which has now been running for 8 hours.

 

This was my mistake, but I have a few suggestions for Tom of things to do to help idiot-proof unRAID.  See them in a separate posting under suggestions (http://lime-technology.com/forum/index.php?topic=1485.0).

 

Hope this never happens to you – but hopefully it gives you some confidence that recovery of data is possible even under some pretty terrible situations!  Don’t give up.  Remember it is easy to make things worse by trying things in the heat of the moment.  Give yourself some time and seek advice before taking a step that you might later regret.  Try not to write to a disk you are trying to recover until you are SURE that there are no other options.  Once you start writing, there is no way to get back.  If you take your time and do the right things, the chances of recovery are high!

 

-Brian

 

Thank you very much for a terrific and detailed write up.

 

thanks for all the info

 

(and you squeezed some "SpinRite" stuff in there again)

 

 

 

 

I probably sound like a Spinrite fanboy.  It has gotten me out of several pickles over the years (I've used it since v5.0).  But have to say it let me down today. 

 

I ran my parity build, it finished successfully, but I noticed that there were 6 errors on an almost new WD 750G drive.  That disk is also completely empty.  There were no errors on any other drives.  The syslog gave me the exact sector numbers that it had trouble with, and I computed a % for SpinRite so it was able to start reading just a little before the error.  It hit the error, and started to try to recover, and then Spinrite crashed with a divide by zero error.  Spinrite is a little long in the tooth - in 2004 (the date on the latest .EXE file), the biggest drives available were maybe 320G or 400G.  It isn't surprising that it doesn't work 100% on a 750G drive.  I'm looking to get a newer tool, but haven't done enough research yet to pick one.  There's one called "HDD Generator" that I learned about on this forum (was that you who recommended it?) and another called "DiskPatch" that I saw favorably reviewd vs Spinrite.

 

The drive is Western Digital, and I downloaded their LifeGuard tool and am running it now.  I think it will also fix the drive, but is much less flexible.  The only option is to do the whole drive, which is going to take >2hours.  If this doesn't work I think I'll try getting HDD Generator.  It has a 30 day money back warranty, so if I don't like it I can return it.

 

-Brian

Well - the non-destructive lifeguard tool did nothing.  I let it go to about 1/2 way through the drive (the bad sector was only 28% into the drive), and then tested with Spinrite, and the bad sector was still there.

 

So I tried HDD Regenerator 1.51.  It looks like crap compared to SpinRite!  But - it remapped the sector!  I don't think it spent any time trying to do a data recovery of the sector - it just remapped it instantly and kept right on truckin'.  I can't say I'll be using it instead of Spinrite, but at least I have a plan B if Spinrite doesn't work!

 

Not sure why, but unRAID wanted to do a parity check - I was going to start it myself, but it went ahead and did it on its own.

 

Finally, after a long time testing, I run into the same fault.

Accidently I put one of my data drive into the parity slot.

 

After unRAID started parity sync, and I realized what I did, I immedietly click on stop button, but after reorder the drives, that data drive still comes up, as unformatted.

 

So I guess, I have to go trough this procedure. My question is, that in case of unRAID 4.3.3 all the answers above are still the same?

 

Thank you!

So sorry to hear you had this problem. I am glad that you were able to find this thread. Can I ask how?  Did you search or find a link in the FAQ or Best of the Forums page?  Just curious.

 

To answer your question, I am 99% sure that nothing about the ReiserFS settings have changed. You could send a note to Tom referencing this thread and ask him to confirm.

 

Keep a cool head and take your time. Don't forget to stop and restart the array after it is complete (or even reboot). Otherwise it will look like nothing was recovered.

 

Good luck!

Thanks bjp999.

 

I found this topic in the Best of the Forums page. I read it before it happened to me, so it is immedietly caming into my mind, as it happend.

Same bad feeling...

 

I started the process 12 hours ago, still ongoing.

It is displaying a lot of very scary messages. I am really shocked seeing them.

 

I am wondering what comes out of that at the end... 

I had lots of scarey messages too. My recovery was not perfect, but I did retrieve most of my data.

Well, bjp999, I owe you with a huge thank you for this how-to. Most of my data seems recovered at the end.

 

Although I am a bit disappointed by reiserfs, as seems also lots of my data lost, in spite there was no writing activity to the disk (accidently started partity sync was stopped after a second). Surely I am not an expert regarding this, but I thought this one second was enough only for formating the drive, so there is a big chance to get almost all my data back. But there is lots of directory and even files without name and extension, and even lots of missing files.

 

One more question: is there any way/method to check whether if the recovered files are corrupted, or not?

 

Nevertheless, again, bjp999, I am very glad to find this thread of you, was really helpful! THANK YOU!

 

 

Well, bjp999, I owe you with a huge thank you for this how-to. Most of my data seems recovered at the end.

 

Excellent!

 

Although I am a bit disappointed by reiserfs, as seems also lots of my data lost, in spite there was no writing activity to the disk (accidently started partity sync was stopped after a second). Surely I am not an expert regarding this, but I thought this one second was enough only for formating the drive, so there is a big chance to get almost all my data back. But there is lots of directory and even files without name and extension, and even lots of missing files.

 

I don't have much of a point of comparison with other file systems, but if you wiped out a FAT* file system file allocation table I am not sure you'd have a prayer of getting your data back.  I'm also not sure that tools like this for rebuilding the drive are available for other file systems.  In short, I'm not sure whether to be impressed or disappointed.  But it is what it is, and you and I did the best we could and saved at least some of our valuable data.

 

Tom, if you are reading.  Please differentiate the parity slot from the data slots on the devices page to make this harder to do by accident!

 

One more question: is there any way/method to check whether if the recovered files are corrupted, or not?

 

Not really.  Some of my data was .zip files, and I was able to test the integrity of these files.  It gave me some confidence that the other data was intact.

 

Nevertheless, again, bjp999, I am very glad to find this thread of you, was really helpful! THANK YOU!

 

The fact that it helped is all the thanks I need.  Glad you found the "Best of the Forums" section.  Without that and the FAQ, finding information here would be quite difficult.

Although I am a bit disappointed by reiserfs, as seems also lots of my data lost, in spite there was no writing activity to the disk (accidently started partity sync was stopped after a second). Surely I am not an expert regarding this, but I thought this one second was enough only for formating the drive, so there is a big chance to get almost all my data back. But there is lots of directory and even files without name and extension, and even lots of missing files.

The parity drive does not have a file-system, therefore it is not formatted... Instead, the speed of writing to parity on most of  our machines is somewhere between 15 and 70 Mb/s. 

 

If you only wrote to the parity drive for a second, then you only destroyed the first 15 to 70 MB of it.  A lot if that is the initial part of the formatting of the data drive that you put in the parity slot.  The fact that any files at all are able to be recovered is a small miracle.

 

Joe L.

Tom, if you are reading.  Please differentiate the parity slot from the data slots on the devices page to make this harder to do by accident!

 

Good idea... additionally, Tom might use the "vol_id /dev/sda1" command to determine if a file system of ANY type is discovered on the first partition of the disk being initially assigned to the parity role.  If it does, then a warning of the file system type detected, and a checkbox to confirm it is OK would save a lot of folks from this kind of problem.  Somebody can still mis-assign the disks, but at least they will be warned.  The checkbox would allow the re-use of drives when the re-allocation of a data drive for use as the parity drive is on purpose..

 

Joe L.

I think I may have triggered the same situation where a disk is showing as "unformatted", but not because I accidentally assigned it to the parity disk. This disk was previously disk1, but was removed and a new disk was put in disk1 slot and rebuilt. Subsequently the parity drive was "missing" (still physically connected) and the old disk1 was re-attached to the machine. When booted, unraid automatically put the old disk1 as a NEW parity disk. It did not start the array, but did the act of simply assigning it as a parity disk (without starting the array) wipe out the reiserfs superblock? I think I need to do reiserfsck --rebuild-sb, but are all the answers to questions (especially reiserfs version) still the same in 5.0b6a?

 

Just wanted to thank bjp999 for this thread.  I had a data disk fail & a failing parity disk in a 4 TB array.  I was able to under stand enough of what is written to not only save my data & loose 1TB in the process (only replaced the parity disk as of now) .  Whew!