One thing that has occurred to me is that as this issue first occurred when trying to add new drives are there now any drives in the array with no data on them?    If so then this might mean it is possible to image a problem drive onto one of these before attempting recovery so that you can revert if necessary to the pre-recovery state.  It would add some time to the overall task to add the imaging step (via the 'dd' command) but does remove some of the risk.  If the recovery is successful then you would image the next disk in the same way and start recovery on that disk.

One thing that has occurred to me is that as this issue first occurred when trying to add new drives are there now any drives in the array with no data on them?    If so then this might mean it is possible to image a problem drive onto one of these before attempting recovery so that you can revert if necessary to the pre-recovery state.  It would add some time to the overall task to add the imaging step (via the 'dd' command) but does remove some of the risk.  If the recovery is successful then you would image the next disk in the same way and start recovery on that disk.

 

Certainly an option. I am just not sure what would be tried if this failed. Reiserfsck has proven to be a very good tool. I might feel differently if this were XFS or BTRFS, with which we have very little recovery experience.

One thing that has occurred to me is that as this issue first occurred when trying to add new drives are there now any drives in the array with no data on them?    If so then this might mean it is possible to image a problem drive onto one of these before attempting recovery so that you can revert if necessary to the pre-recovery state.  It would add some time to the overall task to add the imaging step (via the 'dd' command) but does remove some of the risk.  If the recovery is successful then you would image the next disk in the same way and start recovery on that disk.

 

Certainly an option. I am just not sure what would be tried if this failed. Reiserfsck has proven to be a very good tool. I might feel differently if this were XFS or BTRFS, with which we have very little recovery experience.

I was thinking that one would still be able to use Testdisk or an equivalent.

 

At this point it seems that the right thing to do is find the safest way to attempt recovery on the first disk. One can continue discussing options forever, but the only proof of any proposed approach is going to be to actually try it.  If reiserfsck recovers the first disk completely then one could continue with further disks with reasonable confidence, and not bother with imaging these further disks.    If it does not you have not (yet) prejudiced an alternative approach.

... Thanks Gary, you made this quinquagenarian's day! 

 

Okay, you got me.  I've always been VERY good at technical "stuff" and math [perfect SAT & ACT scores; college at 15; near 4.0 average; PhD at 20; etc.] ... but I think the brain cells have been deteriorating exponentially in recent years !!

 

... in other words, I had to look up quinquagenarian  :)

 

Apparently it's a neat word to let 50-something's talk as if they're old  :)

Trust me, I vividly remember when I thought THIRTY was old ... but these days my perspective has changed a LOT !!  Oh to be a quinquagenarian again 

 

 

...  image a problem drive ... before attempting recovery ...

 

... I am just not sure what would be tried if this failed. Reiserfsck has proven to be a very good tool ...

 

The image would, as itimpi later noted, allow the use of TestDisk if Reiserfsck should happen to fail.    If the goal is to use Reiserfsck first [which isn't a bad approach, as Reiserfsck has indeed, as Brian said, been VERY good at recovering data that seemed gone forever on many occasions], there's an easier way to preserve the ability to use TestDisk.    Simply replace the disk you're going to attempt recovery on with a new 2TB disk ... and let UnRAID rebuild it.    You'll now have the original disk sitting untouched on a shelf, while you do the recovery on its identical replacement.

 

While you could, in theory, do this for every drive, I suspect you'd only want to do it once.  The first drive attempted is either going to be (a) successful with Reiserfsck -- which is a very good sign that all of them will work well;  or (b) not work with Reiserfsck, in which case if you still have a way to try TestDisk you could try that ... and if it happens to work you'll then likely just use it for the rest of the drives;  or  ©  not work with either ... in which case things aren't looking so good.

 

Personally, if I had this system here, I'd remove one of the failed drives; attach it to another system; and try TestDisk FIRST.  This (a) eliminates the need to do anything special to preserve that option; and (b) doesn't alter the drive at all, so it can simply be replaced in the system and then analyzed by Reiserfsck.

 

But I definitely understand the desire to keep the recovery as simply and straightforward as possible -- and just using Reiserfsck isn't a bad choice.    It's VERY likely it will work if anything will ... the only downside is that once it's modified the disk you can't try other approaches.

 

Actually, I believe there's a middle-of-the-road approach ... Brian can correct me if I'm wrong; but I believe you can simply run Reiserfsck using the -check option and it will REPORT what it can do, but not actually make any changes to the disk.    This would at least let you then decide which way to proceed from there, without having modified the disk yet.    I don't think it actually SHOWS what it would recover in this case, but it will give you a "warm feeling" about whether or not it found a bunch of nodes it could reconstruct.

 

In any event, I suspect that the fundamental issue is the same for all of your drives.  So after the first drive you'll know for sure just how successful Reiserfsck is going to be      Whether you want to do something in advance to preserve the option of alternative recovery options;  or even just try one first (i.e. TestDisk) is entirely up to you.

 

 

 

I cant express in words how relieved I was to read your responses to my last message, especially the fact that among our mature members there is so much knowledge and experience. I was equally pleased to note that Brian is a more youthful man; good to have a young bloke on the job 

 

Good of Gary to look the word quinquagenarian which is clearly a very cool word to be using, sadly I am well beyond and probably not the only one either  .

 

So, I have putty, I also have screen, my windows box will remain on throughout the process so the putty window will stay up just like it did when I pre-cleared the drives.

 

I have printed out the array map through UnMain showing me the /dev/md1 and /mnt/disk1 and model/serial number. I know which physical drive is the parity drive.

 

Is there anything else I need to do apart from wait for Brian's instructions?

 

 

Peter

 

Sounds like you're ready to go.

 

I THINK (but do NOT do this without Brian's guidance/confirmation) that the command you need to use is:

 

reiserfsck --rebuild-tree --scan-whole-partition /dev/<partition>

 

where <partition> is the MDx device

 

What I'm not sure of is whether you can add the "scan whole partition" parameter to a -Check command ... i.e. something like:  reiserfsck -check -scan-whole-partition /dev/<partition>  to check the disk BEFORE you do any actual recovery on it.

 

 

One other comment (which I think Brian will agree with):  You COULD open multiple windows and run more than one reiserfsck command at once -- but I would NOT do that !!    I'd just be patient and do one drive at a time.

 

From your friendly local sexagenarian 

 

 

 

 

I will be doing one drive at a time

 

Thought I'd better pop in.  With apologies to Gary, the first commands have to set the environment up correctly, including PuTTY and screen, then the array state, then the disk drive state.  This is v4.7, so no Maintenance mode.  The instructions Brian will use should be the same as the ones in the Check Disk File systems page for v4.7, including the samba stop and the correct umount, and then the reiserfsck command.

 

Do not use the --check option, it will only check the new fresh format, and quickly tell you it's great, has an empty file system, and nothing at all to fix!  (The --check option is always used by itself.)

 

To avoid confusion, the options you need are --rebuild-tree and either -S or --scan-whole-partition.  They are exactly equivalent, one is just the short form of the other.

 

I always do things like this at the console, it's easiest and safest, won't crash.  But if that isn't possible, then you will have to set up screen first, and PuTTY in.

 

As to having an extra disk for rebuilding or imaging, he had some new disks available, freshly Precleared, but he just added them to the array, so that option is gone.  However, he has them in the array available for copying recovered files, if desired.  I'm not sure it is actually needed though.

 

I've been thinking about how these tools work, and I'm not sure copying to another disk really helps.  Normally, it IS good policy to only work on a copy, but in this case, the data is not being overwritten.  You can stop and abort the reiserfsck, and then run it again, and again if needed, because it is only going to use identified file system space and free space to build the new file system.  It has to start at the beginning by examining each sector/cluster/block and identifying what is data, what is former file system space, and what is unused free space, and apparently from the results we get, it is very good at doing that!  If it was overwriting or mis-identifying sectors, file recovery would be much worse than it is.  So I don't see a problem repeating a reiserfsck run, if it's needed.  As to TestDisk, it also has a great reputation at data recovery, and that's testified to by unRAID users somewhere.  But I wouldn't be surprised if it is actually using reiserfsck to do the job, or at least using code based on the reiserfsck code.  Why wouldn't it?  It was developed later, and would need to know how to analyze intimately the ReiserFS structures.  The ReiserFS use of tail-packing really complicates total data recovery.

Rob -- thanks for the details.    I wasn't sure whether you could use -check with the -S option to force a scan of the whole disk or not.    Thanks for confirming you can't do it that way -- and of course if it just scans the current used data it's going to say all is well, since it's a freshly formatted partition.

 

Certainly seems like Peter is ready to go here.

 

Peter =>  Rob is VERY good at this stuff as well.    It's pretty clear at this point that the command you need is:

 

reiserfsck --rebuild-tree -S /dev/MDx

 

... where x varies from 1 to 11 as you step through the disks.

 

As soon as Rob or Brian confirm that exact command, you're ready to start !!

 

I DO agree it's best to do this from a console at the actual server; but it's fine to do it via Screen, which I believe won't cause an issue should the PC you're using inadvertently shut down the window.    THAT is something you need Rob or Brian to walk you through setting up correctly, as I've never done that (I simply use a console).    If you do it through Putty without using Screen it will also work; but if the Putty window is closed it will abort the command.

 

 

It's pretty clear at this point that the command you need is:

 

reiserfsck --rebuild-tree -S /dev/mdx

 

... where x varies from 1 to 11 as you step through the disks.

That is the correct command.  I assume you will start with Disk 1, so it will be

  reiserfsck --rebuild-tree -S /dev/md1

This is Linux so capitalization is important, only the -S is capitalized.

 

I DO agree it's best to do this from a console at the actual server; but it's fine to do it via Screen, which I believe won't cause an issue should the PC you're using inadvertently shut down the window.    THAT is something you need Rob or Brian to walk you through setting up correctly, as I've never done that (I simply use a console).    If you do it through Putty without using Screen it will also work; but if the Putty window is closed it will abort the command.

I have to confess, I too have never used screen, never needed it as I always use the console.  Probably should have tried it, just for the experience.  So I can't help you with the screen commands, but somewhere there are clear instructions for it.  But I'm afraid my screen instructions below are rather vague.

 

So here are the steps, for each disk, changing md1 and disk1 appropriately for each drive done:

  (please check these against the instructions in Brian's post and the v4 Check filesystems section)

* Install screen (if you haven't already)

* Connect using PuTTY and the correct screen commands

* Start the array (the unRAID engine has to be running so parity will be preserved)

* Then type these commands in the PuTTY session:

  cd  (it just makes sure you are in the root folder)

  samba stop  (stops all sharing, both disks and user shares)

  umount /dev/md1  (that's umount not unmount!  reiserfsck expects it to be unmounted, and will handle all mounting needed)

  reiserfsck --rebuild-tree -S /dev/md1  (does the recovery, will take a LONG time!)

* You can monitor the progress and results in the PuTTY screen

* Once it indicates it is finished, it should say something about how much was recovered, which will be your first clue as to how well it worked

* Restore normal operation by typing the following commands:

  mount /dev/md1 /mnt/disk1  (important to match up the 'md1' with 'disk1', 'md2' with 'disk2', etc)

  samba start  (restores normal sharing)

* Done!  well, except for the rest of the drives, and figuring out what to do with the 'lost+found' folder and what has to be renamed and moved!

* At this step, you can continue with the next drive by returning to the first step, or you can begin the work of restoring files with the wrong name or location to their rightful places (see Brian's post for more help).  If you prefer to recover the next drive, at least make sure to examine this drive first, enough to feel that it was successful (even if a little more work is needed).

Screen is easily installed in 4.7 with unmenu package manager.

 

After installing, you run "screen". It will present lengthy message, and you press "space" to commence. You are then inside a telnet session inside of screen. You can type in your long-running command at this point.

 

To "detach", you press "ctrl-a d". Screen will exit, but whatever command is running will continue running.

 

To resume you would run "screen -r" from outside of screen.

 

Other useful commands ...

"ctrl-a c" create a new telnet session within an existing screen session. Useful if you are running more than one lengthy command at the same time.

 

"ctrl-a space" cycles from one screen telnet session to another.

 

If you "exit" from all telnet sessions inside screen, screen will exit.

 

Let me know if any questions.

Brian & Rob

 

Fortunately screen is already installed and I have used PuTTy to preclear most of the disks before without any issues.

 

Is it going to be OK for me to copy and paste the commands Rob has typed into the instructions or do I need to type them myself? I understand the commands I follow are the ones in bold print.

 

I intend to start tonight (10.30pm now) and let it run all night, and will report my progress in the morning.

 

Peter

Update....please cancel my last question, Drive1 is now being worked 

 

When it finishes, you should stop and restart the array before trying to look to see if files were recovered.

UPDATE

 

Drive 1 is restored with what I would call minuscule losses of little importance. Firstly, thank-you, without all your valuable assistance I was best described as ship dead in the water.

 

I used PuTTy with SCREEN (must ensure windows does not go into sleep mode) to undertake the login which was ideal for me being of shall we say of advancing years, the plus side of using it for me is, I can use my mouse, and also copy and paste, that reduces the chances of me making a typo.

 

I trusted what Rob had typed and copied and pasted it straight into PuTTy and followed instructions to the letter it worked its magic.

 

reiserfsck in my opinion is awesome I watched it perform it carried out 4 passes of the disk and uses the term Flushing as part of the description of its process. The process was started around 10.30pm and concluded about 6.30pm the following day.

 

Brian, thank you for the message advising me to stop the array and restart it before examining the results. I would not have done that otherwise.

 

Key Learning points

 

If you make a mistake or something happens that leads to a major mistake, don't panic and do things that you think is going to help. The chances are it won't, and you will minimize the chances of recovery and make things much harder for the people here to help you. Don't write anything to any disk affected.

 

Ask for help here, be totally honest with what has happened and, provide,any information requested by the experts here. Follow the instructions to the letter, if your not sure ASK. Above all dont do anything without guidance.

 

I will update progress of the next 10 drives, yes, that's what I said a further 10 drives that were formatted.

 

Peter

 

 

 

 

Shouldn't matter if windows goes to sleep (or if you get disconnected for any other reason) if you are using screen. You can just run putty again and reconnect to the screen session. That is the reason it is recommended for long processes.

Glad to hear that the first disk has gone well!  I guess that means that you should sleep better tonight knowing that a good recovery is just going to be a matter of time and patience 

 

As you say the amount of damage that reiserfsck can recover from is awesome.

I've seen numerous examples in this forum of folks recovering data with Reiserfsck with miraculous results.    That's one reason I'm not as big a fan of moving everything off of Reiser as many folks seem to do as soon as they move to v6, which supports other formats.

 

In any event, it's good that the recovery went well -- and that means it's almost certain the rest will be just as successful.    So at this point you just need patience and ten more days 

 

At one disk/day you'll be completely recovered then.    You might also want to consider a good backup strategy      [ http://lime-technology.com/forum/index.php?topic=31020.0 ]

UPDATE

 

Drive 1 is restored with what I would call minuscule losses of little importance. Firstly, thank-you, without all your valuable assistance I was best described as ship dead in the water.

 

Great news!

 

Brian, thank you for the message advising me to stop the array and restart it before examining the results. I would not have done that otherwise.

 

When it happened to me, I had no direction to do this, and initially thought that there was no recovery at all.

 

Key Learning points

 

If you make a mistake or something happens that leads to a major mistake, don't panic and do things that you think is going to help. The chances are it won't, and you will minimize the chances of recovery and make things much harder for the people here to help you. Don't write anything to any disk affected.

 

Ask for help here, be totally honest with what has happened and, provide,any information requested by the experts here. Follow the instructions to the letter, if your not sure ASK. Above all dont do anything without guidance

 

Couldn't have said it better myself! You are now prepared for any future curve balls unRAID might throw at you!

 

Disk 2 recovered 

Very nice => I have no doubt you'll be able to post one note/day for the next 9 days documenting all eleven of the disks being successfully restored  :)

 

Disk 3  recovered

Wonderful!

 

Just curious, how much are you finding in 'lost+found'?

Wonderful!

 

Just curious, how much are you finding in 'lost+found'?

Rob you must have a sixth sense, I have just recovered Drive 4, its the first drive that has been completely full in Lost+Found. It will certainly take a while to sort out.