Format button when adding new disk - lost all data?

[Cessquill]

Had a quick search, but am currently panicing a bit too much to focus.

 

Just added a 10th 1TB drive (9+parity) to my 4.5.3 system because all were full.  As I only add one every 6 months or so, I'm not completely au-fait with the procedings.

 

The disk was recognised. It took 4 hours to clear, then I was prompted to format all unformatted disks. Clicked on it. All disks formatted.

 

I was half tired, but half not reading, since I was always led to trust the "your data will not be lost" principle of this system.

 

As you can tell by now, all is not good. All disks formatted. All data and shares appear gone. Is there anything I can do, or is that it?  I've turned it off for now in panic, and I'll go searching to see whether this is an issue. First glances suggest the software has not held up its end of the bargain.

 

Please tell me there's some way of recovering just under 8TB of data... PLEASE!

 

Cheers, Simon

[Cessquill]

Hi - after reading a few other posts, I've got the following...

- Afraid I didn't notice the fact that all drives must've been listed as unformatted. My bad, but in my defence I have always trusted the system to maintain data integrity. I'm not a linux user - just someone who stores his movies, and I pretty much did what I've always done when adding new drives.

- I'd always expected to tick a confirm box when doing something major, so when a "some of your drives are unformatted, click here to format them" message appears, I click it thinking that it knows what it's doing.

- Box was turned off the minute I saw all drives blank.

 

Is there any restore command I can apply? Please say yes!

[papnikol]

dont panic too much. i haven't had an unfortunate experience like yours but i have read many accounts of other people to whom it has happened. there are ways to recover almost all data. just dont write anything on your drives. i think the solution requires invoking the reiserfsck command which recovers the file tree. i cannot tell you much more but you should certainly check for similar solutions in the forum, i know they exist.

 

I think the most helpful instructions are in this message by Joe, which addresses the same problem as yours:

http://lime-technology.com/forum/index.php?topic=6008.msg57379#msg57379

[purko]

Please tell me there's some way of recovering just under 8TB of data... PLEASE!

Is there any restore command I can apply? Please say yes!

Email [email protected]  Give them all the details.

[Joe L.]

Hi - after reading a few other posts, I've got the following...

- Afraid I didn't notice the fact that all drives must've been listed as unformatted. My bad, but in my defence I have always trusted the system to maintain data integrity. I'm not a linux user - just someone who stores his movies, and I pretty much did what I've always done when adding new drives.

- I'd always expected to tick a confirm box when doing something major, so when a "some of your drives are unformatted, click here to format them" message appears, I click it thinking that it knows what it's doing.

- Box was turned off the minute I saw all drives blank.

 

Is there any restore command I can apply? Please say yes!

Whatever you do, DO NOT WRITE ANY NEW FILES TO THE DRIVES YOU ACCIDENTALLY FORMATTED.   

DO NOT PRESS THE BUTTON LABELED AS "restore" AS IT WILL JUST DELETE YOUR CURRENT PARITY AND DISK CONFIGURATION.  It is NOT the way to get back your data.

 

You can get your files back.

 

You will need to perform this set of steps for each of the drives you accidentally formatted (I'll use disk1 and disk2 as an example, change the number for each of your other disks in turn):

 

Turn off any spin-down timers.  We do not want the disks to be spun down in the middle of the repair.

Turn off any sleep/suspend screen savers on your PC if doing this via telnet.  Don't want it shutting down in the middle of a repair step.  It might be better if the following commands are typed on the unRAID system console directly, but as long as your PC does not shut itself down, you'll be fine using telnet.

 

Stop samba (you will not be able to un-mount the disk if it is busy... and samba will keep it busy)

/root/samba stop

un-mount the drive. 

umount /dev/md1

Rebuild the file-tree scanning the entire disk for files.

reiserfsck -–scan-whole-partition -–rebuild-tree /dev/md1

Respond with "Yes" when prompted... Upper-case "Y", lower case "es"  The scan can take many hours for a large drive. 

 

Re-mount the drive:

mount /dev/md1 /mnt/disk1

 

un-mount the next drive. 

umount /dev/md2

Rebuild the file-tree scanning the entire disk for files.

reiserfsck -–scan-whole-partition -–rebuild-tree /dev/md2

Respond with "Yes" when prompted... Upper-case "Y", lower case "es"  The scan can take many hours for a large drive. 

 

Re-mount the drive:

mount /dev/md2 /mnt/disk2

 

Do the above steps for each of your disks, then....

when all done, you should be able to re-start samba

/root/samba start

and see if your files have been restored.  Files in the top-most directory will probably end up in a lost+found directory created by the restore process.  You'll need to re-name them to their correct names.  Files lower in the directory hierarchy will probably have their correct names.

 

While this is all happening, write to lime-technology describing how the "unformatted" bug has caused you to lose all your data and request it be fixed as soon as possible. 

 

Joe L.

[Cessquill]

Thank you so much guys - I'll get that under way.

 

As a side issue (and not unraid related), I've just got DVDs backed up on there. Does anybody know of any tool that can verify that the contents of each folder? I'd rather not watch each movie in turn if I can help it.  I guess I could point something like dvdShrink at each folder in turn, but you never know.

 

Anyway. Thanks again - that's given me something to do today.

[calvinandh0bbes]

i just replaced my parity drive and added the old parity drive into the array yesterday.  upon preclearing my old drive and stopping the array and adding it to the list, when going back to the main screen, every single one of my drives showed as unformatted.  i usually get 1 or 2 extras, but i had never seen every drive show up as unformatted.  a quick refresh fixed it, but I agree, this is a monumental bug that needs to be fixed ASAP.  (i'm running 4.5.1)

 

c+h

[Joe L.]

i just replaced my parity drive and added the old parity drive into the array yesterday.  upon preclearing my old drive and stopping the array and adding it to the list, when going back to the main screen, every single one of my drives showed as unformatted.  i usually get 1 or 2 extras, but i had never seen every drive show up as unformatted.  a quick refresh fixed it, but I agree, this is a monumental bug that needs to be fixed ASAP.  (i'm running 4.5.1)

 

c+h

Disks being reported as "Unformatted" have always been an issue, even in older versions of unRAID.  Most times in the past it was when one or more disks was "busy" when an attempt was made to stop the array.  The disks that successfully were un-mounted were incorrectly reported as un-formatted.

 

The likelihood of somebody pressing the "Format"  in that situation was fairly low.  They were trying to stop the array, not add a new disk.

 

Occasionally, upon start of the array, a file-system takes more time to mount than unRAID allocates before deciding it did not mount.  We've seen system logs where the replaying of a reiserfs journal upon array start took 5 minutes or longer.  The "mount" would show un-formatted for those disks until they finally mounted.  Again, this was infrequent, and rarely was a user attempting to add a new disk at the same time, so it did not catch as many users who then pressed the "Format" button.  It is likely that this is what you saw in the 4.5.1 version of unRAID.

 

This new variant of the un-formatted bug seems to be far more common, and it shows when users are adding new disks...  In that respect it is one of the worst types of bugs I've seen in unRAID in the past 4 1/2 years.

 

Joe L.

[WeeboTech]

This new variant of the un-formatted bug seems to be far more common, and it shows when users are adding new disks...  In that respect it is one of the worst types of bugs I've seen in unRAID in the past 4 1/2 years.

 

Joe L.

 

I think it's the same bug. There is a race condition now.

A timing issue with the mounts taking longer then it used to in the past.

I've seen this condition on startup in the past. Just not as frequently.

I wasn't adding a disk. I just rebooted the server.

 

[calvinandh0bbes]

Again, this was infrequent, and rarely was a user attempting to add a new disk at the same time, so it did not catch as many users who then pressed the "Format" button.   It is likely that this is what you saw in the 4.5.1 version of unRAID.

 

This new variant of the un-formatted bug seems to be far more common, and it shows when users are adding new disks...  In that respect it is one of the worst types of bugs I've seen in unRAID in the past 4 1/2 years.

I can say without a doubt that when adding the last 3 drives (three separate occasions), more than just the 1 new drive showed up as unformatted upon restarting the array after assigning the drive. I've only ever used 4.4.2 and 4.5.1 final releases.  I know that the last two I've added have been under 4.5.1, I can't recall if the 1st of those three was 4.4.2 or 4.5.1.

 

Given that I just used up my 6th and final MB SATA port, my next drive will require an add on card, which will likely be the super micro card that requires 4.5.3.  Hopefully the next version will be out by then with the fix. (off topic, this next one will be expensive for me.....pro key, SATA card, and the drive  ).

[Rajahal]

(off topic, this next one will be expensive for me.....pro key, SATA card, and the drive  ).

...and the appropriate SAS cable.  SAS to SATA forward breakout most likely, or SAS to SAS if you are using a backplane that can take it.

[Cessquill]

Restoring the third disk now. All appears OK,although I'm not familiar with the reiser output.

 

Unfortunately, I have to say that I rarely visit the forum unless something is wrong (so top marks to the regulars that keep the wheels turning).

 

I've always absolutely trusted official releases, but now I'm a little insecure. Sure, I could have picked up the error if I had my wits about me, but I was awaiting the usual 4 hours drive clear to finish before I clicked on format. I'd grown used to the fact that, to do something major, you had to unlock it with a checkbox.

 

The restores so far have listed about 250 unreacable items deleted - is this something to be worried about?

 

By the way, if anybody else is doing this who is unfamiliar with dealing directly with the unraid console, if the screen goes off, don't panic! It's just a screensaver. Pressing a cursor key brings it back.

[Joe L.]

Restoring the third disk now. All appears OK,although I'm not familiar with the reiser output.

 

Unfortunately, I have to say that I rarely visit the forum unless something is wrong (so top marks to the regulars that keep the wheels turning).

 

I've always absolutely trusted official releases, but now I'm a little insecure. Sure, I could have picked up the error if I had my wits about me, but I was awaiting the usual 4 hours drive clear to finish before I clicked on format. I'd grown used to the fact that, to do something major, you had to unlock it with a checkbox.

 

The restores so far have listed about 250 unreacable items deleted - is this something to be worried about?

 

By the way, if anybody else is doing this who is unfamiliar with dealing directly with the unraid console, if the screen goes off, don't panic! It's just a screensaver. Pressing a cursor key brings it back.

The rebuild-tree scans the entire file-system.  fragments of old deleted files also exist.  As long as you did not add or delete files since the "format" was performed you will probably be fine.   The"un-reachable" items are probably those whose directory entries were erased by other file operations.

 

We'll know when you are through how it turns out.   This still beats re-ripping 8TB of movies.

[WeeboTech]

Please see the following support area regarding this situation.

http://lime-technology.com/forum/index.php?board=21.0

[jupilerman]

I recently add a new drive to my unRAID

I'm under 4.5.1 so I did not have the unformated bug.

This bug seems to be located only on the 4.5.3.

My suggestion : downgrade to the 4.5.1 (or less : 4.4.2 witch works perfectly) before adding a new drive.

 

A warning message should be written somewhere in the forum / download page / wiki.

 

Edit : there is a topic about the 4.5.3 bugs :

http://lime-technology.com/forum/index.php?board=21.0

[Cessquill]

Hi

 

Right. I think I'm nearly back to where I was.  I haven't had a chance to write to lime-tech today, but will do that.

 

In the meantime, here's a few of my notes that might be of use to people in a similar situation.  All of my DVDs are backed up as a straight file copy, and I use the My Movies system for cataloguing and playback...

 

- Follow Joe L's suggestions above exactly as written.  Disk 1 refers to the same disk 1 in your unRaid system (we're not concerned with parity here), and do it for all disks except the one that you've just installed (which should still be empty)

 

- Although you can do it over telnet, it is worth doing it on the actual unraid machine, just in case. If your unraid box is in a rack in a cupboard under the stairs, take 10 minutes out to set it up on a desk with a screen and keyboard. The fewer issues the better.

 

- There is likely to be some tidying up at the end.  If you are hiding the individual disks, for the purposes of this, it is helpful to display them. That way, if you end up tidying through Windows Explorer, you can get to them easily.

 

- Likewise, keep the spin down delay set to off until you've completely finished.  Otherwise, you'll end up spending lots of time sitting there waiting for disks to spin up as you're jumping around folders.

 

- While it is restoring, the screen does go off. A key press will bring it back on again to see progress.

 

- The restore of a 1TB drive can take about 5 hours. Multiple this by the amount of drives and arrange your time accordingly.  Some of my drives took 4 hours, some 5.  They are all Western Digital Green Power, but different variants

 

- As written elsewhere, umount is spelled without an "n" (not unmount)...!

 

- I occasionally got errors whilst restoring that all seemed to be about jpg files (however, I wasn't around to see all errors!).  Since these files were generated by My Movies, I was pretty confident that I could refresh them all.

 

- I occasionally got some CPU temperature issues, which I had never seen before.  It throttled the restore speed, but didn't have any other issues.

 

- Most folders restored back completely fine as if nothing had happened.  I also had a lost+found share in it with about 320 objects in (mostly folders, but some files)

 

- All objects in lost+found were named by numbers, and all root files did not have extensions (it was obvious from looking what types they were though, since 1gb files were VOB, etc.).  However, within the folders, all files were in tact.  That is, open a folder and you've got a DVD inside it.  Because My Movies puts meta data and cover artwork in these folders, it was easy to work out what that title was, and move it back into its proper home.

 

- I found that restoring folders from lost+found back to their proper shares (in Movies, or TV for example) was easiest in Midnight Commander (type mc from the console).  That way, you can do it a disk at a time and moving is instantaneous.

 

- I also noticed that the numbering structure of the lost+found corresponded to where it was originally.  That is, the lost+found folders start with a number, then an underscore, then a (seemingly) random number.  The first number corresponded to its original share.  Therefore, all folders starting with a "2_" were moved back into the Movies folder, for example

 

- Once all folders were back in their proper shares, I moved back to the (more familiar) windows interface to rename them.  Fortunately, My Movies places an XML file into a movie folder.  The filename has the same name as its folder, so it's just a case of renaming each folder accordingly.

 

- A couple were not quite so straight forward, having "disk 1" as their title, but you can then refer to the cover jpg to jiggle things about

 

- How do I know that I've restored everything?  Is it all there?  I then moved on to the My Movies interface, which helped me with some of these tasks.

 

- Under Collection, select Optimise Database.  This will highlight any titles that were in your collection that it can no longer find.  I had a few.  Some of these were because I'd made a typo in the folder name, but a whole chunk from my "TV" and "Standup" shares were missing completely.  By this time, my lost+found only had 9.25gb of files in it, and my array was reporting no extra file space free.  Therefore, they had to be there somewhere. Go through each disk and check the casing of the folder names of each share.  On a couple of my disks, there was a "tv" folder.  On others, it was "TV".  Make them all the same, and your files should return back into the file system (use the casing of the files that are all ready available as the master here, and make everything else match that).  Keep running Optimise Database until all titles are there.

 

- Now to flush out the issues I was having with JPG files.  In My Movies, go to Options, Meta Data Storage.  I tried running the "Update" command, but it kept hanging on specific titles.  To flush this out, use the "Clean" command and completely remove all meta data that My Movies has created.  This may take some time, but worth it.  Then, you should be able to run the "Update" command, and all meta data will be completely refreshed.

 

- How do I know that the movies will play?  Well, here's where I'm not so sure.  I all ready know that at this point I've got one title that's pretty much fubar'd, but it's a Disney film that I can put back on whenever.  No big loss.  That was the only one that stuck out in my lost+found folder.  What I have ended up doing was installing DVD Shrink and using its "Open Files" command, pointing it in turn to every title in my collection.  This runs a 30ish section analysis of the title that hopefully highlights titles that are not good.  However, I'm pretty much guessing here.  Performing this has highlighted about 20 discs that it could not perform a full analysis on, which isn't bad over a 1500 disc system.  It did take a while, but setting 3 machines to the task and doing it in the background made fairly light work of it.

 

That said, I have had analysis problems before but the title has played fine.  Also, having never performed this analysis across my disks before, the errors could have all ready been there.  Also, DVD Shrink is a little old and unsupported now - I don't know whether there's anything on any discs that it is just not familiar with.

 

Either way, it's no big deal to copy up 20 or so in my spare time.  Saturday saw me pick out four titles at random and play them flawlessly - at least one of which I know to have been in the lost+found folder.

 

- So what do I do with the lost+found folder? Well, I'm now left with 9.25gb of files (plus a few empty folders) in lost+found.  All have numbers as file names, with no extensions.  Some could be XML files, some could be JPG (judging by their respective sizes) - I know that these would now be redundant, since I've regenerated all meta data.  Some of the disks I've backed up also include the DVD-ROM features (that I will never use), so some of the files could be from there.  The rest look like VOB files.  But to which title?  Some of them may be files that I had all ready deleted that it was able to recover (maybe, not sure). Bearing in mind that I have about 20 titles to copy back over, trying to locate and rehome these files looks to be a superfluous task.  I'll probably back them up to a USB drive and hang on to them for a while, just in case.

 

- Will the files I have play?  I hope so.  Well, I have run them through a basic analysis in DVD Shrink.  I could always run its deeper analysis.  I know I did have a corrupt movie on there all ready (before the calamity), but it passed DVD Shrinks test.  I'm holding on to the hope that if a file not been restored, or is badly damaged, DVD Shrink will have picked it up; otherwise it will be fine.  I might have to let some titles play through to the end before movie nights...

 

-----------------------

 

A huge thank you to you guys who have helped me out on here - if you're ever in the UK (or I'm ever where you are), I owe you a night out for saving me potentially months of work.

 

I'm off to post my experiences to lime-tech.  I appreciate that there is now an area on the forum acknowledging this, but if I'm honest, I wouldn't have checked here before adding a disk to the array.

 

Again, thank you.

[boof]

My suggestion : downgrade to the 4.5.1 (or less : 4.4.2 witch works perfectly) before adding a new drive.

 

The problem with this is that one of the main reasons for 4.5.3 existing is to provide support for the supermicro 8 port SAS -> SATA card. This is / will be a very popular card so if you want to add a drive to that card..you have to use 4.5.3 full stop

 

Otherwise yes, I would agree.

 

OP - glad you got (most) of your data back.

[Cessquill]

The problem with this is that one of the main reasons for 4.5.3 existing is to provide support for the supermicro 8 port SAS -> SATA card. This is / will be a very popular card so if you want to add a drive to that card..you have to use 4.5.3 full stop

 

Otherwise yes, I would agree.

 

OP - glad you got (most) of your data back.

Thank you. And you might have answered a question for me there.

 

I have a 10 drive case, with 8 SATA ports on the mobo, and another 8 on a Supermicro card. Since the case is now full, I am either looking to either expand externally, or replace the case. I've seen a 20 drive case (plus 2 2.5" mounts) that has 5 SAS backplanes.

 

Can I assume that one of those SAS cards could drive two of the backplanes, with the other three coming off SATA->SS cables? If so, that would be ideal, and would leave me enough SATA ports to add an SSD cache drive.

[betaman]

If you wanted to check the contents of the larger VOB files still in the lost+found, couldn't you just copy them to your PC and use a media player like VLC to play them?

[Joe L.]

- So what do I do with the lost+found folder? Well, I'm now left with 9.25gb of files (plus a few empty folders) in lost+found.  All have numbers as file names, with no extensions.  Some could be XML files, some could be JPG (judging by their respective sizes) - I know that these would now be redundant, since I've regenerated all meta data.  Some of the disks I've backed up also include the DVD-ROM features (that I will never use), so some of the files could be from there.  The rest look like VOB files.  But to which title?  Some of them may be files that I had all ready deleted that it was able to recover (maybe, not sure). Bearing in mind that I have about 20 titles to copy back over, trying to locate and rehome these files looks to be a superfluous task.  I'll probably back them up to a USB drive and hang on to them for a while, just in case.

 

You might be able to learn quickly of their types by using the "file" command

 

download from here:

http://slackware.cs.utah.edu/pub/slackware/slackware-12.2/slackware/a/file-4.21-i486-1.tgz

install by typing

installpkg file-4.21-i486-1.tgz

 

 

invoke by typing

cd /mnt/disk1/lost+found

file *

 

It will look like this if it is a collection of VOB files:

 

root@Tower:/mnt/disk3/data/VIDEO_TS# ls

VIDEO_TS.BUP*  VIDEO_TS.IFO*  VIDEO_TS.VOB*  VTS_01_0.BUP*  VTS_01_0.IFO*  VTS_01_1.VOB*

 

root@Tower:/mnt/disk3/data/VIDEO_TS# file *

VIDEO_TS.BUP: Video manager, v11

VIDEO_TS.IFO: Video manager, v11

VIDEO_TS.VOB: MPEG sequence, v2, program multiplex

VTS_01_0.BUP: Video title set, v11

VTS_01_0.IFO: Video title set, v11

VTS_01_1.VOB: MPEG sequence, v2, program multiplex

[Cessquill]

Ahh, thanks for that.

 

To be honest, after running DVD Shrink across all titles, I've re-backed-up any that had missing files and all seems fine.

 

Some of the smaller files could have been MyMovies meta-data, IFO files, BUP files, and maybe even DVD-ROM extras (since it was a complete file copy).

 

I figured I could identify the VOB files, play them in something like VLC and then attribute them to the correct movie (hopefully!), since I'd all ready decided to run an analysis over all titles, I can save myself the hassle and redo the 30 or so discs.

 

Therefore, I'm hoping that I shouldn't really need the lost+found folder; and if I did, I'm not sure how confident I'd be that the files in there were OK.