yup, you hit the nail on the head, trurl...that was *exactly* what i was looking for...i did exactly that, and now the array is happily rebuilding Disk 13...it'll take another cool 10hrs, but it is doing it! Hallelujah! i'll say Eureka when i don't see the lost+found folder anymore, but that may be asking for too much...hope always dies last with me, though.

 

thanks trurl, and thanks garycase for getting me to this point...and everyone else here! i'll report back tomorrow with what i ended up with.

Excellent -- glad the rebuild is underway.

 

Hopefully the rebuilt disk will be a "better #13" => that simply depends on when the corruption happened vs. the last updates of parity.    <Fingers crossed for you>

 

... by the way -- I thought the steps trurl suggested must not have worked; that's why I had looked for the notes Tom had suggested as an alternative.  I thought that because I had suggested exactly the same set of steps on 26 Jan 

 

What MAY work in that situation is to do a New Config with Trust Parity (including ALL of the drives);  then unassigning drive #13 and restarting the array -- so it shows a "missing" drive;  then stopping the array, re-assigning drive #13; and then starting the array again and letting it rebuild #13 onto itself.

well, damn me for not having tried them 2 weeks ago, as you clearly suggested the very same thing then! not that i could have then, bc i still had to replace the drive cages in-between, but had i paid more attention to that post of yours, i wouldn't have had to ask all the wrong questions for the past week, as i would have surely preferred this over going to the command prompt!

 

they do seem to work, although - so far - i see exactly the same lost+found data on it as was there before the rebuild...unless anything can be expected to change later on in the rebuild, which i seriously doubt.

 

with that said, this would indicate that the file-corruption happened *before* the last parity sync...however, just to get a better idea of what those files are, i took some samples from the largest of those cryptic folders and found them to be a few thousand of my timelapse JPGs, which i also found in their proper locations on the intact data of my unRAID system...so it could very well be that i synced whatever data was lost at an earlier time back to the array since then...i've gone through so much data management these past 3 years that i simply can't cache all of what i did in that brain of mine...right now it seems like those 12GB of data in lost+found contain little that i don't have somewhere else, and even less of what is truly important (the other fragments appear to be from old software backups, which are too old to be of much use to me anyway).

 

is there any damage done in stopping the rebuild if i have already decided that i'll simply delete that lost+found folder and do a Parity Sync? or is there anything "magical" that can happen in the 8 or so hours it still has to do, that would warrant me waiting for it to finish? frankly, i'd rather have a rebuilt parity tomorrow than a drive 13 that is exactly what it is right now.

Easy to understand how you missed my outline while you were in the process of ordering/waiting/replacing the cages.

 

r.e. the rebuild:  I would not stop it -- I don't know for sure what the consequences might be, but I wouldn't take the chance.    I would expect that it would simply be fine; but just don't know for sure.

 

Just be patient 

 

Once the rebuild finishes, you can do a parity check (be sure it's a correcting check) ... and when it's done you'll know for sure your parity is okay [Hopefully it is already -- as the rebuild is depending on it !! ... but as you've got backups of all the current data for drive 13, it's not catastrophic if it's not]

 

It's too late now, but while you had the array set with the drive "missing", you could have browsed the drive (it would have been emulated by the other drives + parity) and you would have seen exactly what it was going to look like after the rebuild.

 

 

Yeah, alright...I'll let'er finish this rebuild then and start tomorrow with a Parity Sync...when you say "corrective", is that an option when I start the sync?

You don't actually need to do a "sync" -- just do a normal parity check (the result is the same as if you did a new sync).

 

... Note the little box that says "Correct any Parity-Check errors by writing the Parity disk with corrected parity"

By default, this is checked (that's what you want) => but some folks like to do non-correcting checks -- so just in case you were in that camp I wanted to remind you to be sure you did a correcting check.

 

Got it.

 

Well, looks like tomorrow will be the day to initiate the final countdown, and come Tuesday I should be able to end this inflated thread with the word SOLVED, inshallah.

Yes, it looks like the saga has effectively ended ... and with what I'd assume you'd consider pretty good results => i.e. no major data loss.

 

The only other thing you COULD do is run the Reiserfsck utility against disk #13 (after the rebuild) just to see if it's able to recover anything else.    But if you're fairly confident there's no important missing data, I'd be more inclined to just "clean up" the disk [delete the stuff you don't need on it] and move forward.

 

 

yes indeed.

 

although i will not put myself through a reiserfsck after all this...my timelapse frames are safe and saved and the rest is really just some dead weight that i should have deleted already, but you know how it is with spending so many hours and days tracking down data to delete, with storage prices being as cheap as they are these days, and time being so valuable...because time isn't money, as is said so often today, but time is Life, as i can attest to after surviving the 2% odds i was given after my 2009 Glio IV diagnosis...so in the big picture, all is great, even if some data was lost

 

Wow -- surviving a stage 4 brain tumor is so far beyond the problems you had with this server it's not even in the same universe !!

 

Congrats on kicking the tumor -- as you noted, the survival rate for these isn't very encouraging, but you've clearly beaten the odds.

 

Sure makes a corrupted disk in your UnRAID server seem like a trivial issue.

 

yeah, it sure does pale in comparison...all this data management is a form of madness, anyway...to attach so much importance to a bunch of zeros and ones...but i can't stop it either at this point, although i'd love to find ways to be less dependent on all this technology.

 

but, if nothing else, it exposes me to the kindness of strangers, such as you and all the other helpful souls here and some of the other forums on the interwebs...since the old BBS (Compuserver) days i have always found Forums to be the most valuable aspect, the staple of the Internet (with Wikipedia and a few other collections of intellectual assets).

 

so, thank you, graycase and everyone else for being here and being kind!

ooops, one more question (well, i hope it's the last one):

 

the "lost+found" folder appears to have some weird permissions and after backing it up, i don't seem to be able to delete it from drive 13...what would be the proper terminal syntax to force delete this folder?

I'm decidedly NOT a "Linux guy" ... but according to a few notes I've kept of useful commands, this will delete the folder for you:

 

rm -rf <dirname>

 

... where <dirname> is the name of the folder you want to delete.

This does have to be run with root privileges ... but that's the standard login to UnRAID, so it should work.

 

If that doesn't do the trick, post back and we'll let a "Linux person" weigh in 

yeah, i've tried -r and -rf, but keep getting "Read-only file system" errors...tried a chmod 777 (same error)...obviously i have got some sort of user permissions problem on this folder, and am far from comfortable in the command prompt, so i guess i'll have to wait for a "Linux guy/gal" to chime in.

When a lost+found folder is created then it and all the files it contains will be set with 'root' as the owner.  If you want to be able to delete these via a network share then you need to run a command of the form:

newperms path_to_lost+found

to fix this.

 

However since you indicate that you are getting told that the file system is read-only that strongly suggests that there is file system corruption on the disk in question and that will prevent you modifying the disk.  Assuming the disk is in reiserfs format then the way to recover is to run the reiserfsck utility against the disk in question to fix the corruption.

looks like you're right, itimpi...the newperms command also gives me the "Read-only file system" error.

 

what's the correct syntax to do a reiserfsck, and are there parameters i'll need to add?

Looks like you need to run the trusty old reiserfsck after all ... in which case you may as well see if it can fix anything and/or recover some additional files with the --rebuild-tree option.

 

But first run the "check" option and see what it recommends.

 

Details here:

http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

 

i was just reading through that wiki entry...will have to do some more reading on this, and then do my best attempting that.

Remember that if you open a Telnet window you have to keep it open for the entire command -- otherwise the process will be shut down.    This isn't the case if you use Screen.    I always do command-line options directly from the UnRAID console, but I don't believe you have a connected console, so be sure you are cognizant of these constraints.

 

I've never had to use reiserfsck; but many folks on the forum have had stellar results with it -- it's a VERY reliable utility that can sometimes do near magic with its recovery.    If you're lucky (as I've said before) it may even recover some files you didn't expect ... although at this stage it seems you're pretty happy with what you already have.  I suspect you just need to run the --check option and then do what it says ... likely the --fix-fixable choice.  (but possibly the --rebuild-tree).

 

according to instructions, the first step is to put the array into maintenance mode, but even though i have done all i can to eject all mounted unRAID disks from my Mac, the Web GUI shows that it cannot unmount a disk share, and i don't seem to be able to find a way to find what could possibly still be mounted...it's always somethin'.

ah, it was the telnet session that dis-allowed it...i had to exit from that session and now it stopped the array.

i think i'm on my way now with the first "check" step:

 

Connected to 172.25.150.120.

Escape character is '^]'.

Password:

Login incorrect

 

unRAID login: root

Password:

Linux 3.9.11p-unRAID.

root@unRAID:~# reiserfsck --check /dev/md13

reiserfsck 3.6.21 (2009 www.namesys.com)

 

*************************************************************

** If you are using the latest reiserfsprogs and  it fails **

** please  email bug reports to [email protected], **

** providing  as  much  information  as  possible --  your **

** hardware,  kernel,  patches,  settings,  all reiserfsck **

** messages  (including version),  the reiserfsck logfile, **

** check  the  syslog file  for  any  related information. **

** If you would like advice on using this program, support **

** is available  for $25 at  www.namesys.com/support.html. **

*************************************************************

 

Will read-only check consistency of the filesystem on /dev/md13

Will put log info to 'stdout'

 

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

###########

reiserfsck --check started at Mon Feb  9 00:40:44 2015

###########

Replaying journal: Trans replayed: mountid 4294967145, transid 4294936299, desc 1391, len 1, commit 1393, next trans offset 1376

Trans replayed: mountid 4294967145, transid 4294936300, desc 1394, len 1, commit 1396, next trans offset 1379

Replaying journal: Done.                                                       

Reiserfs journal '/dev/md13' in blocks [18..8211]: 2 transactions replayed

being that i backed up drive 13 already, i am doing a rebuild-tree now after i got this:

 

Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs.

Bad nodes were found, Semantic pass skipped

1 found corruptions can be fixed only when running with --rebuild-tree

###########

reiserfsck finished at Mon Feb  9 01:25:15 2015

###########

 

being that i backed up drive 13 already, i am doing a rebuild-tree now after i got this:

 

Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs.

Bad nodes were found, Semantic pass skipped

1 found corruptions can be fixed only when running with --rebuild-tree

###########

reiserfsck finished at Mon Feb  9 01:25:15 2015

###########

Yes - that is the way to go.

 

At the end the file system should be good.  There is very likely to be a lost_found folder present at the end that will need the 'newperms' command running against it so that you can manage its contents via a network share.