I suspect you won't ship this system with the drives in the cages again 

Back when heatsinks were HUGE on normal systems, I shipped a customer in Alaska a custom built PC and the heatsink broke the plastic mounts, came loose and made mincemeat out of everything inside the case. Big chunk of metal with sharp fins bouncing around completely totalled that PC.

well, yeah, i mean, i should have known better, and kinda did know better but gambled (again) and lost...now it's time to pay up and learn my lesson.

That can still happen with systems that use some of the higher-end 3rd party heatsinks or large liquid-cooling radiators.    These need to be "filled" with those little air-filled packing balloons -- that will generally ensure there's no breakage ... and in the event a heatsink DOES break off will mitigate any residual damage.

 

But with servers, the biggest risk is shipping them with the disks installed.  The disks add a LOT of weight; and if installed in hot-swap cages can easily break either the connectors or the circuit boards (as we've seen in this case).  An empty cage will easily survive shipment, since there's virtually no pressure on it.

 

I always ship the drives in WiebeTech DriveBoxes [ http://www.amazon.com/DriveBox-3851-0000-11-Hard-Disk-Case/dp/B004UALLPE ], but simply wrapping them well in bubble wrap and boxing them securely works fine as well.

 

ok, i'm finally back.

 

the other two new cages came in so late yesterday that i refrained from getting started with all of this then, but came in a couple of hours ago to start the tedious task of transferring all the remaining drives and re-cabling the two new cases...but it sure felt good when i booted up the machine, found them to be surprisingly quiet (some of the reviews i had read were complaining about the noise of these cages (they are also available on Amazon.com, but cost 20% more there then i paid here, curiously...no reviews on the US site, though: http://www.amazon.com/BackPlane-5x3-5-SATA-in-3x5-25/dp/B006BQYSFA/ref=sr_1_1?ie=UTF8&qid=1422966579&sr=8-1&keywords=IB-545SSK).

 

when i looked the Web GUI, i found all drives to be green...i un-assign the parity drive and started the array to see what its condition is and found it to start into a work-able status (see the attached screen-cap).

 

*however*, as you can see, Disk 13 has a high number of Reads and Writes, probably left from the failed attempt to rebuild it from parity, and looking at the contents of my system through Finder, i can see that Drive 13 is the only one that has a "lost+found" folder on it that contains 63 cryptic folders with about 12GB of dis-jointed data-fragments in it...not a huge amount, in the big picture, but i guess any lost data is not a good thing, considering i don't really know what most of it is.

 

so my questions now are this:

 

- assuming that the Parity drive is still trust-worthy, Drive 13 should be rebuildable from Parity, right?

 

- what is the proper order of actions now to do this?

 

i am not going to touch this system until i get your guidance, garycase, as history shows that my actions are prone to be mis-guided when it comes to recovering from a situation such as this, while yours...well, you just know your sh*t

 

awaiting your commands, master!

Well ... things are looking pretty good except for #13.

 

If we ASSUME that your parity drive is good, you should be able to do a New Config with the "Trust Parity" option and rebuild disk 13 onto a new disk.

 

HOWEVER ... whether or not that will help depends on the state of parity => if the change you're seeing in the data on #13 happened with parity enabled, the parity disk may have been updated to include those.    If that's the case, the rebuilt disk will have the same issues.

 

Regardless, the current disk #13 won't be impacted by doing a rebuild (since you'll do the rebuild to another disk) ... so that's what I'd do next.    If you're lucky, the rebuilt disk will have all your data intact;  if not, you can then proceed to attempt to recover what you can from #13 (or from the rebuilt #13, which should be just as good).

 

Be sure you follow the correct steps to initiate a rebuild after a New Config -- anything done incorrectly can cause even more problems.  Frank provided a link to Tom's instructions for this a while back ... I'll repeat it here so you don't have to search for it:  http://lime-technology.com/forum/index.php?topic=32491.msg298738#msg298738

 

[Obviously the disk # you want to use has to be correct -- 13 if you assign all the disks to the same slots they're in now when you do the New Config]

 

 

damn, i had hoped to be able to use the Disk 13 that i already have in there...the cages already tapped me dry, as i was not expecting to have to pay over $350 just to get started...what would be the steps i'd have to take to pass off the old Disk 13 as a new Disk 13 to rebuild to? or would that be a horrible thing to attempt? another 3TB disk would set me back yet another $150, when i haven't even paid for the $350 for the cages yet...if the worst that can happen is that i'll end up with the same lost+found folder on the rebuilt Disk 13, then so be it...by the look of it, most of what i see in those cryptic folders are timelapse JPGs and software packages that i have in a different location, so i am not as worried about that as i am about spending yet another $150...this move back to Berlin has not been helping my financial situation (so far), as most moves have a way of draining ones money and energy.

 

so if i had to guess, i'd have to somehow remove Disk 13 first and format it, then do a New Config with the "Trust Parity" option, then re-insert Disk 13 and let it do a rebuild?

 

i'm too tired tonight to still even think about it, so i'll re-read the last few posts from you, garycase, and Frank to make sure i'll be doing the right thing.

sorry if i get annoying with repeating the same questions or not following your advice properly...it's been a rough few days with this and the real work i've had to do.

damn, i had hoped to be able to use the Disk 13 that i already have in there...the cages already tapped me dry, as i was not expecting to have to pay over $350 just to get started...what would be the steps i'd have to take to pass off the old Disk 13 as a new Disk 13 to rebuild to? or would that be a horrible thing to attempt? another 3TB disk would set me back yet another $150, when i haven't even paid for the $350 for the cages yet...if the worst that can happen is that i'll end up with the same lost+found folder on the rebuilt Disk 13, then so be it...by the look of it, most of what i see in those cryptic folders are timelapse JPGs and software packages that i have in a different location, so i am not as worried about that as i am about spending yet another $150...this move back to Berlin has not been helping my financial situation (so far), as most moves have a way of draining ones money and energy.

 

so if i had to guess, i'd have to somehow remove Disk 13 first and format it, then do a New Config with the "Trust Parity" option, then re-insert Disk 13 and let it do a rebuild?

 

i'm too tired tonight to still even think about it, so i'll re-read the last few posts from you, garycase, and Frank to make sure i'll be doing the right thing.

sorry if i get annoying with repeating the same questions or not following your advice properly...it's been a rough few days with this and the real work i've had to do.

Formatting is never part of the rebuild process. All formatting does is create an empty file system. Rebuild will build the not-empty filesystem from parity and the other drives. So formatting a drive before rebuilding on it is pointless and often gets people in trouble. In your case that would not probably get you into any more trouble than you're already in, since you will be doing New Config and then forcing unRAID to do a rebuild.

 

I think if you follow the instructions you should be able to use the old drive 13 instead of a new one. Keeping the old one aside is just for the purpose of having something to fall back on if the rebuild fails for some reason.

 

I've never done the "non-standard" procedure they have proposed, so I will let them comment further. But you have no way to rebuild a different disk if there is a problem with one of them until you get your array back to parity-protected.

I've also never done the rebuild-after-New-Config procedure, but it clearly works fine.

 

If you're okay with simply losing whatever's on disk 13 now, then it's fine to use it for the rebuild.  In fact, you don't have to do ANYTHING to the disk to do that -- just follow the process outlined and it will rebuild onto that disk.    The only potential issue is if your parity isn't really good, the "new" 13 may not be what you expect/hope for.  Rebuilding to a new disk simply keeps open the option of doing a recovery from the old disk.

 

 

Another alternative:  Do any recovery attempts BEFORE you do the New Config and rebuild.    Simply create a "one drive" UnRAID with ONLY disk 13 assigned; and run Reiserfsck on the drive and see if it recovers all (or most) of your data.    Clearly if you do this, be CERTAIN you don't do ANYTHING to any of the other disks in the process.    I would also copy any important data that IS already accessible on disk 13 to another location, assuming you have space for it (perhaps on your Mac or an external drive).

 

Once you've done the Reiserfsck, if it happens to be successful and you're happy with the contents of disk 13 at that point, you won't need to do the rebuild at all -- just do a New Config and you're ready to go.    If not, then just do the rebuild -- and hopefully it will in fact recover you back to an earlier (and better) state for that drive.

 

 

wise words, as always, garycase.

 

for starters i am now doing a full backup of Disk 13 to an external 4TB Firewire 800 LaCie drive that has enough free space to house the 1.7TB that are on Disk 13...unfortunately a fairly slow process, so that it'll be another day or two until i can proceed...maybe by then i will have made my mind up of whether i'll have the nerve to go the extra mile and attempt the single-drive unRAID config with a Reiserfsck.

 

if i do, is there a way to backup my current config on my current USB boot-stick and create the minimal config on it? i am most un-nerved by the prospect of dealing with building up a whole other USB boot-stick...so if there is a method that isn't all that difficult for me to do, then i may just attempt to do a Reiserfsck.

 

but would a Reiserfsck somehow put back the contents of "lost+found" where they belong? i have a hard time believing that, so what could i hope for or expect from doing this extra step?

Sure, you can use your current USB stick.  Since you're going to do a "New Config" anyway, simply do a "New Config" and only assign Disk 13 to it => then you have your one-disk UnRAID.

 

It seems very unlikely that a Reiserfsck is going to alter the "Lost and Found" directories you already have.    I'm not by any means a "Linux guy", so I really can't say ... but I don't think it will modify the existing folders and files ... it just may find more and/or fix corrupt file structure info.    I do know folks have had some amazing success stories with this process ... data I would have thought for sure was long gone has been recovered.    I suspect the "Lost and Found" folders were created at some point by some recovery attempt ... but real question is whether this was after you shipped the system (in which case hopefully your parity disk still retains the data from BEFORE the shipment).    If not, the parity info has likely been updated to include these folders -- in which case the rebuild will simply have the same data.

 

In any event, I'd take the time to do this in 3 stages:

 

(1)  Backup the data you can already read on the disk (which you're doing);

(2)  Do the Reiserfsck runs and see if this changes things for the better -- if it seems to recover additional data, copy that to your backup disk before doing #3;

and finally

(3)  Do the New Config with Trust Parity and follow the directions to force a rebuild of disk #13. 

 

After that, you need to decide which has the best data to keep -- your backup disk or the rebuilt #13.

 

the Backup Synch operation i am doing is estimating another 1.5 to 2.5 days to complete, so this will continue to be an exercise in patience.

ok, with only 40mins (which could easily change to 4hrs, the way this software's prediction algo has been going) to go until Drive 13 is backed up to a safe location, i do have yet another question which may have been answered already, maybe more than once, so please don't get too annoyed with me, but here it is:

 

> since i now backed up the data that was intact *and* the countless file-fragments and disjointed files, and since you, garycase, said that it is highly unlikely that Reiserfsck would put those files back where they belong, would the only reason to do the single-drive unRAID with Drive 13 be to maybe find even more of these file fragments? because, frankly, save for the 2-3k JPG files that were found already (and which i do have backed up elsewhere), the rest is a total loss...there is simply no way that i'd spend the dozens, if not hundred+ hours to figure out what they were a part of...so with that said, would the next logical step be to do the New Config *with* the "Trust Parity" option, and to then ask Disk 13 to be rebuilt? and in order to do that, i would have to follow the method that Frank (and then you) linked to (using the "invalidslot" command etk29321 was instructed to use by Tom), right?

 

i know i must sound like a broken record by now, but it is this step that everything has been building up to (pun not really intended), so i am a bit nervous.

things is, isn't there a way to get Disk 13 to be rebuilt from Parity without having to use the command line? maybe i'm not the only one that gets a little weak in the knees every time i'm at the command prompt.

 

(backup operation now claims 3hrs 40mins to completion ... ~ sigh ~ ...i may just wait until tomorrow and a night's worth of sleep...5hrs 30mins...pfff)

hmmm...i hope i didn't make another mistake, but here is what i did since my last post:

 

> waited for Disk 13 to be backed up to an external disk

 

> did a "New Config" and re-assigned all disks as they had been

 

> (here is where i got very apprehensive) the instructions for rebuilding a disk say to first type the command - in my case "mdcmd set invalidslot 13" - then  click the "Start" button in the Web GUI...*however*, since the default option in the Web GUI is to Start the array and immediately do a Parity Sync, and the option to trust parity is not enabled, i was worried that if i followed the instructions i wouldn't get a chance to keep parity as it is and rebuild disk 13 from the *old* parity, but rather have the "lost+found" data from disk 13 written to the parity...so in order to tell the system to trust parity i *first* started the array with the "parity is valid" option and *then* tried to execute the "mdcmd set invalidslot 13"...i wasn't too surprised when nothing seemed to happen, so after waiting 10-15mins i stopped the array again, then tried to execute the command again...but again, nothing seemed to be happening.

 

now, i still haven't touched Parity...it says "Parity has not been checked yet" in the Array Operations tab, and for now i un-checked "Write corrections to parity disk" next to the "Check" button, but i am not very confident at all that i did this correctly or that i haven't closed a door here  that would have allowed me to do the re-construction of disk 13 to what is on the old parity drive...but i do have hope that it may still be possible...it's just that i know so little about the command line that i do need your help to get this invalidslot command to work for me...if it isn't too late.

 

i'll leave it as is for the night...i'm hitting the sack, but hope to continue this in the morning, if i get the instructions i need...again, sorry if my inexperience got me on the wrong track again.

Okay, you've backed up what you can already read from #13;  you don't want to take the time (potentially a day or two) to do Resierfsck and see what it might be able to do (I understand that -- and quite candidly I think it's relatively unlikely to find anything that you couldn't get later by doing Reiserfsck on the rebuilt disk anyway).

 

... but you DO want to rebuild disk #13.

 

Hopefully you haven't lost that option with your latest actions -- as long as the system did NOT start a parity sync and/or a correcting parity check then you should be okay.  You did NOT want to Start the array until you had done the correct command to force the rebuild.

 

When you did the New Config, did you check the "Parity is valid" box (the so-called "Trust Parity" option)?    As long as you did that AND were careful to assign all of your disks correctly, then you simply have to run the "mdcmd set invalidslot 13" command FROM A COMMAND PROMPT.    If your server has a console and keyboard, do it there (that's the best way);  if not, login via Telnet to do it.  THEN you Start the array, and it should show disk 13 reconstruction in progress.

 

 

 

Note the IMPORTANT NOTE at the bottom of the instructions that Tom had provided:

 

> 1. Execute Utilities/New Config.

>

> 2. on Main, assign all your drives, being very careful to assign Parity and your new disk (disk10) correctly.

>

> 3. From a console or a telnet session type this command:

>

> mdcmd  set  invalidslot  10

>

> (the 10 corresponds to disk10)

>

> 4. Click 'Start' on the webGui.

>

> What should happen now is array gets started with disk10 reconstruct in process.

>

> IMPORTANT: between steps 3 and 4 do NOT refresh your browser or navigate to any other pages in the webGui - just click the Start button which is already being displayed there (if you navigate to a different page or even refresh the browser after step 3 it will "cancel" the effect of that 'mdcmd' and result in your parity disk getting written - not good).

i do not have a keyboard and monitor on my unRAID yet, so i executed invalidslot from a command prompt that i used to telnet into the server.

 

and like i said, the "Trust Parity" option, as you call it (it said "parity is valid" next to an empty check-mark box in the Array Operation tab) was grayed out, and since the instructions seemed to assume a proper protected state with valid Parity, i was concerned that if i didn't have that first, it would start rebuilding parity first.

 

be that as it may, the fact that "Parity has not been checked yet" should mean that i still have the same Parity information that i did before i did the New Config, no?

 

*if* that is the case, and since the invalidslot command doesn't seem to do anything (at least i see zero feedback in the telnet session of the Web GUI), how could i "fake" a state of Drive 13 that would trigger unRAID to want to rebuild it? i know that, in theory, the invalidslot command should be the most direct way to do this, but if it isn't triggering anything in my particular situation, what's the next best method? shouldn't it be possible to just "clear" or "format" the drive outside of the array and then have it be rebuilt as a replacement disk for the old Disk 13?

 

i have no idea why that command didn't seem to do anything, and i did not navigate away from the Array Operation page since last night, when i executed that command...i did read those instructions very carefully and did notice that "IMPORTANT:" addendum...trust me, i've been extremely careful with every step, but i am aware that due to my lack of experience, there is always a chance that i mis-understand certain steps...and the one last night, where i didn't want the array to start before Parity was trusted, may have been a wrong step.

I've never done this with Dynamix ... so I tried it tonight, and it does NOT seem to behave in accordance with the guidelines Tom gave.

 

I tried several different approaches to force a rebuild, but simply wasn't successful.  I did the following, which is about as close to Tom's suggested steps as you can do in Dynamix:

 

(1)  Went to Tools - New Config; checked "Yes I want to do this" and clicked on Apply

(2)  Went to Main and assigned my disks (including Parity)

(3)  Went to Array Operations and checked the "Parity is already valid" box

(4)  On the UnRAID console (you could Telnet since you don't have one) I typed the mdcmd line to set the slot invalid

(5)  I Started the array.

The result was an immediate parity check (which I immediately aborted).

 

I tried that several times with slight variations ... but was never able to force a rebuild.

 

The simple fact is I don't know what variations to Tom's instructions are needed to do this through Dynamix.  I'd be inclined to simply reload your flash drive with the basic v5 (don't forget to keep your key file), so you are using the original LimeTech Web GUI ... which I assume is what Tom's notes are based on.    That should let you force the rebuild.

 

On the other hand, if, when you Started the array last night it did the instant parity check I noted (and you didn't Stop it very quickly), it may very well be too late to do the rebuild anyway.

 

It's VERY late and I definitely need to hit the sack.  I'll look at this some more tomorrow and see if I can give you a better option.  The #1 thing if you still want to try and recover data is DO NOT do ANY writes to the array.  If any array disk or the parity disk is modified, then you'll lose the ability to do a rebuild.

 

What you COULD do is try the Reiserfsck on drive #13 => there's always a chance that will recover more of your data ... and it clearly won't hurt to "mess" with #13, since you've already copied everything you can off of it; and you're going to simply rebuild it (if we can get that to work).

 

 

 

 

 

 

 

 

 

i feel a little better that you also found the instructions to not lead to the intended effect...to the best of my knowledge, i never allowed the parity to start doing a check, which is why i took the extra precaution of *first* starting the array with the "trust parity" option enabled, before doing anything else...so i *think* that i am still in a position from which i'd be able to do a full rebuild of Disk 13.

 

but i'd rather wait until you've had some well-deserved sleep and maybe another insight before i do anything else...i'm just going to stop the array in the meantime.

have a good night, garycase!

 

PS:

 

just to clarify one more time: it did *not* start a Parity Check last night when i did what i did!

since you also didn't succeed under the Dynamix Web GUI, using those instructions, and suggested trying it again under the default Web GUI, i reverted back to a stock system and this is where i am right now (see attached).

 

before i telnet into unRAID  and execute the invalidslot command, do i need to worry that it's going to do a parity sync with what i have right now? is there another method for marking Drive 13 as a "new" replacement for the "old" Drive 13, thereby tricking unRAID into starting a rebuild? i am not at the location of the server this weekend, but wouldn't formatting Disk 13 outside of the system make it look like a replacement? or maybe there is a wy to do it from the command prompt within the unRAID system?

 

i feel like i'm so close now, but i am reluctant to do anything else without being sure that i won't give up my chances for a rebuild (if i haven't already).

I am fairly sure that with the setup you have right now this would work fine.

 

I'd follow Tom's steps exactly -- i.e. do the invalidslot command via telnet; do NOT do any refreshing of the GUI; then simply Start the array.    I think it will give you a note r.e. that it's going to do the rebuild (which you'll have to confirm);  but it may also simply start that.

 

Watch VERY carefully when you Start the array -- and when the display refreshes after the Start command, if it happens to show a parity sync in progress IMMEDIATELY stop it.    But I don't think that's what will happen -- I think instead you'll see a rebuild in process.

 

Note that no matter what, if you don't rebuild parity, there's GOT to be a way to get this rebuild to work.    If this attempt doesn't work, I'll see if I can get Tom to review this thread and weight in with any suggestions.

 

... by the way, I'm going to be very scarce for the next couple days, but I'll try and check a couple times a day to see if you have any progress/further questions.

 

unfortunately, the behavior under the stock Web GUI is just like under the Dynamix Web GUI: nothing happens.

 

when i hit the "Start" button, the array is started and in an all-green state...if there is any rebuilding going on (which i strongly doubt), there is no feedback given in the Web GUI or the terminal window.

 

Parity still shows as "valid", but the system knows that "Parity has not been checked yet.", suggesting a Parity check, which i will *not* do until after a successful rebuild of Disk 13.

 

i have asked this question before, but i'll ask it again: what speaks against formatting Disk 13 and adding it as a "replacement" disk? if that is a valid thing to attempt, what would the terminal command be on unRAID to do this, and what would the correct steps in the Web GUI be to provoke a rebuild this way?

 

so, knowing that you'll spend a couple of well-deserved days doing offline stuff (way to go!), i'll just remain in a holding pattern until i get further instructions as to what to try next.

unfortunately, the behavior under the stock Web GUI is just like under the Dynamix Web GUI: nothing happens.

 

when i hit the "Start" button, the array is started and in an all-green state...if there is any rebuilding going on (which i strongly doubt), there is no feedback given in the Web GUI or the terminal window.

 

Parity still shows as "valid", but the system knows that "Parity has not been checked yet.", suggesting a Parity check, which i will *not* do until after a successful rebuild of Disk 13.

 

i have asked this question before, but i'll ask it again: what speaks against formatting Disk 13 and adding it as a "replacement" disk? if that is a valid thing to attempt, what would the terminal command be on unRAID to do this, and what would the correct steps in the Web GUI be to provoke a rebuild this way?

 

so, knowing that you'll spend a couple of well-deserved days doing offline stuff (way to go!), i'll just remain in a holding pattern until i get further instructions as to what to try next.

As I said before, formatting is not part of a rebuild. If you format the disk in unRAID, unRAID will update the parity to reflect the fact that you have a newly formatted disk as part of your array. If you format the disk in some other system, it is pointless because when you rebuild a disk it doesn't matter what was originally on the "new" disk because it will be completely overwritten bit by bit. I stress this because I don't want you to make the mistake of letting unRAID format the disk.

 

The key to understanding this is to realize that a "formatted" disk is not an empty disk, it is an empty filesystem.

 

I think what you're trying to get at is making unRAID "forget" disk 13 and have it treat it as a "new" disk to rebuild onto. That is usually done by stopping the array, unassigning the disk, starting the array so that it shows up as missing, stopping the array, assigning the disk, starting the array and letting it rebuild. I'm not completely sure I have this exactly right since I have not done it. Perhaps someone else will comment.

trurl, you points are all well taken, and i admit to have mis-understood some of the terminology...thanks for saving me from a potentially grave mistake of trying to format the disk within the unRAID system.

 

the procedure you "guessed" is exactly what i was asking for, but it's been so long that i've had to do this (i did do something exactly like this many years ago when an almost new disk kept mis-behaving but passed all SMART and health checks with flying colors).

 

let me research this again and if i find that it is the same thing i did years ago, i will attempt to do this.