Red Drive 8, Orange Start

[Joe L.]

Joe L, trying to get all my questions out for step by step when you are unavailable later.  At some point when I go back to rebuild the rest of the disks, can I just stop the array with the web browser, and then put my command in as usual? Looks like it's taking about 8 hours per disk.  For now, I'm going to do nothing, until I here from you about what is next.

 

Roger

Once we get disk 8 to be part of the array, (after you run --rebuild-tree -S you can stop the array and assign it)

 

Then, you can assign the parity drive and let it recalculate.

 

Then, for each disk you did not run --rebuild-tree -S you can follow the directions as in the wiki :http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

to check each disk in turn as needed (example here for  disk7)

umount /mnt/disk7

reiserfsck --rebuild-tree -S /dev/md7

mount /dev/md7 /mnt/disk7

and repeat as needed for each disk in turn where you did not yet rebuild the file-system-tree.

You do this with the array started...  When you are done parity will be in sync with all you have recovered.

 

 

[Roger]

Joe L, so just to be clear, next step, with the array running, I type the following into the console at the # prompt:

 

reiserfsck --rebuild-tree -S /dev/sdj1

 

This will rebuild disk 8.  

 

If it works, then stop the array, and add the unassigned disk 8 by assigning it.  correct?

Will it then want to format it or not?

 

Once I have done the above, based on your answers, I do the following for Disk 7

 

umount /mnt/disk7

reiserfsck --rebuild-tree -S /dev/md7

mount /dev/md7 /mnt/disk7

 

Since disk 7 is part of the array, I do not reference it as "sdi1" as I would if it were not, like I did for all the disks with the array stopped.  Is this correct?

 

Also, and this is important, you are telling me once complete the rebuild on Disk 8, I can stop array, and then start it, and let the parity rebuild, even though I haven't fixed Disk 7, or run rebuild on any disks other than Disk 1, Disk 3, and Disk 8, at this point?  So once the parity is all up and all drives are seen, then I go on to try and fix Disk7?

 

[Joe L.]

Joe L, so just to be clear, next step, with the array running, I type the following into the console at the # prompt:

 

reiserfsck --rebuild-tree -S /dev/sdj1

 

This will rebuild disk 8.  

assuming disk 8 is /dev/sdj, yes.  (  I did not go back in the thread to verify the device)

 

If it works, then stop the array, and add the unassigned disk 8 by assigning it.  correct?

Yes.  After assigning it, and before starting the array, you again need to issue the

initconfig

command to let unRAID you have a new disk configuration.  Again, all the drives should be "blue" when you go back to the main management page.  You can then start the array.  (If not all blue, do not continue, do not start the array, and definitely do not press the Format button just yet)

 

Will it then want to format it or not?

If it has a file system, it will not want to format it.

 

Once I have done the above, based on your answers, I do the following for Disk 7

 

umount /mnt/disk7

reiserfsck --rebuild-tree -S /dev/md7

mount /dev/md7 /mnt/disk7

 

Since disk 7 is part of the array, I do not reference it as "sdi1" as I would if it were not, like I did for all the disks with the array stopped.  Is this correct?

Correct.  programatically, sdi1 is connected through the /dev/md7 driver.  Note, I used disk7 as an example.  you need to run that set of commands for each of the disks where you did not run "--rebuild-tree -S"

 

Also, and this is important, you are telling me once complete the rebuild on Disk 8, I can stop array, and then start it, and let the parity rebuild, even though I haven't fixed Disk 7, or run rebuild on any disks other than Disk 1, Disk 3, and Disk 8, at this point?  So once the parity is all up and all drives are seen, then I go on to try and fix Disk7?

yes.  At that point you can do the rest with parity protection in place.

 

rebuild-tree on disk 8

stop array

assign disk8

initconfig

start array

if all looks good, stop array

assign parity

start array, let it calculate parity.

then rebuild-tree -S on disk7, and others as needed

 

[Roger]

Thank you. I begin tonight with Disk 8. Likely need to work dlk night. Thank you

 

Roger

[Roger]

Joe L, I may have made a mistake.  I am running the reiserchk -- rebuild command on /dev/sdh1, with the array still running.  I thought that was Disk 8, but; on the device menu that looks like Disk 14 which was not having a problem.  It found 2 trnsactions replayed, and it's working away, but; I did not to the umount or anything.  I figured if I was wrong, and I got confused it wouldn't run.  Hopefully I didn't mess anything up.

 

Looking at the Disk numbers in the post, I really got confused as to whether it was sdj1 or sdh1 or now I think it's actually sdo1 I should have run the test on.  Going back to the post, the unassigned disk, disk 8 appears to be "sdo"  "sdo" is not in the assigned device table but sdh is, and it is as Disk 14.

 

When I'm done, before I stop the array, before I assign the repaired disk 8, do I just run the rebuild one more time on sdo, realize it's disk 8, then stop, assign, run the interconfig command, and move on?  Or, is it rebuilding something that is attached to the array, that I needed to umount and didn't and so I messed things up?

 

I guess I got confused by the order of things vs. my old print out of devices which had it as "sdh" vs. assigned vs. unassigned, but; in this case the actual physical on the drive, seems to appear as "sdo" regardless

 

Roger

[Roger]

Also, interestingly, what I thought was Disk 13 with all the problems I think is Disk 8, because when I look at the post you made it appears as "sdo"  Which makes sense why the Disk 13 that is attached is working, with all files attached.  It shows up on the devices page as "sdl"

 

Oh this is very hard for me.

 

Roger

[Roger]

Getting all kind of errors in the the Reiserfs check (device md14), so this seems consistent with my fear.  Hopefully, this means at the end of Reiserfs it will just give up, and all will remain good.  If drive 8 is in fact "sdo" which I thought was drive 13, what next?  How long will it run before it gives up?  Stuck on "REISERFS error (device md14): vs-5150 search_by_key: invalid format found in block ck 21965612.  Fsck? REISERFS error-read_locked_inode: i/o failure occurred trying to find stat data of [5658 5731 0x0 SD].  Page of these messages repeating

 

Roger

[Joe L.]

Joe L, I may have made a mistake.  I am running the reiserchk -- rebuild command on /dev/sdh1, with the array still running.  I thought that was Disk 8, but; on the device menu that looks like Disk 14 which was not having a problem.  It found 2 trnsactions replayed, and it's working away, but; I did not to the umount or anything.  I figured if I was wrong, and I got confused it wouldn't run.  Hopefully I didn't mess anything up.

 

Looking at the Disk numbers in the post, I really got confused as to whether it was sdj1 or sdh1 or now I think it's actually sdo1 I should have run the test on.  Going back to the post, the unassigned disk, disk 8 appears to be "sdo"  "sdo" is not in the assigned device table but sdh is, and it is as Disk 14.

 

When I'm done, before I stop the array, before I assign the repaired disk 8, do I just run the rebuild one more time on sdo, realize it's disk 8, then stop, assign, run the interconfig command, and move on?  Or, is it rebuilding something that is attached to the array, that I needed to umount and didn't and so I messed things up?

 

I guess I got confused by the order of things vs. my old print out of devices which had it as "sdh" vs. assigned vs. unassigned, but; in this case the actual physical on the drive, seems to appear as "sdo" regardless

 

Roger

The three letter device names can change from one boot to the next.  They are assigned as the disks initialize and present themselves to the Linux OS.  With identical drives, it happens occasionally.   If you reboot, look at the "devices" page to see the currently assigned device names.   They will sometimes change if you plug a new disk into one of your disk controllers.

 

I did not think you could run a reiserfsck on a mounted disk if you were using the dev/mdX devices.   Since you have no parity disk assigned, I don't think you did any harm by using the /dev/sdX1 partition.   That would NOT be good if you did have parity assigned and calculated, but since you don't at this time, no issue.   Just don't try to run reiserfsck on the parity disk.  It has no file-system to repair, so it will appear to not have a superblock to reiserfsck, since it does not.   Just ignore the parity disk for now.  When eventually stop the array and assign it, it will take care of itself.

 

You are getting a crash course on how linux devices are assigned under unRAID's device assignment screen.  If you need, jot down the disk number, the last 4 digits of their serial number, and verify using the "Devices" screen as you go.   Check each physical disk off the list as you progress.  I know it is slow, but the disks can only be read at about 100 MB/s at best.  10 seconds for 1GB, about 6GB per minute, and you have 2000 GB on a 2TB disk.  When you use the -S option to rebuild-tree, it needs to scan the entire disk.     It will be at least 5 1/2 hours for any given 2TB disk, and probably a bit longer if your disks cannot keep up a 100MB/s read rate. (most cannot, and drop to 80MB/s or less on inner cylinders)

[Joe L.]

Getting all kind of errors in the the Reiserfs check (device md14), so this seems consistent with my fear.  Hopefully, this means at the end of Reiserfs it will just give up, and all will remain good.  If drive 8 is in fact "sdo" which I thought was drive 13, what next?  How long will it run before it gives up?  Stuck on "REISERFS error (device md14): vs-5150 search_by_key: invalid format found in block ck 21965612.  Fsck? REISERFS error-read_locked_inode: i/o failure occurred trying to find stat data of [5658 5731 0x0 SD].  Page of these messages repeating

 

Roger

Are you running the --check on /dev/md14 ?  or the --rebuild-tree ?

Are you seeing the errors on your system console? or in the syslog?

[Roger]

One the console.  It's pretty much stuck in the reiserfs --rebuild /s /dev/sdh1, because as I have found out, I ran this on the wronge device and this device is mounted, so I guess that's why I am getting device md14 reference.  Figured it out but didn't know what to do without your help to stop it, so left it running.  It's been about 12 hours, and I don't see how I can get out of this screen without rebooting the tower, but; waiting for you to tell what to do next.

 

As the correct disk 8, appears to be "sdo" for now, don't know what it will be when I reboot (learning), and "sdo" crashed earlier on the rebuild, running it again before we try and assign it, initconfig won't work. BUt; if I assignn it, init config, after I reboot, it will likely give me the chance to format it, what do you think?

 

So, to be clear on my answers to your questions:

 

Are you running the --check on /dev/md14 ?  or the --rebuild-tree ?  I am running reiserfs --rebuild /S on /dev/sdh1, which is in fact md14, rather than the Disk8 I thought.

 

Are you seeing the errors on your system console? YES

 

 

 

ROger

[Joe L.]

One the console.  It's pretty much stuck in the reiserfs --rebuild /s /dev/sdh1, because as I have found out, I ran this on the wronge device and this device is mounted, so I guess that's why I am getting device md14 reference.  Figured it out but didn't know what to do without your help to stop it, so left it running.  It's been about 12 hours, and I don't see how I can get out of this screen without rebooting the tower, but; waiting for you to tell what to do next.

 

As the correct disk 8, appears to be "sdo" for now, don't know what it will be when I reboot (learning), and "sdo" crashed earlier on the rebuild, running it again before we try and assign it, initconfig won't work. BUt; if I assignn it, init config, after I reboot, it will likely give me the chance to format it, what do you think?

 

So, to be clear on my answers to your questions:

 

Are you running the --check on /dev/md14 ?   or the --rebuild-tree ?  I am running reiserfs --rebuild /S on /dev/sdh1, which is in fact md14, rather than the Disk8 I thought.

 

Are you seeing the errors on your system console? YES

 

 

 

ROger

When using the system console you can interrupt (kill) and process you started by pressing the Control-C key.  (Hold the control-key down and press the letter C)

 

When using the system console you are actually using one of 6 virtual consoles.  You can switch between them by typing

Alt-F1 through Alt-F6

At each virtual console you will get a login prompt, and you can log in.  You can have up to 6 reiserfsck commands running at the same time, one on each virtual console.  Switching to a different virtual-console does not terminate what is running on the current console.  Normally, you are using the first virtual console, so you'll need to type Alt-F2 to get to the second one, and Alt-F1 to get back to the first, etc.

 

 

[Roger]

Joe L., so you can confirm, I should hit the Control C, because it's certainly stuck.  Than, what do I do next?  Do I try and assign, initconfig Disk 8, like we earlier discussed?  Even though we know I can't run the rebuild first because last time I tried it failed and gave me the errors?  Then try and format 8 through the web browser?  Then if it works at the parity?  

 

Is this correct?

 

Then, based on the information you gave me, can I run 6 rebuilds at the same time to try and get the other disks done sooner than 8 days.

 

Roger

[Joe L.]

Joe L., so you can confirm, I should hit the Control C, because it's certainly stuck.

Yes.

 Than, what do I do next?  Do I try and assign, initconfig Disk 8, like we earlier discussed?  Even though we know I can't run the rebuild first because last time I tried it failed and gave me the errors?  Then try and format 8 through the web browser?

Sounds good.

 Then if it works at the parity?  

If it works, assign parity.

Is this correct?

As good as any at this time

 

Then, based on the information you gave me, can I run 6 rebuilds at the same time to try and get the other disks done sooner than 8 days.

Yes.

[Roger]

SEEMS LIke it's taking a long time to stop the array and unmount the drives, is that something to worry about? Looks like it's stuck doing this, should I use shut down command from the console?

[Joe L.]

SEEMS LIke it's taking a long time to stop the array and unmount the drives, is that something to worry about? Looks like it's stuck doing this, should I use shut down command from the console?

a disk cannot be un-mounted if it is busy.  It is busy if a file on it is open for reading or writing, or if it is the current directory for some process.  Make sure you change directory back to /boot or / or it might be you keeping the array from stopping.

 

Joe L.

[Roger]

Not really doing anything. Maybe it's time between cancel of command. Can just reboot.

[Joe L.]

Not really doing anything. Maybe it's time between cancel of command. Can just reboot.

Since you don't really have much to lose at this time, you can probably reboot.  Before you do, type

sync

to try to flush the disk buffers to the data disks.

[Roger]

Getting encouraged.  It is formatting disk 8 now.  When I started the array using web browser, I got the following on the console:

"reiserfs error (device : vs-5150 syearch_by_key: invalid format found in block 0. Fsck?

Reiserfs error (device md8): vs-13070 reiserfs_read_locked_inode: i/o failure occured trying to find stat data of (1 2 0x0 SD).  Then everything came up in the web browser correctly, and only one disk, Disk 8 showed unformatted, so I hit format?

 

Is this good news/

 

Roger

[Roger]

Everything is green, Disk 8 formatted, but now shows nothing on it.  What do I do next?

 

ROger

[Roger]

Joe L, so I figure, why not try the Alt F2 command and run a rebuild on Disk12, while the rebuild on Disk8 is happening.  Did the umount, etc. then when I tried to run it, it said, no such file or directory.  Is this because I can't do multiple rebuild's until I get Disk 8 finished, so I can get it remounted and than the parity going?  So that without the parity drive online with all disks online I cannot rebuild more than one Disk at a time?

 

Not sure if I misunderstood, but; I thought I was supposed to run the rebuild on Disk 8 after the format before assigning the parity.  I'm doing that now, is that O.K.?

 

Roger

[Joe L.]

Joe L, so I figure, why not try the Alt F2 command and run a rebuild on Disk12, while the rebuild on Disk8 is happening.  Did the umount, etc. then when I tried to run it, it said, no such file or directory.  Is this because I can't do multiple rebuild's until I get Disk 8 finished, so I can get it remounted and than the parity going?  So that without the parity drive online with all disks online I cannot rebuild more than one Disk at a time?

 

Not sure if I misunderstood, but; I thought I was supposed to run the rebuild on Disk 8 after the format before assigning the parity.  I'm doing that now, is that O.K.?

 

Roger

sounds fine.  you cannot un-mount a drive if it was not mounted to start with.

type

mount

to see what is currently mounted.

 

soon, the --rebuild-tree -S will be completed on disk 8.

[Roger]

Got the same long message, with a failure on the Disk 8.  Says the Problem has occurred looks like a hardware problem.  If you have bad blocks, we advise you to get a new hard drive, because once you get on bad block that the disk drive internals cannot hide from your sight, the cnances of getting more are generally said to become much higher... If it cannot remap the block, use the badblock option (-B_ with reiserfs utils to handle this block directly.

 

Now what?  Do I just mount up and hit parity?

 

Roger

[Roger]

Can't even mount Disk 8 back.  Do I unassign and assign back.  Gives me the choice to format it, now when I start the array again.

 

Roger

[Joe L.]

Can't even mount Disk 8 back.  Do I unassign and assign back.  Gives me the choice to format it, now when I start the array again.

 

Roger

Assign it, then type

initconfig

before starting the array, and it should accept it without re-formatting, assuming it has a file-system that checks OK.

if not, let it format it... then you can run the --rebuild-tree -S on it after it formats it.

[Roger]

Done this 3 times now.  Each time it gives me an error when I run rebuild command.  Do you know much about the -B command so I can fix the bad block?  It's now showing as O.K. on the web browser, but you can see when it was formatting on the console it was having issues agailne with locked inode and i/o failure.  Any ideas?