Red Drive 8, Orange Start

[Joe L.]

Disk 4 is O.K., so I think the theory that Disk 3, 5, and 8 have the issue may hold up.  No corruptions found on it.  Got to the answer pretty quick. Does this mean it's O.K., or could there still be something wrong with the tree?

or, you re-formatted them so they checked quickly.  Again, only care where it says corruptions are found.

[Roger]

Disk 5 just came back with no corruptions, but; unlike Disk 1, it processed really quick and didn't show me a list of files it was checking.  Disk 1 went through file after file mp3's mostly, so far disk 2, 4, and 5, which show no corruptions aren't doing that.

[Joe L.]

Disk 5 just came back with no corruptions, but; unlike Disk 1, it processed really quick and didn't show me a list of files it was checking.  Disk 1 went through file after file mp3's mostly, so far disk 2, 4, and 5, which show no corruptions aren't doing that.

As I sais earlier, the bug you fell into when all the disks said "unformatted" and you asked it to format them had resulted in re-formatting disks that had previously held data.

 

You can recover the deleted files in many cases by asking reiserfsck to rebuild the file-tree.  Only do that if there are no other corruptions.

 

So, keep track of those needing repair.  those will be the second pass through the disks.  After they are fixed, in the third pass you'll ask it to rebuild the file-trees with all the files it finds that might have been previously deleted.  For that, you again do each disk in turn, even those that did not need repair.

 

Joe L.

[Roger]

Moving right along, no errors so far except Disk 3, wondering how I get to the stdout.txt file, and how I get to the TXT files I am creating for each disk with the Tee command.  Just trying to get ready for the next post, where we get to the fix issues. So far, only Disk 3, surprised Disk 5 didn't show an error.

[Joe L.]

Moving right along, no errors so far except Disk 3, wondering how I get to the stdout.txt file, and how I get to the TXT files I am creating for each disk with the Tee command.  Just trying to get ready for the next post, where we get to the fix issues. So far, only Disk 3, surprised Disk 5 didn't show an error.

if you can connect via windows file explorer, browse to

\\tower\flash

and they will be there.

[Roger]

So far, errors found on sdd1, or disk 3, and sdi1, or disk 7, just reporting

[Roger]

O.K., keeping track, for some reason, can't access through windows explorer, when I initially booted, I could, but somewhere along the line, with these reiserfsck's lost connectivity. 

[Roger]

Can I access stout or the txt files I created with tee command from console?

[Roger]

Joe L, and team, I'm done with the first pass of the reiserfsck, here is what I got: All disks check out O.K., except the following, with the following information:

 

riserfsck --check /dev/sdd1 DISK3    1 corruption

reiserfsck --check /dev/sdi1 DISK7    1 corruption

reiserfsck --check /dev/sdj1 DISK8    1 corruption

reiserfsck --check /dev/sdn1 DISk12 2 corruptions

reiserfsck --check /dev/sdo1 Disk13  1 corruption

All kinds of stuff comes up with Disk 13, after the journal parameters superblock mismatch, journal header fixed, I get the following after the Replaying journal:

Trans replayed: mountid 205, transid 48, desc 139, len , commit 141, next trans offset 124

Trans replayed: mountid 205, transid 49, desc 142, len 1 commit 144, next ransoffset 127, Fatal corruptions were found on the comparing bitmaps

 

What's next?  Can't get to the stout file or the text files I created through the tee command unless I reboot, I'm afraid to reboot, because things will change, what can I do to get to this information without rebooting and having all things change?

 

 

[Roger]

I can get to the gui screen by going to the tower ip, but; don't know how to get the stdout or the txt of each failed drive to post without rebooting, need help

[Roger]

Should I take the USB out of server and access the flash that way, then return to tower?

[Joe L.]

Should I take the USB out of server and access the flash that way, then return to tower?

use window's file explorer, not Internet explorer.

[Roger]

O.K, Joe L, got all the files, I can look in them, but even zipping them they are too big to upload

[Joe L.]

O.K, Joe L, got all the files, I can look in them, but even zipping them they are too big to upload

I don't need any files.  Your syslog told the story, and you are doing the rest.

 

Joe L.

[Roger]

O.K., so what's next?  What command to I run on each of the disks that are corrupt?

[Roger]

Joe L, I'm guessing the next thing is I need to run the reiserfsck with the (--fix-fixable option) on the second pass on the broken disks, and then on the 3rd pass run the rebuild-sb on all disks 1-14?  is this correct?  Or do I run the --fix-fixable on all 14, or do something different?

 

Once I have done the fixes, you said running any of the repair options in reiserfsck (--fix-fixable, --rebuild-tree, --rebuild-sb, etc ) on the /dev/sdX1 device will invalidate parity.  That will then require a parity calc be re-run to correct parity once all the data disks are back to having good file systems.  HOW DO I DO A PARITY CALC TO CORRECT PARITY?

 

I remember you gave me lots of warnings here, so I stand ready for you to tell me what's next?  Sorry I'm so lame, but; I also don't know what "if a check option indicates repair is needed, how to post the output of the --check output, so I can get help?

 

Roger

 

 

[Roger]

Is the next step run fix-fixable?  If so, do I run only on the 5 drives that had the issues?

[Roger]

Can someone confirm that the next thing I do is run

 

reiserfsck --fix-fixable, then,

 

reiserfsck --rebuild-tree -S /dev/sdX1

 

And, I don't run the rebuild-tree - S until I get some help?

[Roger]

Just wondering, sdd1_check.txt file, which is huge and I can't post, I found:

 

sdd1_check.txt says:

 

ON this disk we have bad_leaf: block and items 1 & 2 the wrong order of items at the beginning, and at the end

The on-disk and the correct bitmaps differs.

 

Does this mean anything other than run the fix command?  And can someone tell me exactly what to type

 

[Roger]

Hate to bother everyone, but; still confused on things.  Where do I find the  reiserfsck report that was supposedly output and supposed to tell me exactly what to do?

 

[Joe L.]

Hate to bother everyone, but; still confused on things.  Where do I find the  reiserfsck report that was supposedly output and supposed to tell me exactly what to do?

 

Start with the first that reported errors, run reiserfsck with --fix-fixable.  Its output will then tell you if there is a need for anything else.

[Roger]

Joe, thank you so much, I did this, it says to run the rebuild-tree to fix, So as I understand the command for that disk it is reiserfsck --rebuild-tree -S /dev/sdd1.  I should expect stuff to tell me items have been deleted correct?  And if it does, don't worry, just run this on the rest of the disks that have errors:

 

reiserfsck --check /dev/sdi1 DISK7    1 corruption

reiserfsck --check /dev/sdj1 DISK8    1 corruption

reiserfsck --check /dev/sdn1 DISk12 2 corruptions

reiserfsck --check /dev/sdo1 Disk13

 

Once I fix all errors on the bad disks, run the same command on all the good disks, correct?  On Disk 13 I got the most messages:

After the journal parameters superblock mismatch, journal header fixed, I get the following after the Replaying journal:

Trans replayed: mountid 205, transid 48, desc 139, len , commit 141, next trans offset 124

Trans replayed: mountid 205, transid 49, desc 142, len 1 commit 144, next ransoffset 127, Fatal corruptions were found on the comparing bitmaps.

Does this mean I have a SB or superblock problem, needing you to walk me carefully through?

 

Also, when I'm done how do I run the recalcl parity.  Do I do that through the terminal or just start my array and let it do it?

 

Roger

 

 

[Roger]

Reading my output files I created with reiserfs, and wondering what this means?

 

/  1 (of   9)/  1 (of 162)/  1 (of  85)bad_leaf: block 3786354, items 1 and 2: The wrong order of items: [1 2 0x1 DIR (3)], [2 4 0x1 DIR (3)], or this

/  1 (of  11)/  1 (of  90)/  1 (of  88)bad_path: block 32770, pointer 0: The used space (472) of the child block (8211) is not equal to the (blocksize (4096) - free space (3900) - header size (24))

Also, if it takes 5 hours per drive to run rebuild-tree, on the 2nd pass to fix the 5 drives, that's 25 hours, to get to pass 3, then I have run the exact same command on all 14 drives, is that correct?  This will take another 70 hours. Then when I'm done, I run the redo parity calc thing, which I am still needing help for, correct?  And obviously, if I get a message that says to run the SB command, I need someone to walk me through that.

 

I know this is days off, but to rerun parity calc, do I type this command?  "mdcmd check CORRECT"  Or something else.

 

 

I feel like a complete loser, can't sleep, while I'm like in the 6th hour of trying to rebuild the directory tree on just one of my 5 drives.

[Roger]

JOE l, or any other savor.  I have now run fix-fixable and rebuilding tree command on sdd1 (disk 3), and on sdi1 (Disk 7) and sdj1 (Disk , I have only had to run --fix-fixable.  I then went back and did a reiserfsck --check, to see if the disks were O.K., and they came back O.K They all come back O.K., except sdo1 (disk 13) when I go back and run the check/dev/ command.  Need to fix by running with the --rebuild-tree, and I'll do that while I'm sleeping now. 

 

Am I on the right path?  Still have my other questions, but; do I really need to do the rebuild command on all 14 drives, including the Disk3, I already ran in on before I can boot my array, and (I think through the GUI), rebuild the parity?  

 

Can I get step by step from here to finish line?

Sorry to keep asking questions, but; I am a real newbie.

 

Roger

[Joe L.]

JOE l, or any other savor.  I have now run fix-fixable and rebuilding tree command on sdd1 (disk 3), and on sdi1 (Disk 7) and sdj1 (Disk , I have only had to run --fix-fixable.  I then went back and did a reiserfsck --check, to see if the disks were O.K., and they came back O.K They all come back O.K., except sdo1 (disk 13) when I go back and run the check/dev/ command.  Need to fix by running with the --rebuild-tree, and I'll do that while I'm sleeping now. 

 

Am I on the right path?  Still have my other questions, but; do I really need to do the rebuild command on all 14 drives, including the Disk3, I already ran in on before I can boot my array, and (I think through the GUI), rebuild the parity?  

 

As long as you are not writing files in the array, you can start it as soon as you like.  Once you start it you need to switch to using the /dev/mdX devices to run the reiserfsck commands, and need to stop SAMBA and unmount the disks.  For now, it is a complication you really do not need to bother with.

 

Can I get step by step from here to finish line?

Sorry to keep asking questions, but; I am a real newbie.

 

Roger

You are getting close.

 

Once you get all the disks to pass a "--check", then do a full round of "--rebuild-tree -S"  the "-S" is the additional argument that forces the check to scan the entire disk for deleted files.  It will place those directories and files in a new lost+found folder it will create.

 

Then, we need to force a new disk configuration to get disk 8 back in the array and to force it to rebuild parity.

You do that by typing

initconfig

while the array is stopped.  answer "Yes" on the command line (Capital "Y", lower case "es") to its prompt.

 

Then back on the management web-page, refresh it and all the disks should show as "blue"  Then start the array.  It will completely rebuild parity.

 

Joe L.