Red Drive 8, Orange Start

[Roger]

Joe L, thanks for your response, but; I have a glitch that looks like it will make it harder.  On the --rebuild-tree command on Disk 13, the one I had the most issues with, I got a paragraph long failure that ends with Aborted.  It says:  The problem has occurred looks like a hardare problem.  If you have bad blocks, we advise you to get a new hard drive, because once you get one bad block that the disk drive internals cannot hide from your sight, the chances of getting more are generally said to become much higher (precise statistics are unknown to us), and this disk drive is probably not expensive enough for you to you to risk your time and data on it.  If you don't want to follow that follow that advice then if you have just a few bad blocks, try writing to the bad blocs and see if the drive remaps the bad blocks (that means it takes a block it has in reserver and allocates it for use for of that block number).  If it cannot remap the block, use badblock option (-B) with reiserfs utils to handle this block correctly.

 

bread: Cannot read the block (52050987): (input/output error).

 

When it started it said their were 2 issues, block 3786354: item 2: The item (2 4 0x1 DIR (3) ], which follows non StatDat item [1 2 0x1 Dir (3)], is deleted. It looks like my Disk 13 is a 1.5terabyte drive, stats were 1,465,138,552 (size) 65,376,880 (free).

 

OMG, what now?  Getting physically sick.

 

Assuming I'm not dead, do I run the build command on all 14 drives, including the ones I have already done in this pass, which will be at least Disk 3, and this 13, with some additional command like (-b).

 

Do I somehow use smartctl?  If so what is the exact command I need to run?  Tried to do a test using it, can't figure it out

 

 

Roger

[Roger]

I'm so lame, can't figure out how to do the smartctl test.  It would seem from my reading there is a way to repair this.  Likely when I powered off Drive 13, as opposed to Drive 8,which I would have expected, wrote a bad block.

:'(

[Joe L.]

I'm so lame, can't figure out how to do the smartctl test.  It would seem from my reading there is a way to repair this.  Likely when I powered off Drive 13, as opposed to Drive 8,which I would have expected, wrote a bad block.

:'(

Or, it had bad blocks.

 

Type

smartctl -a /dev/sdo

to get the smart report.

 

You are looking for the lines that are for re-llocated sectors, or sectors pending re-allocation.

[Roger]

when I type that, stuff goes by the screen really fast.  You mean me to type smartctl -a /dev/sdo1 or just sdo?  And how do I capture the information before it goes by.  What I seem to get is two sections, one that says Commands leading to the command that caused the error.  Read DMA EXT is the first line. 25 00 01 93 31 d1 18 08.  The second section says Smart Selective self-test log data structure revision number 1, and on Spans 1-5, it says Not-testing

[Joe L.]

when I type that, stuff goes by the screen really fast.  You mean me to type smartctl -a /dev/sdo1 or just sdo?  And how do I capture the information before it goes by

Just /dev/sdo

To capture, use the same technique with the "tee"

 

smartctl -a /dev/sdo | tee >/boot/sdo_smart.txt

[Roger]

This tells me no space left on device.  Should I delete some of the other stuff I put on it?

[Roger]

Moved excess files to desktop, but still get the same response, /boot/sdo_smart.txt: No space left on device. Also, don't really get anything other than what I posted, Not sure why

[Roger]

Panic is setting in from other posts I read.  Smarctl isn't working right.

[Joe L.]

Panic is setting in from other posts I read.  Smarctl isn't working right.

What is not working right? 

 

Are you using "telnet" or working on the system console?  If you unplugged the flash drive, odds are you'll need to reboot to get it recognized.

 

If not, type:

df | grep boot

to learn of its free space.

 

As far as your files... with an apparent defective disk, and unusable parity, odds are you will lose some data.    First step is NOT to panic.

Next is to see if the disk is really defective.

 

Joe L.

[Roger]

If I reboot. Because I did unplug USB. To grab files. Will everything change?

The issue I am having with smartctl us that when I type in your exact command I don't really get info needed to fix the bad block. I don't think dusk 13 itself is bad only 1 or 2 blocks which happened when I shit down during format. If you would call me and walk me through this it would be great. I'll fo home. How much wod you charge, if anything.

[Joe L.]

If I reboot. Because I did unplug USB. To grab files. Will everything change?

The issue I am having with smartctl us that when I type in your exact command I don't really get info needed to fix the bad block. I don't think dusk 13 itself is bad only 1 or 2 blocks which happened when I shit down during format. If you would call me and walk me through this it would be great. I'll fo home. How much wod you charge, if anything.

Nothing will "fix" a unreadable sector on ANY disk.  you can only re-write the sector, and then the disk will first try to re-write the un-readable sector and then, if unable, re-map the un-readable sector to a spare one and write the new value to it.

 

You are not answering my questions though.

Are you using the system console?  Or, are you using a telnet session? to type the linux commands?

 

as far as the defective/un-readable disk, I'd just re-format it.  (exactly as it was attempting to do when you pulled the power)

It will attempt to write the bad sector, and it will re-map it if it can.  Then, you can use "--rebuild-tree -S"

to try to get the old files back.

 

Joe L.

[Roger]

I have a keyboard and monitor hooked up directly to the machine. 

[Roger]

Actually Disk 13 wasnt formatting. Disk 8 was.  Disk 13 worked fine.  Disk 3, 5 and 8 showed no files.

I apologize. I really don't understand what to do next. Do you want me to reboot. Start array. Which Wilkins have it try and rebuild parity. Including Diak 8 which is now blue. When done hit format some how on Diak 13. Help I am really confused

[Roger]

Joe L,

 

I am a real newbie and don't want to mess this up.  Just a reminder, I have run the fix fixable on 5 disks, 3 fixed, and confirmed with check. 2 did not.  Disk 3 needed the rebuild, once I did that and did the check, it came back O.K.  Disk 13 is the one that rebuild won't work, for the message I posted, and I can't seem to get the smartctl to tell me exactly what's wrong, other than what I posted from the rebuild error and the smartctrl screen output.  Disk 13 was fully readable last time the tower worked.  Only Disk 3, Disk 5, and Disk 8 seemed to have no files. At this point, I am not using the GUIC from //tower/main, I am only using the console, or the keyboard and the monitor plugged directly into the tower.

 

Are you telling me, before I run the rebuild again, as you earlier suggested, on Disk1-14, I reboot the tower, go into the GUIC on my regular computer //tower/ hit start, let it rebuild the parity, etc., and then ask it to reformat disk 13?  RIght now all disks show green, including Disk 13, except for Disk 8, which shows blue.  Array is of course, stopped

 

So, step by step, what exactly is it I need to do, what button do I push, what command do I type.  Assume I'm an idiot

[Joe L.]

Joe L,

 

I am a real newbie and don't want to mess this up.  Just a reminder, I have run the fix fixable on 5 disks, 3 fixed, and confirmed with check. 2 did not.  Disk 3 needed the rebuild, once I did that and did the check, it came back O.K.  Disk 13 is the one that rebuild won't work, for the message I posted, and I can't seem to get the smartctl to tell me exactly what's wrong, other than what I posted from the rebuild error and the smartctrl screen output.  Disk 13 was fully readable last time the tower worked.  Only Disk 3, Disk 5, and Disk 8 seemed to have no files. At this point, I am not using the GUIC from //tower/main, I am only using the console, or the keyboard and the monitor plugged directly into the tower.

 

Are you telling me, before I run the rebuild again, as you earlier suggested, on Disk1-14, I reboot the tower, go into the GUIC on my regular computer //tower/ hit start, let it rebuild the parity, etc., and then ask it to reformat disk 13?  RIght now all disks show green, including Disk 13, except for Disk 8, which shows blue.  Array is of course, stopped

 

So, step by step, what exactly is it I need to do, what button do I push, what command do I type.  Assume I'm an idiot

 

Step by step.

 

First, get as much of your array up and sane as possible.  Let's un-assign disk8 and the parity disk.  (that will ensure you do not try to rebuild from parity)

Then, let's set a new disk configuration by typing

initconfig

and answering "Yes" to its prompt.

 

When you then refresh the management screen all should be blue.  Then start the array.  It will not have a parity disk, so all the others should mount and no re-construction will occur.  Do NOT write and new files to the data disks in the array.  Not until you recover the deleted files.

 

If you have already run

"--rebuild-tree -S"

on a disk, and it was successful, then you can write to it.

 

You can look around and see what has happened.  We'll deal with disk8 and 13 last.  When you start the array, it may show disk13 as unformatted.  If it is the ONLY disk showing unformatted, and it was the one with the bad blocks, then you can press the button to format it.  BUT DO NOT WRITE ANY NEW FILES TO IT (or delete any files, or modify any files)!!

 

Once all the files are recovered that can be, we'll re-assign the parity disk and it will automatically re-calculate itself.

 

Joe L.

[Roger]

Joe L, still confused and would love to talk to you by phone, I will rush home if you say yes.  My questions are ALL CAP

 

Step by step.

 

First, get as much of your array up and sane as possible.  Let's un-assign disk8 and the parity disk.  (that will ensure you do not try to rebuild from parity)

 

HOW DO I DO THIS ( I AM ASSUMING THERE IS A WAY TO DO THIS FROM THE CONSOLE)? If not do I do this from the //tower/main page?

 

Then, let's set a new disk configuration by typing

 

initconfig

and answering "Yes" to its prompt.

 

When you then refresh the management screen all should be blue.  Then start the array.  It will not have a parity disk, so all the others should mount and no re-construction will occur.  Do NOT write and new files to the data disks in the array.  Not until you recover the deleted files.

 

If you have already run

"--rebuild-tree -S"

on a disk, and it was successful, then you can write to it.

 

AT THE END, ARE YOU STILL GOING TO WANT ME TO DO THE --rebuild-tree -S on all 14 disks, one last time?

 

You can look around and see what has happened.  We'll deal with disk8 and 13 last.  When you start the array, it may show disk13 as unformatted.  If it is the ONLY disk showing unformatted, and it was the one with the bad blocks, then you can press the button to format it.  BUT DO NOT WRITE ANY NEW FILES TO IT (or delete any files, or modify any files)!!

 

Once all the files are recovered that can be, we'll re-assign the parity disk and it will automatically re-calculate itself.

 

HOW DO I ASSIGN THE PARITY DISK?

 

Joe L.

[Joe L.]

Joe L, still confused and would love to talk to you by phone, I will rush home if you say yes.  My questions are ALL CAP

 

Step by step.

 

First, get as much of your array up and sane as possible.   Let's un-assign disk8 and the parity disk.  (that will ensure you do not try to rebuild from parity)

 

HOW DO I DO THIS ( I AM ASSUMING THERE IS A WAY TO DO THIS FROM THE CONSOLE)? If not do I do this from the //tower/main page?

I do not know an easy way to un-assign from the command line, so you must use the default unRAID web-interface.  Go to the devices web-page, un-assign the parity drive and disk8.

Then, let's set a new disk configuration by typing

 

initconfig

and answering "Yes" to its prompt.

 

When you then refresh the management screen all should be blue.   Then start the array.  It will not have a parity disk, so all the others should mount and no re-construction will occur.   Do NOT write and new files to the data disks in the array.  Not until you recover the deleted files.

 

If you have already run

"--rebuild-tree -S"

on a disk, and it was successful, then you can write to it.

 

AT THE END, ARE YOU STILL GOING TO WANT ME TO DO THE --rebuild-tree -S on all 14 disks, one last time?

Only on those you did not already do it on.

 

You can look around and see what has happened.  We'll deal with disk8 and 13 last.   When you start the array, it may show disk13 as unformatted.  If it is the ONLY disk showing unformatted, and it was the one with the bad blocks, then you can press the button to format it.   BUT DO NOT WRITE ANY NEW FILES TO IT (or delete any files, or modify any files)!!

 

Once all the files are recovered that can be, we'll re-assign the parity disk and it will automatically re-calculate itself.

 

HOW DO I ASSIGN THE PARITY DISK?

The same way you did initially,... Stop the array using the web-managemnt page, go to the "devices" link, re-assign parity (using the correct drive of course)

 

[Roger]

Joe L, I think this makes sense, I'm going to go home and try and do it.  Any chance I could call you on the phone, or give you my number?

[Joe L.]

Joe L, I think this makes sense, I'm going to go home and try and do it.  Any chance I could call you on the phone, or give you my number?

sorry, but no.  too many other commitments for my time.  Right now, I too am at work... responding from a personal PC rather than the work PC.

[Roger]

Joe L, did exactly what you said, I now have parity not assigned, disk 8 not assigned, the array started, all disks are green. Disk 13 does not show as unformatted.  Now what?

[Joe L.]

Joe L, did exactly what you said, I now have parity not assigned, disk 8 not assigned, the array started, all disks are green. Disk 13 does not show as unformatted.  Now what?

That is great news...

 

Now, check to see what files are on disk13.  (enable disk shares for now, disable user-shares if you like, that way you can browse the disk shares from window's explorer)

[Roger]

Joe L, I can now fully access all drives, except of course Drive 8.  I see lots of files on Disk1 in the lost+found.  Most of them look like pictures, maybe a couple of movies.  I have written nothing, moved nothing copied nothing to or from anywhere.  Oddly, Disk 13, looks to the eye correct, all directories and files are in there.  Disk 7, for some odd reason, does not have anything in it, but; I have only run the rebuild on Disk1, Disk3, and Disk 13, but; obviously it failed on Disk13.

 

Waiting for what to do next.

 

Roger

[Roger]

Joe L, user shares are working, everything looks there.  Pretty decent lost+found on Disk1, nothing seems wrong on Disk3 or Disk13, can't explain it.

[Roger]

Joe L, trying to get all my questions out for step by step when you are unavailable later.  At some point when I go back to rebuild the rest of the disks, can I just stop the array with the web browser, and then put my command in as usual? Looks like it's taking about 8 hours per disk.  For now, I'm going to do nothing, until I here from you about what is next.

 

Roger

[Joe L.]

Joe L, user shares are working, everything looks there.  Pretty decent lost+found on Disk1, nothing seems wrong on Disk3 or Disk13, can't explain it.

Disk13 has corruption in the file system somewhere...   But for now be thankful it looks ok.

 

Now, you can run

--rebuild-tree -S

on disk8   (since it is not yet assigned to the array, you'll need to run it on the /dev/sdX1 device)