So while troubleshooting another issue (http://lime-technology.com/forum/index.php?topic=24907.0), I ran into a new one.  I'm sure nobody here can relate.

 

The tower main webpage gave me a red ball for one of my drives, and labeled it as UNFORMATTED.

 

Since this drive was working fairly recently, I thought I would google the forums to see if anything obvious would strike out.

 

Here is the syslog file: https://dl.dropbox.com/u/18394268/syslog-2012-12-26-badBlocks.txt

 

I tried to run

reiserfsck --check /dev/md6

but it just told me it looks like a hardware error and to replace the drive. 

 

The problem as I already stated earlier is that I was troubleshooting another issue, so my parity was not up to snuff, so replacing the drive and recovering from parity isn't really an option.

 

I plugged in the drive directly into the motherboard instead of going through the controller, and nothing (nothing different).

 

Tried rebooting as well...

 

Going to bed having queued up

 badblocks -svn /dev/sdn

 

I'm ok with losing the drive and rebuilding parity, but I would like to be able to recover my data ...any suggestions?

Woke up this morning, and saw that the test was only 2% complete, so I stopped it (CTRL-C).  I looked at the syslog, saw that it reported blocks 2 and 16 as being reported as bad and so I ran

 

 badblocks -b 4096 -svn -o /boot/badblocks_out_sdn.txt /dev/sdn 20 1

 

This came back as 0 bad blocks found.

 

Any other suggestions?

Ok, here things get weird, and I think I may have found a legitimate bug.

 

When I plugged in my second/backup unRAID usb (after transferring over config data and reverting to an original go file), everything fires up just fine.  No bad blocks, no anything.

 

Found someone with what appears to be the exact same issue, but from several years ago.  After extensive troubleshooting, there didn't appear to be any kind of resolution unfortunately

 

Here is the link to the thread: http://lime-technology.com/forum/index.php?topic=5060.0

Attach a syslog. See here: http://lime-technology.com/forum/index.php?topic=15385.15

Attach a syslog. See here: http://lime-technology.com/forum/index.php?topic=15385.15

 

I linked to a syslog on my first post, and here is a link to a newer one (same issue though).

 

http://dl.dropbox.com/u/18394268/syslog-2012-12-27.txt

Look into rip linux and ddrescue. try and copy to another drive. Then do your recovery from there.

I was able to recover 99.9% of a failed drive with bad sectors by copying from one drive forward, then copying backward using the bad secttor map from the first step.

I reran the backward copy over and over until it was down to all but 1 final sector.

 

 

I was able to recover all of the 1tb drive down to 1 sector.

 

 

Do allot of reading on the dd rescue first. I don't remember the syntax anymore. I don't have access to my shells.

I can't help you more since I no longer have an unraid system since hurricane sandy.

 

 

See this post, do a search on the board also.

http://lime-technology.com/forum/index.php?topic=16734.msg153098#msg153098

Look into rip linux and ddrescue. try and copy to another drive. Then do your recovery from there.

I was able to recover 99.9% of a failed drive with bad sectors by copying from one drive forward, then copying backward using the bad secttor map from the first step.

I reran the backward copy over and over until it was down to all but 1 final sector.

 

 

I was able to recover all of the 1tb drive down to 1 sector.

 

 

Do allot of reading on the dd rescue first. I don't remember the syntax anymore. I don't have access to my shells.

I can't help you more since I no longer have an unraid system since hurricane sandy.

 

 

See this post, do a search on the board also.

http://lime-technology.com/forum/index.php?topic=16734.msg153098#msg153098

 

Thanks for the help and the shove in the right direction.  Sucks so much about the loss from the weather

 

I'm heading out of town for the weekend actually, but I'll dive into this as soon as I get back.

Look into rip linux and ddrescue. try and copy to another drive. Then do your recovery from there.

I was able to recover 99.9% of a failed drive with bad sectors by copying from one drive forward, then copying backward using the bad secttor map from the first step.

I reran the backward copy over and over until it was down to all but 1 final sector.

 

 

I was able to recover all of the 1tb drive down to 1 sector.

 

 

Do allot of reading on the dd rescue first. I don't remember the syntax anymore. I don't have access to my shells.

I can't help you more since I no longer have an unraid system since hurricane sandy.

 

 

See this post, do a search on the board also.

http://lime-technology.com/forum/index.php?topic=16734.msg153098#msg153098

 

ok, finally made it back home and had a good look through the links, but I must say I think I'm way over my head as I'm still a bit of a CLI/linux noob (don't mind going into the CLI, but generally need fairly clear instructions). 

 

I'll keep trying at this and see if I made any headway.

I don't understand why when a drive shows up as bad folks insist on trying to force it otherwise. Swap the drive to something you can rely on, protect the data, and then spend time playing with the suspect drive. Unless you have indications of a loose cable or bad controller swap the drive and work on the bad one offline. Suppose this drive really is bad? What has running these utils done to the data and parity? You had protection, do you now? Once a drive goes bad you're on a ticking clock hoping a second one doesn't go bad and destroy two drives worth of data. Why second guess the software? If the drive really isn't bad then the time to figure this out is after you've swapped in a new one and rebuilt parity. So far I've yet to have a drive show as dead and not have something wrong with it. It can certainly happen that something weird causes the software to make an error but the chances of that are far less likely than it being right. Why gamble at all?

While I was away for the weekend, I realized I was actually in a perfect position to preserve the data from following the other thread, as I was able to mount the "bad" drive in maintenance mode.

 

What I did was start the array in maintenance mode, and putty into the system.

 

mkdir /mnt/disk6
mount -t reiserfs -o noatime,nodiratime /dev/sdf1 /mnt/disk6

mkdir /mnt/disk1
mount -t reiserfs -o noatime,nodiratime /dev/sdb1 /mnt/disk1

mc

 

Disk 6 was the disk that was reporting bad blocks, however it was mounting just fine like this.  Since I had enough free space on disk 1, I mounted that and just transferred the data overnight.  I've now pulled the disk from the array, I'm rebuilding parity, and I'll be running the preclear script on the "bad" disk.

 

 

I don't understand why when a drive shows up as bad folks insist on trying to force it otherwise. Swap the drive to something you can rely on, protect the data, and then spend time playing with the suspect drive. Unless you have indications of a loose cable or bad controller swap the drive and work on the bad one offline. Suppose this drive really is bad? What has running these utils done to the data and parity? You had protection, do you now? Once a drive goes bad you're on a ticking clock hoping a second one doesn't go bad and destroy two drives worth of data. Why second guess the software? If the drive really isn't bad then the time to figure this out is after you've swapped in a new one and rebuilt parity. So far I've yet to have a drive show as dead and not have something wrong with it. It can certainly happen that something weird causes the software to make an error but the chances of that are far less likely than it being right. Why gamble at all?

 

You're making a few assumptions here which I don't believe to be accurate.

 

1) The drive being reported as bad is being actually bad.

2) That I have another spare drive sitting around to swap in its place.

3) I needed to preserve parity.

 

I had many reasons to second guess the software, as I explained in the original post.  This error occurred while troubleshooting a different issue (slow parity speed).  The timing was less than believable, and considering both badblocks and the smart report were not showing there was anything wrong with the drive.

 

Furthermore, given that I was able to mount the disks without any issue using

mount -t reiserfs -o noatime,nodiratime /dev/sdf1 /mnt/disk6 

, I'm still not convinced there is anything actually wrong with the drive at all. 

considering both badblocks and the smart report were not showing there was anything wrong with the drive.

I wouldn't completely trust badblocks (at least not alone anyway which I realize you aren't) based on this thread.

Slight update to this thread.  After I cleared out the drive, I ran the preclear script on it, which it just passed. 

 

Is unread unraid (had an autocorrect mistake on my phone) just incorrectly reading bad blocks on drives?

Slight update to this thread.  After I cleared out the drive, I ran the preclear script on it, which it just passed. 

 

Is unread just incorrectly reading bad blocks on drives?

Unfortunately I can only point you at the thread.  Hopefully someone else can answer your question.

Next time you can do a badblocks in read-only mode to read all the sectors.

Then do a smartctl -t long test.  Let it run to the end.

 

Check the smart reports.

 

 

Frankly, I would not mount either of the drives outside the array until I had somehow recovered the data via the normal array or rebuilt the failed drive onto a spare drive.

 

Once you mount a drive outside of parity protection, the parity protection is no good. Even if you mount the drive readonly, there are fields in the superblock that are still updated.  Chances are you will be OK.

So supposedly, you could mount the failed drive readonly and an external USB drive, then copy the data.

 

If SMART reports any form of unrecoverable, LBA's or HEALTH test failures, the drive is bad, assume it is bad and recover what you can.

 

Once recovered, you can do other things to try and recover a failed LBA to put the drive back into service.

 

In the past when SMART reported an unrecoverable read at a specific LBA, I would get my data off the drive, then do a 4 pass badblocks in write mode. I have recovered a number of drives back to usable state. This forced the firmware to re-assign the block and in future reads/writes the failed block will not be accessed directly.