Even though the drive is emulated you can still run reiserfsck on it  (It will fix the emulated drive)

Then, when you put a good drive into place, it should rebuild onto it.

 

i also thought about it but i think that is the last resort because fixing file system depends on how bad the corruption in file system is

some, or in worst case all, data might be lost forever, that is point of no return once the trigger is pulled.

 

Anyway, JustinAiken, if this is the only option i guess that is it. Do a read-only check first see how it goes as instruction in following link before you decide to pull this trigger.

 

http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

 

But there is no disk2 to run a filecheck on... even when I remove the disc from the array, and start it, so it should be emulating... if I type mount in a telnet window, there's just no disc2...

We understand.

 

Type

reiserfsck --check /dev/md2

The disk would have to be un-mounted to run the file-system check anyway.  Report back on the results before you run any "fix" options.

 

But there is no disk2 to run a filecheck on... even when I remove the disc from the array, and start it, so it should be emulating... if I type mount in a telnet window, there's just no disc2...

 

You can telnet and "cd /dev" to find out if there is a md2 entry there.

 

As Joe point out, your unRAID should still remember this disk is part of RAID. whatever you are writing on this disk you are altering parity in parity disk, by fixing file system through many disk writes that in reality fixing parity, later on when re-mounting this disk, the data re-constructed from all survived disks should be good and pass file system checking.

But there is no disk2 to run a filecheck on... even when I remove the disc from the array, and start it, so it should be emulating... if I type mount in a telnet window, there's just no disc2...

 

You can telnet and "cd /dev" to find out if there is a md2 entry there.

 

As Joe point out, your unRAID should still remember this disk is part of RAID. whatever you are writing on this disk you are altering parity in parity disk, by fixing file system through many disk writes that in reality fixing parity, later on when re-mounting this disk, the data re-constructed from all survived disks should be good and pass file system checking.

 

http://pastebin.com/kpVpukua

 

Also, I was going to see which disk I still had HPA on (just to see... wasn't going to remove it if it was part of the array), but I can't figure out how to find which on it was... I was able to find and remove the other one, but I can't figure out which one still has it; probably 'cause I have a bad stomach bug and have to keep running to the bathroom every couple of minutes...

 

So yeah, besides this emulated md2 I just ran the scan on, I have 2 2TB drive that should be identical; when one failed, I tried to rebuild it after getting it to show up again, and when then didn't work, I tried preclearing a new 2TB drive, which also had the rebuild fail.

 

 

 

My server was in a mess... 2 drives with HPA, 3 Seagates with the dreaded CC34, a mystery disc... thanks for helping me sort this out guys.

 

I'm going to flash my 2 Seagates to CC35 after I get these problems sorted out... time to make my machine a bit more reliable (one of the Seagates already bit it, but unRaid successfully restored it onto a 2TB while I wait for Seagate to process the RMA... )

The file-system check found corruption that it can fix if you use the --rebuild-tree option

 

so....

 

reiserfsck --rebuild-tree /dev/md2

 

is your next step.  This could take a very long time (many hours) on a large drive, so don't worry if it takes its time.

 

Once it is done, you can re-assign a disk to that slot in the array and have the fixed contents written to it.

 

As far as the HPA goes.  I'm not sure you realize, but if you remove the HPA, effectively, as far as unRAID is concerned, you replaced the disk with a different larger disk.  If you already have a failed disk, you basically then have two failed disks and you will lose data unless you know exactly what you are doing. 

 

Please do not mess with the HPA until after you get your disk2 back online and parity re-calculated.

The file-system check found corruption that it can fix if you use the --rebuild-tree option

so....

reiserfsck --rebuild-tree /dev/md2

is your next step.   This could take a very long time (many hours) on a large drive, so don't worry if it takes its time.

 

It is better to re-run read-only checking for verification purpose after finish tree-rebuild

because some checking has been skipped at 1st check.

 

-------------------------------------------------------------------------------------------------------------

Checking internal tree.. \block 234818035: The level of the node (23026) is not correct, (4) expected

the problem in the internal node occured (234818035), whole subtree is skipped

finished

Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs.

Bad nodes were found, Semantic pass skipped

1 found corruptions can be fixed only when running with --rebuild-tree

 

This could take a very long time (many hours) on a large drive, so don't worry if it takes its time.

 

 

Very long time!!

 

 

It is better to re-run read-only checking for verification purpose after finish tree-rebuild

 

 

Did that, this time the --check came back clean (0 corruptions found)

 

Rebooted the array, disc2 is now emulated... although it looks like I lost around 1TB of video files... oh well, at least I saved 800GB worth!  Looks like I'll have alot of poking around in the lost+found to move everything back...

 

Now then... which HD should I rebuild disc2 onto? The original, or the replacement? Because this was all filesystem related, I'm wondering if the WD EARS drive that was originally in the disc2 slot is actually ok?

 

Now then... which HD should I rebuild disc2 onto? The original, or the replacement? Because this was all filesystem related, I'm wondering if the WD EARS drive that was originally in the disc2 slot is actually ok?

 

If there is no SATA level error on this disk before and SMART result is also good, i think it should be ok. besides, when you put it back, you should go through data reconstruction process again then you will know.

Oh, why do the storage gods hate me so?

 

So I stopped the array and spun down the discs the other day, then right before I go to bed I was going to start the data rebuild (wanted to do it overnight)... just before bed I I assigned my disk to disk2 and clicked data-rebuild... pop!! Drive 3 shows up as 0 degrees C and starts giving me a ton of errors... I tried restarting it, and now disc3 fails to mount; now I've got two bad drives at once.  And drive2 failed right after another had (but it was rebuilt okay at least)... that's 3 drive failures within a week! 

 

Syslog attached.

syslog-2010-09-02.txt

Oh, why do the storage gods hate me so?

 

So I stopped the array and spun down the discs the other day, then right before I go to bed I was going to start the data rebuild (wanted to do it overnight)... just before bed I I assigned my disk to disk2 and clicked data-rebuild... pop!! Drive 3 shows up as 0 degrees C and starts giving me a ton of errors... I tried restarting it, and now disc3 fails to mount; now I've got two bad drives at once.  And drive2 failed right after another had (but it was rebuilt okay at least)... that's 3 drive failures within a week! 

 

Syslog attached.

I'd start looking at something in common. 

 

I'm not saying it cannot happen, but the odds of three drive failures in a week are very slim unless you are in a dusty, smoky, hot, environment and torturing the drives...

 

On the other hand, it is easy for a common disk-controller, drive cage, bad batch of SATA cables, or a power supply issue to cause you to think the drives are failing.

 

 

Okay, I've tried moving the row of drives where drive2 and drive3 failed to a different backplane..

 

Also tried a different power cable.

 

Also tried swapping that row for another row, moving the row with troublesome discs2+3 from a PCIE card to the motherboard... Reassigning the appropriate drives in the devices screen.

 

In everycase, I'm getting the same thing when I turn it on; disk2 is still orange because I was never able to rebuild it, and drive3 is red because it can't mount (Tower emhttp: disk3 mount error: 32)

 

Any ideas what I should do?

Perhaps I should unassign the parity, check the resier and rebuild the tree on my drives 2-3, cut the losses, rebuild the parity, and then always be ready to rebuild a drive THE SECOND it fails...