March 18, 201115 yr Well, here's what's happened so far: 1. I went to replace a 750GB hard drive (disk20) in my array (that 750GB hard drive is gone for good) 2. I put a new 1TB drive in and started rebuild 3. Woke up in the morning and saw a bunch of error messages in my syslog that indicated disk 2 had failed (this started about 4 hours into the rebuild). (In my defense I had run a successful parity check just 8 days ago. ) Mar 17 05:42:25 saraswati kernel: ata12: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Mar 17 05:42:25 saraswati kernel: ata12: status=0x41 { DriveReady Error } Mar 17 05:42:25 saraswati kernel: ata12: error=0x04 { DriveStatusError } Mar 17 05:42:25 saraswati kernel: ata12: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Mar 17 05:42:25 saraswati kernel: ata12: status=0x41 { DriveReady Error } Mar 17 05:42:25 saraswati kernel: ata12: error=0x04 { DriveStatusError } Mar 17 05:42:25 saraswati kernel: ata12: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Mar 17 05:42:25 saraswati kernel: ata12: status=0x41 { DriveReady Error } Mar 17 05:42:25 saraswati kernel: ata12: error=0x04 { DriveStatusError } Mar 17 05:42:25 saraswati kernel: ata12: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Mar 17 05:42:25 saraswati kernel: ata12: status=0x41 { DriveReady Error } Mar 17 05:42:25 saraswati kernel: ata12: error=0x04 { DriveStatusError } Mar 17 05:42:25 saraswati kernel: ata12: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00 Mar 17 05:42:25 saraswati kernel: ata12: status=0x41 { DriveReady Error } Mar 17 05:42:25 saraswati kernel: ata12: error=0x04 { DriveStatusError } Mar 17 05:42:25 saraswati kernel: sd 2:0:5:0: [sdm] Result: hostbyte=0x00 driverbyte=0x08 Mar 17 05:42:25 saraswati kernel: sd 2:0:5:0: [sdm] Sense Key : 0xb [current] [descriptor] Mar 17 05:42:25 saraswati kernel: Descriptor sense data with sense descriptors (in hex): Mar 17 05:42:25 saraswati kernel: 72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00 Mar 17 05:42:25 saraswati kernel: 00 00 00 4f Mar 17 05:42:25 saraswati kernel: sd 2:0:5:0: [sdm] ASC=0x0 ASCQ=0x0 Mar 17 05:42:25 saraswati kernel: sd 2:0:5:0: [sdm] CDB: cdb[0]=0x28: 28 00 32 9f ff 9f 00 02 00 00 Mar 17 05:42:25 saraswati kernel: end_request: I/O error, dev sdm, sector 849346463 Mar 17 05:42:25 saraswati kernel: md: disk2 read error Mar 17 05:42:25 saraswati kernel: handle_stripe read error: 849346400/2, count: 1 Mar 17 05:42:25 saraswati kernel: md: parity incorrect: 849346400 Mar 17 05:42:25 saraswati kernel: md: disk2 read error Mar 17 05:42:25 saraswati kernel: handle_stripe read error: 849346408/2, count: 1 Mar 17 05:42:25 saraswati kernel: md: parity incorrect: 849346408 Mar 17 05:42:25 saraswati kernel: md: disk2 read error Mar 17 05:42:25 saraswati kernel: handle_stripe read error: 849346416/2, count: 1 Mar 17 05:42:25 saraswati kernel: md: parity incorrect: 849346416 Mar 17 05:42:25 saraswati kernel: md: disk2 read error Mar 17 05:42:25 saraswati kernel: handle_stripe read error: 849346424/2, count: 1 Mar 17 05:42:25 saraswati kernel: md: parity incorrect: 849346424 Mar 17 05:42:25 saraswati kernel: md: disk2 read error Mar 17 05:42:25 saraswati kernel: handle_stripe read error: 849346432/2, count: 1 Mar 17 05:42:25 saraswati kernel: md: parity incorrect: 849346432 Mar 17 05:42:25 saraswati kernel: md: disk2 read error Mar 17 05:42:25 saraswati kernel: handle_stripe read error: 849346440/2, count: 1 Mar 17 05:42:25 saraswati kernel: md: parity incorrect: 849346440 Mar 17 05:42:25 saraswati kernel: md: disk2 read error Mar 17 05:42:25 saraswati kernel: handle_stripe read error: 849346448/2, count: 1 Mar 17 05:42:25 saraswati kernel: md: parity incorrect: 849346448 Mar 17 05:42:25 saraswati kernel: md: disk2 read error These went on for about, oh, 65MB+ of log file. 4. At this point, my disk20 was green-balled, my disk2 was red-balled (disabled), but browsing the user shares it seemed like the data online was mostly intact (i.e. being reconstructed via parity when factoring disk20). However, of significant concern was the fact that unMenu was reporting that I'd had something like 183 *million* parity corrections. (That is, what did it have to correct?) 5. So, being hopeful and having just picked up 3 Hitachi 2TB drives, I put a new drive in for disk2 and started Data Rebuild. 6. As the rebuild occurred, I started getting a endless stream of errors like this (I don't know the block numbers, but about 0.8% into the rebuild they started): REISERFS error (device md20): vs-5150 search_by_key: invalid format found in block (something). Fsck? REISERFS error (device md20): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2688 150 0x0 SD] In conjunction, the rebuild rate slowed to a crawl and the unRAID UI became very unresponsive. 7. Oops.. so I ran reiserfsck on disk20 and it's calling for --rebuild-tree. I'm running it but actually I'm not hopeful that disk20 is any good even though it was green-balled earlier today (possible unRAID bug saying disk20 is good when it was not?) Anyway, that's where I'm at. However there are a couple things I might be able to do: 1. I still have the 2TB drive that failed. I think I can bring it online manually (not part of the actual unRAID array) for at least a limited time - perhaps long enough to get some or all of the data off. 2. In some ways, I don't care about the data on the lost 750GB hard drive... my disk20 is being used exclusively as share for backups from my network. Yes, I'll be without backups for a short period, but the data should be essentially redundant for now. 3. I have 2 other 2TB hard drives to spare at the moment. Taking into consideration these factors, what is the best way to proceed? Right now I'm running reiserfsck --rebuild-tree on /dev/md20 (disk20) but if it puts out any lost+found I'm pretty much willing to call it invalid. And right now I don't think I trust my parity either in light of #4 in the events above. I think I can presume disks 1 and 3-13 (all my other disks except for 2 and 20) are OK. What I'd also like to ask is: If I can pull data (some or all) from the failed/failing former disk2, should I attempt a disk to array copy or a disk to disk copy? If the latter, what would be the best unRAID procedure to introduce the disk as the replacement disk2 (can I take the array offline, add the new disk2 with data, then rebuild parity?) If I can't pull data from the former disk2, is it worth an attempt do a Data Rebuild? (Especially now given that I'm trying to rebuild disk20 and maybe should have waited for better advice...) Thanks!
March 18, 201115 yr 7. Oops.. so I ran reiserfsck on disk20 and it's calling for --rebuild-tree. I'm running it but actually I'm not hopeful that disk20 is any good even though it was green-balled earlier today (possible unRAID bug saying disk20 is good when it was not?) Green ball means disk is good. Flashing green means a good disk is spun down. Red is bad/disabled. Blue means new disk - not part of array. So all of your disks would be "green-balled" if your array is healthy.
March 18, 201115 yr Author 7. Oops.. so I ran reiserfsck on disk20 and it's calling for --rebuild-tree. I'm running it but actually I'm not hopeful that disk20 is any good even though it was green-balled earlier today (possible unRAID bug saying disk20 is good when it was not?) Green ball means disk is good. Flashing green means a good disk is spun down. Red is bad/disabled. Blue means new disk - not part of array. So all of your disks would be "green-balled" if your array is healthy. So you're saying a disk with a corrupted file-system (according to reiserfsck) immediately after an attempted rebuild of that disk where another disk in the array had suffered a failure (likely during that rebuild process) should be considered OK? You can see why I'm skeptical... Side note: At the moment, I have the failed disk2 mounted (thanks unMenu!!!) and am working on copying the files over to a Windows box in stages. I don't know if they'll all make it.
March 18, 201115 yr A corrupted file system does not indicate a bad disk. Can you put the original 750M disk back in?
March 18, 201115 yr Author A corrupted file system does not indicate a bad disk. Can you put the original 750M disk back in? I wish I could but it's definitely gone for good. I definitely can't rebuild disk2 with the current disk20 (even if green-balled) because with the filesystem issues, the rebuild speed is something like 3K/sec, which I think would work out to about 8-11 days. BTW, thanks for your feedback! I've found that the failed disk2, which I currently have mounted outside the array, is showing an increase in current_pending_sector count (50, up from the prior 2). Checking logs, I see that it'd had a number of times where it had suffered some emask media errors. Wish I'd caught those sooner, though like I said the array had survived parity check without any logged errors about 8 days ago... I just didn't see the media errors that had occurred several days before that.
March 18, 201115 yr Green just means the drive is part of the array, flashing is spun-down and solid is spinning. unRAID doesn't do any testing of the drives to ensure they are fully healthy. By this I mean if the drive is connected and can be written to then it stays green. As an example, you can cancel a parity build part way through and get green indicators even though the parity build never completed and parity is still bad. You can start and then cancel a parity check and unRAID will indicate parity was checked 0 days ago even though it never really was. As for your failure, If you could make a disk copy using dd then it might be possible to recover the 750meg drive. However, at this point you indicate that drive doesn't matter much so try to just plain copy as much content off the disk2 as you can. Put it back to other disks in the array or just another spare drive. Peter
March 18, 201115 yr Author Thanks Peter, that's pretty much what I'm doing now. My plan is to: 1. Give up on disk20 (old and new) 2. Copy data off of failing old disk2 onto Windows system (I've gotten about 800GB of 1.6TB so far... it's gonna take some time but that part I don't to attend while it occurs) 3. Remove old disk2 physically, and new disk2 (there's nothing on it anyway) and new disk20 (which because I ultimately aborted its reiserfsck now has an unreadable superblock) from array 4. Reset my configuration and rebuild parity 5. Preclear new disk2 and new disk20 6. Add new disk2 and new disk20 to array 7. Copy data from Windows system back to new disk2 For the record, the disk2 that failed was acquired in July of 2009 from Dell, is still under warranty, and is a Western Digital WD20EADS.
Archived
This topic is now archived and is closed to further replies.