July 3, 201511 yr Hi guys, Background info: Running a Norco 4220 case 19 4TB drives in array, plus 1 4TB parity and a SSD cache Drive 10 is red balled Replaced with new drive, drive still red balls Sequence of events: [*]Drive 10 red balls after a monthly parity check. Shows a few hundred errors [*]Replaced drive with spare that has been precleared and has no errors [*]Start rebuild. Come back a few hours later to a red balled drive 10 with 428 (or so) errors. Try to run SMART test, cannot (see below) [*]Decide to try again, in case this is a fluke. Stop array, remove, etc to get to where I can re-add the same drive again. Had to reboot before I could re-add drive. [*]Rebuild array. Same results. Drive 10 has about 428 errors [*]Place array into maintenance mode in order to run reiserfsck --check /dev/md10. Results below Latest syslog is attached (I foolishly didn't get a syslog from before I rebooted...sorry). Errors start at around the 02:11 timestamp. Regarding the SMART report, here's what I get when running the command: root@Gallifrey:~# smartctl -a -A /dev/sdt smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: /6:0:5:0 Product: Physical block size: 0 bytes Lowest aligned LBA: 8949 Logical block provisioning enabled, LBPRZ=0 >> Terminate command early due to bad response to IEC mode page A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. Results of reiserfsck --check /dev/md10: root@Gallifrey:~# reiserfsck --check /dev/md10 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md10 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Fri Jul 3 09:41:25 2015 ########### Replaying journal: Done. Reiserfs journal '/dev/md10' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 593982 Internal nodes 3637 Directories 427 Other files 4266 Data block pointers 600329559 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Fri Jul 3 16:33:43 2015 ########### From the reading I've done this sounds like it could be cabling, but I wanted to check in before I go doing anything else. I'm not exactly confident of my abilities to troubleshoot beyond what I've already done, so any insight you guys can offer would be greatly appreciated. syslog-2015-07-03.txt.zip
July 3, 201511 yr Author Ugh. Posted in v6. Should have been in v5. Sorry. EDIT: Thank you for moving the post!
July 4, 201511 yr I have nearly the same setup and haven't touched firmware updates in 4+ years. I would plug "Disk10" into the motherboard and rebuild it there (If I counted right you should have one SATA port left). I agree it is likely cables or a Norco backplane. Easy to troubleshoot those. Could also be your SATA card or motherboard, but less likely. Good luck.
July 7, 201511 yr Author Thanks ohiwiler. That's exactly what I ended up doing. The new disk failed completely. The original managed to rebuild, but now shows as unformatted. So, I ran another reiserfsck --check /dev/md10 and got the below. Now, per this thread, I'm running --rebuild-tree. root@Gallifrey:~# reiserfsck --check /dev/md10 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md10 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Tue Jul 7 01:30:15 2015 ########### Replaying journal: Done. Reiserfs journal '/dev/md10' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. \/ 1 (of 26|/ 1 (of 96//126 (of 129\block 124427444: The level of the node (34576) is not correct, (1) expected the problem in the internal node occured (124427444), whole subtree is skipped / 4 (of 96|/ 95 (of 96\block 789628251: The level of the node (13331) is not correct, (1) expected the problem in the internal node occured (789628251), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 2 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Tue Jul 7 02:28:49 2015 ###########
July 8, 201511 yr Author Looks like --rebuild-tree did the trick. I'm sure I've lost a few (unimportant) media files along the way, but nothing like the loss I was fearing. I'm now running a parity check, and so far so good. So, looks like the sequence of events is this: [*]Disk 10 red balls after a monthly parity check. Shows a few hundred errors [*]Mistakenly assume the disk is at fault [*]Replaced disk with spare that has been precleared and has no errors [*]"New" disk 10 won't rebuild. Not in its bay, not directly connected to the one remaining motherboard slot. Refuses. [*]Replace directly connected "new" disk 10 with original disk 10. [*]Rebuild successful. Conclusion: The original disk was fine. The replaced disk was faulty. The connection is bad somehow in the bay (backplane? Cable? No idea). The replaced drive did actually have a fault - or it gained a fault during the first write in the bad bay. I hope to update the thread with a successful parity check within the next day and a half, but in the meantime my question: How can I determine what component is at fault? Norco backplane? Cable to the backplane? Something else? I'm now officially at my technical limit. Complication is that my array is completely full, so no spare bays to use.
July 9, 201511 yr Author Update: Parity check completed fine. Array is back up and functioning normally, but I'm still left with the question of how to figure out what component is faulty.
July 9, 201511 yr Update: Parity check completed fine. Array is back up and functioning normally, but I'm still left with the question of how to figure out what component is faulty. To check the new 'bad' disk, test it with the preclear script. You will find it here: http://lime-technology.com/forum/index.php?topic=2817.0
July 9, 201511 yr I'm happy to see that you have things 'mostly' figured out. Those are the frustrating problems, that pop up and then can't be replicated. I've had a couple over the years. Update: Parity check completed fine. Array is back up and functioning normally, but I'm still left with the question of how to figure out what component is faulty. I'd replace the SFF-8087 cable. They are cheap, easy to swap out and it wouldn't hurt to have a spare.
July 10, 201510 yr Author Thanks Frank, but I pre cleared the new drive and it was just fine. ohwiler, thanks. I'll order a couple of SFF-8087 cables. Worth trying that first. However, you're right in the frustrating feeling that comes from not knowing what the actual problem is. Though it does give me the impetus to get that 24-bay Norco case for array #2.
Archived
This topic is now archived and is closed to further replies.