January 30, 201313 yr Hi, Array looks like this.. Disk3 is in the process of being swapped. The new disk(3) had just been precleared and added to the array as an unformatted disk. I started the array and immediately Disk2 failed with 4 errors. Since then, I saved the syslog, checked cabling and reboot. Attached syslog for reference. Note: Attempted start array at 14:52 today. Looking for advice on least painful way to proceed. syslog-2013-01-30.zip
January 30, 201313 yr Author Smart Report for Disk2. Jan 30 14:52:09 Tower kernel: ata4.00: irq_stat 0x08000000, interface fatal error Jan 30 14:52:09 Tower kernel: ata4: SError: { Handshk } Jan 30 14:52:09 Tower kernel: ata4.00: failed command: WRITE DMA Hmm... wiki says Drive Interface issue, cable problem likely. Will double check all the cabling. smart_report.txt
January 30, 201313 yr Open system and recheck cabling... If all looks ok then try and plug over the failing drive to another sata connector to see if that helps.. Also: added anything to the system lately that causes to psu to need to deliver more juice ?
January 30, 201313 yr Author @Helmonder - nothing added to system lately, just the drive swap. Rechecked all cables, its still possible the cable for Disk2 was disturbed whilst swapping out the new Disk3. Disk2(disabled, red-balled) seems to be fine. I can mount it an see all the files, reiserfsck runs through with no errors, SMART is a pass. Upon starting the array now, it comes up as "unformatted". I need this, and the parity, to rebuild the replaced Disk3. I did think of replacing the old Disk3, which is fine, and using it to rebuild Disk2, BUT the old Disk3 is smaller than the new Disk3(500GB < 2TB), and this causes problems too. Summary, replaced disk(3) with larger one, planning to rebuild from parity, but seems likely I "bumped" the cable on an adjacent disk, resulting in it being "red-balled" on array start. Still a little unsure on the safest way forward.
January 30, 201313 yr Author Found reason for "unformatted", the mount is failing to read the superblock. Jan 30 20:31:53 Tower emhttp: shcmd (99): set -o pipefail ; mount -t reiserfs -o user_xattr,acl,noatime,nodiratime /dev/md2 /mnt/disk2 |$stuff$ logger (Other emhttp) Jan 30 20:31:53 Tower kernel: REISERFS warning (device md2): sh-2006 read_super_block: bread failed (dev md2, block 2, size 4096) (Minor Issues) Jan 30 20:31:53 Tower kernel: REISERFS warning (device md2): sh-2006 read_super_block: bread failed (dev md2, block 16, size 4096) (Minor Issues) Jan 30 20:31:53 Tower kernel: REISERFS warning (device md2): sh-2021 reiserfs_fill_super: can not find reiserfs on md2 (Minor Issues) Jan 30 20:31:53 Tower logger: mount: wrong fs type, bad option, bad superblock on /dev/md2, root@Tower:/mnt# reiserfsck --check /dev/sdg1 Will read-only check consistency of the filesystem on /dev/sdg1 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Wed Jan 30 20:51:50 2013 ########### Replaying journal: Done. Reiserfs journal '/dev/sdg1' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 138769 Internal nodes 858 Directories 2493 Other files 9942 Data block pointers 139267407 (9134 of them are zero) Safe links 0 ########### reiserfsck finished at Wed Jan 30 20:55:47 2013 ########### Seems I am not the first to have this problem, http://lime-technology.com/forum/index.php?topic=10375.msg98860#msg98860... replacing old Disk3 and "initconfig" may be the only way forward.
January 30, 201313 yr I know the reiserfsck seems to find nothing, but if the disk will still not mount I've seen one case where a --rebuild-tree fixed it. Before you do anything, can you run the following command and post the output? dd if=/dev/sdg count=195 | od -c -A d | sed 30q
February 1, 201313 yr Author @ Joe L. - output is below. root@Tower:~# dd if=/dev/sdg count=195 | od -c -A d | sed 30q 195+0 records in 195+0 records out 99840 bytes (100 kB) copied, 0.0250573 s, 4.0 MB/s 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000448 \0 \0 203 \0 \0 \0 @ \0 \0 \0 p 210 340 350 \0 \0 0000464 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0000496 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 U 252 0000512 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0098304 016 021 034 035 017 376 245 024 224 H F 002 022 \0 \0 \0 0098320 \0 \0 \0 \0 \0 \0 \0 \0 004 \0 \0 261 025 331 ! 0098336 204 003 \0 \0 036 \0 \0 \0 \0 \0 \0 \0 \0 020 314 003 0098352 002 \0 001 \0 R e I s E r 2 F s \0 \0 \0 0098368 003 \0 \0 \0 005 \0 9 : 002 \0 \0 \0 l 006 \0 \0 0098384 001 \0 \0 \0 265 N % 023 \v 342 E q 271 a 362 ` 0098400 251 003 C 257 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0098416 \0 \0 \0 \0 003 \0 036 \0 # 356 \b Q \0 N 355 \0 0098432 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0098496 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 \0 \0 \0 0098512 G 3 \0 \0 230 2 \0 \0 344 2 \0 \0 230 2 \0 \0 0098528 344 2 \0 \0 230 2 \0 \0 344 2 \0 \0 230 2 \0 \0 * 0098912 344 2 \0 \0 [ + \0 \0 d + \0 \0 [ + \0 \0 0098928 d + \0 \0 [ + \0 \0 d + \0 \0 [ + \0 \0 * 0099840 I have rebuilt my array over the last two days without losing any data. Backed up data to external disks(Disk2 & Disk3) initconfig'ed the array Replaced original Disk3 Disk2 mounted to array fine(!) Rebuilt parity Upgraded Disk3 and rebuilt from parity extensive data checks show all good. Lesson learnt(the hard way): be DAMN careful with cabling when swapping disks!
February 1, 201313 yr Glad you are back to normal. If the file-system starts on sector 63 you would see (for disks <= 2.2TB) : 0097840 220 \0 002 \0 R e I s E r 2 F s \0 \0 \0 If the file-system starts on sector 64 you would see (for disks <= 2.2TB): 0098352 002 \0 002 \0 R e I s E r 2 F s \0 \0 \0 Note, the starting address is different. Your "dd" output shows a drive with the file system starting on sector 64.
Archived
This topic is now archived and is closed to further replies.