July 17, 201015 yr Help! My 4.3.3 system has been running flawlessly, with various disk upgrades and the occasional drive failure all handled correctly. Now, I am faced with a problem I can't seem to solve myself. I had a 1.5TB drive fail, so I pulled it and sent it off for RMA. I bought a WD 2TB EARS model to replace it while I wait for the replacement 1.5, which I planned on upgrading one of my smaller drives when it arrives. Now, after installing the 2TB, the array detects the new drive, prompts me to start a data rebuild, and when it is complete, fails repeated parity checks with 100 or so errors. I restarted the system, and it immediately starts a data rebuild cycle on its own. It completed, and once again I start a parity check, and get a failure with less than 100 errors. I restart the system, and once again, it automatically starts a data rebuild cycle. Can anyone give me some troubleshooting ideas? Thanks! BTW, before I pulled the failed drive, I moved its virtual contents to other drives in the system, so it appears empty on the disk share, but shows 3,636,976 used in the disk status, which I assume is formatting. I also jumpered the EARS drive on 7-8, I read somewhere that was needed. Here is the current syslog. syslog.txt
July 17, 201015 yr You are getting CRC errors communicating with the drive. Replace and/or re-seat the SATA cable to it. (most likely suspect) Jul 17 12:38:47 AVFILES kernel: ata8.00: exception Emask 0x10 SAct 0x2 SErr 0x780100 action 0x2 Jul 17 12:38:47 AVFILES kernel: ata8.00: irq_stat 0x08000000 Jul 17 12:38:47 AVFILES kernel: ata8: SError: { UnrecovData 10B8B Dispar BadCRC Handshk } Jul 17 12:38:47 AVFILES kernel: ata8.00: cmd 60/88:08:3f:8f:af/01:00:04:00:00/40 tag 1 ncq 200704 in Jul 17 12:38:47 AVFILES kernel: res 40/00:0c:3f:8f:af/00:00:04:00:00/40 Emask 0x10 (ATA bus error) Jul 17 12:38:47 AVFILES kernel: ata8.00: status: { DRDY } Jul 17 12:38:47 AVFILES kernel: ata8: hard resetting link Jul 17 12:38:48 AVFILES kernel: ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 17 12:38:48 AVFILES kernel: ata8.00: configured for UDMA/133 Jul 17 12:38:48 AVFILES kernel: ata8: EH complete Jul 17 12:38:48 AVFILES kernel: sd 8:0:0:0: [sdi] 3907029168 512-byte hardware sectors (2000399 MB) Jul 17 12:38:48 AVFILES kernel: sd 8:0:0:0: [sdi] Write Protect is off Jul 17 12:38:48 AVFILES kernel: sd 8:0:0:0: [sdi] Mode Sense: 00 3a 00 00 Jul 17 12:38:48 AVFILES kernel: sd 8:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jul 17 12:53:54 AVFILES in.telnetd[2645]: connect from 192.168.0.3 (192.168.0.3)
July 17, 201015 yr Author The cable was not disturbed, as it is connected to the dock. I reseated the drive in the dock, sliding it in and out multiple times so as to clean the connections as much as possible. I am now booting the system back up, and will attach a syslog. I don't see that message in this log, but it just came up and started another rebuild. I'll let you know how it works this time. syslog.txt
July 18, 201015 yr Author Another "successful" rebuild, but it immediately started rebuilding again after I stopped and rebooted the machine. I didn't capture a syslog before I rebooted, I'll do that after this auto rebuild is done before I do anything else. Update... The errors start sometime into the rebuild, this portion of the syslog is from the most recent attempt. I am no longer using the same cable, and the drive is not in the backplane, but directly cabled. Jul 18 21:01:24 AVFILES kernel: ata5.00: exception Emask 0x10 SAct 0x3 SErr 0x280100 action 0x2 frozen Jul 18 21:01:24 AVFILES kernel: ata5.00: irq_stat 0x08000000, interface fatal error Jul 18 21:01:24 AVFILES kernel: ata5: SError: { UnrecovData 10B8B BadCRC } Jul 18 21:01:24 AVFILES kernel: ata5.00: cmd 60/00:00:b7:bf:86/04:00:00:00:00/40 tag 0 ncq 524288 in Jul 18 21:01:24 AVFILES kernel: res 40/00:08:b7:c3:86/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:01:24 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:01:24 AVFILES kernel: ata5.00: cmd 60/08:08:b7:c3:86/01:00:00:00:00/40 tag 1 ncq 135168 in Jul 18 21:01:24 AVFILES kernel: res 40/00:08:b7:c3:86/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:01:24 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:01:24 AVFILES kernel: ata5: hard resetting link Jul 18 21:01:25 AVFILES kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:01:25 AVFILES kernel: ata5.00: configured for UDMA/133 Jul 18 21:01:25 AVFILES kernel: ata5: EH complete Jul 18 21:01:25 AVFILES kernel: sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors (1000205 MB) Jul 18 21:01:25 AVFILES kernel: sd 5:0:0:0: [sdf] Write Protect is off Jul 18 21:01:25 AVFILES kernel: sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 Jul 18 21:01:25 AVFILES kernel: sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jul 18 21:02:04 AVFILES ntpd[2132]: synchronized to 153.16.4.131, stratum 2 Jul 18 21:02:09 AVFILES kernel: ata5.00: exception Emask 0x10 SAct 0x1 SErr 0x280100 action 0x2 frozen Jul 18 21:02:09 AVFILES kernel: ata5.00: irq_stat 0x08000000, interface fatal error Jul 18 21:02:09 AVFILES kernel: ata5: SError: { UnrecovData 10B8B BadCRC } Jul 18 21:02:09 AVFILES kernel: ata5.00: cmd 60/f8:00:8f:cc:a0/03:00:00:00:00/40 tag 0 ncq 520192 in Jul 18 21:02:09 AVFILES kernel: res 40/00:00:8f:cc:a0/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:02:09 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:02:09 AVFILES kernel: ata5: hard resetting link Jul 18 21:02:11 AVFILES kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:02:11 AVFILES kernel: ata5.00: configured for UDMA/133 Jul 18 21:02:11 AVFILES kernel: ata5: EH complete Jul 18 21:02:11 AVFILES kernel: sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors (1000205 MB) Jul 18 21:02:11 AVFILES kernel: sd 5:0:0:0: [sdf] Write Protect is off Jul 18 21:02:11 AVFILES kernel: sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 Jul 18 21:02:11 AVFILES kernel: sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jul 18 21:02:43 AVFILES kernel: ata5.00: exception Emask 0x10 SAct 0x1 SErr 0x280100 action 0x2 frozen Jul 18 21:02:43 AVFILES kernel: ata5.00: irq_stat 0x08000000, interface fatal error Jul 18 21:02:43 AVFILES kernel: ata5: SError: { UnrecovData 10B8B BadCRC } Jul 18 21:02:43 AVFILES kernel: ata5.00: cmd 60/f8:00:07:c6:b3/03:00:00:00:00/40 tag 0 ncq 520192 in Jul 18 21:02:43 AVFILES kernel: res 40/00:00:07:c6:b3/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:02:43 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:02:43 AVFILES kernel: ata5: hard resetting link Jul 18 21:02:45 AVFILES kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:02:45 AVFILES kernel: ata5.00: configured for UDMA/133 Jul 18 21:02:45 AVFILES kernel: ata5: EH complete Jul 18 21:02:45 AVFILES kernel: sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors (1000205 MB) Jul 18 21:02:45 AVFILES kernel: sd 5:0:0:0: [sdf] Write Protect is off Jul 18 21:02:45 AVFILES kernel: sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 Jul 18 21:02:45 AVFILES kernel: sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jul 18 21:03:33 AVFILES kernel: ata8.00: exception Emask 0x10 SAct 0x2 SErr 0x780100 action 0x2 Jul 18 21:03:33 AVFILES kernel: ata8.00: irq_stat 0x08000000 Jul 18 21:03:33 AVFILES kernel: ata8: SError: { UnrecovData 10B8B Dispar BadCRC Handshk } Jul 18 21:03:33 AVFILES kernel: ata8.00: cmd 60/c8:08:3f:72:d0/02:00:00:00:00/40 tag 1 ncq 364544 in Jul 18 21:03:33 AVFILES kernel: res 40/00:0c:3f:72:d0/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:03:33 AVFILES kernel: ata8.00: status: { DRDY } Jul 18 21:03:33 AVFILES kernel: ata8: hard resetting link Jul 18 21:03:34 AVFILES kernel: ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:03:34 AVFILES kernel: ata8.00: configured for UDMA/133 Jul 18 21:03:34 AVFILES kernel: ata8: EH complete Jul 18 21:03:34 AVFILES kernel: sd 8:0:0:0: [sdi] 3907029168 512-byte hardware sectors (2000399 MB) Jul 18 21:03:34 AVFILES kernel: sd 8:0:0:0: [sdi] Write Protect is off Jul 18 21:03:34 AVFILES kernel: sd 8:0:0:0: [sdi] Mode Sense: 00 3a 00 00 Jul 18 21:03:34 AVFILES kernel: sd 8:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jul 18 21:09:06 AVFILES kernel: ata5: limiting SATA link speed to 1.5 Gbps Jul 18 21:09:06 AVFILES kernel: ata5.00: exception Emask 0x50 SAct 0x3 SErr 0x280900 action 0x6 frozen Jul 18 21:09:06 AVFILES kernel: ata5.00: irq_stat 0x08000000, interface fatal error Jul 18 21:09:06 AVFILES kernel: ata5: SError: { UnrecovData HostInt 10B8B BadCRC } Jul 18 21:09:06 AVFILES kernel: ata5.00: cmd 60/00:00:8f:78:95/04:00:01:00:00/40 tag 0 ncq 524288 in Jul 18 21:09:06 AVFILES kernel: res 40/00:00:8f:78:95/00:00:01:00:00/40 Emask 0x50 (ATA bus error) Jul 18 21:09:06 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:09:06 AVFILES kernel: ata5.00: cmd 60/08:08:8f:7c:95/01:00:01:00:00/40 tag 1 ncq 135168 in Jul 18 21:09:06 AVFILES kernel: res 40/00:00:8f:78:95/00:00:01:00:00/40 Emask 0x50 (ATA bus error) Jul 18 21:09:06 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:09:06 AVFILES kernel: ata5: hard resetting link Jul 18 21:09:08 AVFILES kernel: ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Jul 18 21:09:08 AVFILES kernel: ata5.00: configured for UDMA/133 Jul 18 21:09:08 AVFILES kernel: ata5: EH complete Jul 18 21:09:08 AVFILES kernel: sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors (1000205 MB) Jul 18 21:09:08 AVFILES kernel: sd 5:0:0:0: [sdf] Write Protect is off Jul 18 21:09:08 AVFILES kernel: sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 Jul 18 21:09:08 AVFILES kernel: sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jul 18 21:12:47 AVFILES ntpd[2132]: kernel time sync status change 0001 Jul 18 21:16:51 AVFILES kernel: ata8.00: exception Emask 0x10 SAct 0x3 SErr 0x780100 action 0x2 Jul 18 21:16:51 AVFILES kernel: ata8.00: irq_stat 0x08000000 Jul 18 21:16:51 AVFILES kernel: ata8: SError: { UnrecovData 10B8B Dispar BadCRC Handshk } Jul 18 21:16:51 AVFILES kernel: ata8.00: cmd 60/00:00:07:3e:ac/04:00:02:00:00/40 tag 0 ncq 524288 in Jul 18 21:16:51 AVFILES kernel: res 40/00:0c:07:42:ac/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:16:51 AVFILES kernel: ata8.00: status: { DRDY } Jul 18 21:16:51 AVFILES kernel: ata8.00: cmd 60/a0:08:07:42:ac/01:00:02:00:00/40 tag 1 ncq 212992 in Jul 18 21:16:51 AVFILES kernel: res 40/00:0c:07:42:ac/00:00:02:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:16:51 AVFILES kernel: ata8.00: status: { DRDY } Jul 18 21:16:51 AVFILES kernel: ata8: hard resetting link Jul 18 21:16:52 AVFILES kernel: ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:16:52 AVFILES kernel: ata8.00: configured for UDMA/133 Jul 18 21:16:52 AVFILES kernel: ata8: EH complete
July 19, 201015 yr Author New update... Moved new cable to a different port on a different card, and still kicking errors. Jul 18 21:57:16 AVFILES kernel: ata5.00: exception Emask 0x10 SAct 0x1 SErr 0x280100 action 0x2 frozen Jul 18 21:57:16 AVFILES kernel: ata5.00: irq_stat 0x08000000, interface fatal error Jul 18 21:57:16 AVFILES kernel: ata5: SError: { UnrecovData 10B8B BadCRC } Jul 18 21:57:16 AVFILES kernel: ata5.00: cmd 60/f8:00:8f:96:19/03:00:00:00:00/40 tag 0 ncq 520192 in Jul 18 21:57:16 AVFILES kernel: res 40/00:00:8f:96:19/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:57:16 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:57:16 AVFILES kernel: ata5: hard resetting link Jul 18 21:57:17 AVFILES kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:57:17 AVFILES kernel: ata5.00: configured for UDMA/133 Jul 18 21:57:17 AVFILES kernel: ata5: EH complete Any ideas?
July 19, 201015 yr New update... Moved new cable to a different port on a different card, and still kicking errors. Jul 18 21:57:16 AVFILES kernel: ata5.00: exception Emask 0x10 SAct 0x1 SErr 0x280100 action 0x2 frozen Jul 18 21:57:16 AVFILES kernel: ata5.00: irq_stat 0x08000000, interface fatal error Jul 18 21:57:16 AVFILES kernel: ata5: SError: { UnrecovData 10B8B BadCRC } Jul 18 21:57:16 AVFILES kernel: ata5.00: cmd 60/f8:00:8f:96:19/03:00:00:00:00/40 tag 0 ncq 520192 in Jul 18 21:57:16 AVFILES kernel: res 40/00:00:8f:96:19/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:57:16 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:57:16 AVFILES kernel: ata5: hard resetting link Jul 18 21:57:17 AVFILES kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:57:17 AVFILES kernel: ata5.00: configured for UDMA/133 Jul 18 21:57:17 AVFILES kernel: ata5: EH complete Any ideas? Could be a defective drive. (or marginal power supply/ induced noise on cabling) Bad CRC indicated the checksum used in the protocol between the disk and the controller sees errors. If you are using a different cable, and a different port, that leaves the drive.
July 19, 201015 yr Are you having issues on two drives? Looks like it's throwing an exception for ata5 and ata8? Jul 18 21:02:43 AVFILES kernel: ata5.00: exception Emask 0x10 SAct 0x1 SErr 0x280100 action 0x2 frozen Jul 18 21:02:43 AVFILES kernel: ata5.00: irq_stat 0x08000000, interface fatal error Jul 18 21:02:43 AVFILES kernel: ata5: SError: { UnrecovData 10B8B BadCRC } Jul 18 21:02:43 AVFILES kernel: ata5.00: cmd 60/f8:00:07:c6:b3/03:00:00:00:00/40 tag 0 ncq 520192 in Jul 18 21:02:43 AVFILES kernel: res 40/00:00:07:c6:b3/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:02:43 AVFILES kernel: ata5.00: status: { DRDY } Jul 18 21:02:43 AVFILES kernel: ata5: hard resetting link Jul 18 21:02:45 AVFILES kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:02:45 AVFILES kernel: ata5.00: configured for UDMA/133 Jul 18 21:02:45 AVFILES kernel: ata5: EH complete Jul 18 21:02:45 AVFILES kernel: sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors (1000205 MB) Jul 18 21:02:45 AVFILES kernel: sd 5:0:0:0: [sdf] Write Protect is off Jul 18 21:02:45 AVFILES kernel: sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 Jul 18 21:02:45 AVFILES kernel: sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jul 18 21:03:33 AVFILES kernel: ata8.00: exception Emask 0x10 SAct 0x2 SErr 0x780100 action 0x2 Jul 18 21:03:33 AVFILES kernel: ata8.00: irq_stat 0x08000000 Jul 18 21:03:33 AVFILES kernel: ata8: SError: { UnrecovData 10B8B Dispar BadCRC Handshk } Jul 18 21:03:33 AVFILES kernel: ata8.00: cmd 60/c8:08:3f:72:d0/02:00:00:00:00/40 tag 1 ncq 364544 in Jul 18 21:03:33 AVFILES kernel: res 40/00:0c:3f:72:d0/00:00:00:00:00/40 Emask 0x10 (ATA bus error) Jul 18 21:03:33 AVFILES kernel: ata8.00: status: { DRDY } Jul 18 21:03:33 AVFILES kernel: ata8: hard resetting link Jul 18 21:03:34 AVFILES kernel: ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Jul 18 21:03:34 AVFILES kernel: ata8.00: configured for UDMA/133 Jul 18 21:03:34 AVFILES kernel: ata8: EH complete Jul 18 21:03:34 AVFILES kernel: sd 8:0:0:0: [sdi] 3907029168 512-byte hardware sectors (2000399 MB) Jul 18 21:03:34 AVFILES kernel: sd 8:0:0:0: [sdi] Write Protect is off Jul 18 21:03:34 AVFILES kernel: sd 8:0:0:0: [sdi] Mode Sense: 00 3a 00 00 Jul 18 21:03:34 AVFILES kernel: sd 8:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
July 19, 201015 yr Author Are you having issues on two drives? Looks like it's throwing an exception for ata5 and ata8? Oh crud... Joe, are you sure the errors are on the WD20EARS replacement drive? I did some more in depth looking at my logs, and it looks to me like it's two OTHER drives kicking the errors. Also, GREAT NEWS, (sarcasm) my latest logs look like my USB drive is failing as well. Is there any way to back up the config info if the drive itself isn't readable, but the system is currently still booted? I'm starting to seriously fear for my data. syslog-2010-07-19.txt
July 19, 201015 yr With as many drives as you have, and apparently random CRC and other errors on multiple drives, I'd suspect the power supply (or something common to all involved) What power supply are you using? (specific make/model) Joe L.
July 19, 201015 yr PC Power and Cooling S61EPS 610W Not likely to be an issue with its single 12 Volt rail. (But I wanted to try to eliminate anything common that might be obvious) Joe L.
July 19, 201015 yr Author So, is there any way to recover my user and share settings from a running machine that won't allow the unraid boot volume to be read? Ideally I'd like to be able to have config files to put on a new USB key.
July 19, 201015 yr It is probably not even mounted... If the flash drive is corrupted you might be able to fix it by putting it in a windows PC and running scandisk on it. Does it show when you type mount If not, just un-plug it and fix it in your windows PC. It would also be the reason for your "loop" since if it cannot be written it cannot be updated with the status that a successful parity check has been completed.
July 19, 201015 yr Author It shows up in the "mount" results, but I've got these (and more) errors in the syslog. Jul 18 23:21:25 AVFILES kernel: Buffer I/O error on device sda1, logical block 849 Jul 18 23:21:25 AVFILES kernel: lost page write due to I/O error on sda1 Jul 18 23:21:25 AVFILES kernel: write_file: write error 5 Jul 18 23:21:25 AVFILES kernel: md: could not write superblock from /boot/config/super.dat Jul 18 23:21:25 AVFILES emhttp: shcmd (70): /usr/sbin/nmbd -D Jul 18 23:21:25 AVFILES emhttp: shcmd (71): /usr/sbin/smbd -D Jul 18 23:21:26 AVFILES kernel: Buffer I/O error on device sda1, logical block 2 Jul 18 23:21:26 AVFILES kernel: lost page write due to I/O error on sda1 Jul 18 23:21:26 AVFILES kernel: Buffer I/O error on device sda1, logical block 250 Jul 18 23:21:26 AVFILES kernel: lost page write due to I/O error on sda1 Jul 18 23:22:13 AVFILES kernel: FAT: FAT read failed (blocknr 2) Jul 18 23:22:13 AVFILES kernel: FAT: Directory bread(block 849) failed
July 19, 201015 yr It shows up in the "mount" results, but I've got these (and more) errors in the syslog. Jul 18 23:21:25 AVFILES kernel: Buffer I/O error on device sda1, logical block 849 Jul 18 23:21:25 AVFILES kernel: lost page write due to I/O error on sda1 Jul 18 23:21:25 AVFILES kernel: write_file: write error 5 Jul 18 23:21:25 AVFILES kernel: md: could not write superblock from /boot/config/super.dat Jul 18 23:21:25 AVFILES emhttp: shcmd (70): /usr/sbin/nmbd -D Jul 18 23:21:25 AVFILES emhttp: shcmd (71): /usr/sbin/smbd -D Jul 18 23:21:26 AVFILES kernel: Buffer I/O error on device sda1, logical block 2 Jul 18 23:21:26 AVFILES kernel: lost page write due to I/O error on sda1 Jul 18 23:21:26 AVFILES kernel: Buffer I/O error on device sda1, logical block 250 Jul 18 23:21:26 AVFILES kernel: lost page write due to I/O error on sda1 Jul 18 23:22:13 AVFILES kernel: FAT: FAT read failed (blocknr 2) Jul 18 23:22:13 AVFILES kernel: FAT: Directory bread(block 849) failed All of those indicate you need to run scandisk on windows on your flash drive. Odds are it is corrupted. It is why you are constantly doing a parity check since the success status on its completion cannot be written to the flash drive.
July 19, 201015 yr Author If I unplug it at this point with the server running, will I be able to remount it without rebooting the server?
July 19, 201015 yr If I unplug it at this point with the server running, will I be able to remount it without rebooting the server? I never tried... It might even mount itself based on the inittab rules. I would try You really don't have much choice... one way or another, you are gong to go through another parity calc to write the proper status to the flash drive. You might try on the command line: dosfsck /dev/disk/by-label/UNRAID If it finds corruption you'll probably need to re-run it with the "-a" or "-r" option It it does not work I'd stop the array fix the corruption on the USB drive on a window's box. move it back, and try to start the array once more. Worst case, you 'll need to reboot.
July 19, 201015 yr Author Oh well, I guess I'll pull the drive and see what happens. Linux 2.6.24.4-unRAID. root@AVFILES:~# dosfsck /dev/disk/by-label/UNRAID dosfsck 2.11, 12 Mar 2005, FAT32, LFN Got 126464 bytes instead of 126838 at 512 root@AVFILES:~# dosfsck -a /dev/disk/by-label/UNRAID dosfsck 2.11, 12 Mar 2005, FAT32, LFN Got 126464 bytes instead of 126838 at 512 root@AVFILES:~# dosfsck -r /dev/disk/by-label/UNRAID dosfsck 2.11, 12 Mar 2005, FAT32, LFN Got 126464 bytes instead of 126838 at 512
July 20, 201015 yr Author Well, I guess I'm officially screwed. I pulled the USB stick, checked it in windows, fixed errors, put it back, and the system panicked when I tried to restart the array, so I rebooted, and now all my drives are blue, and it says "Stopped. Initial configuration" Is there a way to tell it that I have 1 failed drive that needs to be rebuilt from existing parity?
July 20, 201015 yr Well, I guess I'm officially screwed. I pulled the USB stick, checked it in windows, fixed errors, put it back, and the system panicked when I tried to restart the array, so I rebooted, and now all my drives are blue, and it says "Stopped. Initial configuration" Is there a way to tell it that I have 1 failed drive that needs to be rebuilt from existing parity? Yes. Which drive is the one you wish to force to be rebuilt? The basic technique is a modification of the "trust" procedure as outlined in the wiki here: http://lime-technology.com/wiki/index.php?title=Make_unRAID_Trust_the_Parity_Drive,_Avoid_Rebuilding_Parity_Unnecessarily You perform these exact steps; Log in via telnet and type initconfig /root/mdcmd set invalidslot X where X is the number of the disk you wish to be re-constructed from the others. Example: If disk9 is the one to be rebuilt, the command would be /root/mdcmd set invalidslot 9 Then go back to the web-interface and press "Start"
July 22, 201015 yr Author I have positive progress. I was able to use some windows file recovery magic to get back a super.dat that was new enough to use, and am currently assessing what data I may have lost, if any. Unmenu is hosed at the moment, but I have more important stuff to worry about right now. I assume when I get to that point, I can just delete the /boot/unmenu folder and rerun the install script? Thanks for your help guys, it turns out my biggest problem was the corrupted flash causing unraid not to be able to write status updates, just like you said Joe. The crc errors seem to be a red herring at the moment, I'll start chasing them after I am sure I have copies of my important data. Even though the errors are still occuring, they don't seem to be doing anything but slowing down access. Jonathan
July 22, 201015 yr I have positive progress. I was able to use some windows file recovery magic to get back a super.dat that was new enough to use, and am currently assessing what data I may have lost, if any. Supet.dat is only used to track the disk assignments. If you were to lose it entirely it would just be re-created automatically. But, regardless, you're back running. Unmenu is hosed at the moment, but I have more important stuff to worry about right now. I assume when I get to that point, I can just delete the /boot/unmenu folder and rerun the install script?Exactly. Don't even need to remove the folder, just: cd /boot/unmenu unmenu_install -u -d /boot/unmenu Thanks for your help guys, it turns out my biggest problem was the corrupted flash causing unraid not to be able to write status updates, just like you said Joe. The crc errors seem to be a red herring at the moment, I'll start chasing them after I am sure I have copies of my important data. Even though the errors are still occuring, they don't seem to be doing anything but slowing down access. Jonathan The CRC errors are real. They indicate an issue, possibly with the motherboard, or disk controller, or power supply, or cabling. They are detected, and, as you said, the communication with the disk re-tried. It does slow the disk communications down. But, are they just one symptom of a problem? They would not have caused corruption of the flash drive's file system, but a bad bit in memory would..., or a flaky power supply could... So first, get your critical data protected, and backed up elsewhere... then, figure out the CRC error cause.
July 23, 201015 yr Author Short lived success. I think I still have major issues. The lack of unmenu was caused by the flash drive showing up completely blank this time. I tried to stop the array, and it panicked, so I powered it down. Pulled the USB thumbdrive, it was VERY warm to the touch. Put it in my windows box, scandisked it, and everything showed back up. It's back in the server now, and doing a parity check. Before I stopped the array, I copied the syslog to my cache drive. It looks like the USB thumbdrive just disappeared. It was still mounted, but all reads just show errors. This is a snippet of the syslog. It had been running with no obvious errors for 24 hours at the time of this error. This was after a theoretically good rebuild, clean shut down, I initiated a parity check, it came up clean, I shut down clean again, and was copying important files and checking checksums looking for data errors. Jul 22 07:33:35 AVFILES dhcpcd[2046]: sending DHCP_REQUEST for 192.168.0.30 to 192.168.0.1 Jul 22 07:33:35 AVFILES dhcpcd[2046]: dhcpIPaddrLeaseTime=43200 in DHCP server response. Jul 22 07:33:35 AVFILES dhcpcd[2046]: DHCP_ACK received from (192.168.0.1) Jul 22 09:31:21 AVFILES kernel: mdcmd (467): clear Jul 22 09:31:56 AVFILES kernel: end_request: I/O error, dev sda, sector 60977 Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60945) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60946) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60947) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60948) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60949) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60950) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60951) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60952) failed Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507648 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507649 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507650 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507651 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507652 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507653 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507654 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507655 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507648 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507649 Jul 22 11:04:39 AVFILES unmenu[4827]: gawk: ./17-unmenu-syslog.awk:1: fatal: can't read sourcefile `./17-unmenu-syslog.awk' (Input/output error) Jul 22 11:04:50 AVFILES unmenu[4827]: gawk: ./16-unmenu-syslog.awk:1: fatal: can't read sourcefile `./16-unmenu-syslog.awk' (Input/output error) Jul 22 11:05:03 AVFILES unmenu[4827]: sh: ./200-unmenu-smarthistory.cgi: Input/output error Jul 22 11:05:38 AVFILES kernel: printk: 326 messages suppressed. Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507648 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507649 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507650 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507651 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507652 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507653 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507654 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507655 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507648 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507649 Jul 22 11:05:39 AVFILES unmenu[4827]: gawk: ./17-unmenu-syslog.awk:1: fatal: can't read sourcefile `./17-unmenu-syslog.awk' (Input/output error) Jul 22 11:05:47 AVFILES unmenu[4827]: gawk: ./25-unmenu-dupe_files.awk:1: fatal: can't read sourcefile `./25-unmenu-dupe_files.awk' (Input/output error)
July 23, 201015 yr Short lived success. I think I still have major issues. The lack of unmenu was caused by the flash drive showing up completely blank this time. I tried to stop the array, and it panicked, so I powered it down. Pulled the USB thumbdrive, it was VERY warm to the touch. Put it in my windows box, scandisked it, and everything showed back up. It's back in the server now, and doing a parity check. Before I stopped the array, I copied the syslog to my cache drive. It looks like the USB thumbdrive just disappeared. It was still mounted, but all reads just show errors. This is a snippet of the syslog. It had been running with no obvious errors for 24 hours at the time of this error. This was after a theoretically good rebuild, clean shut down, I initiated a parity check, it came up clean, I shut down clean again, and was copying important files and checking checksums looking for data errors. Jul 22 07:33:35 AVFILES dhcpcd[2046]: sending DHCP_REQUEST for 192.168.0.30 to 192.168.0.1 Jul 22 07:33:35 AVFILES dhcpcd[2046]: dhcpIPaddrLeaseTime=43200 in DHCP server response. Jul 22 07:33:35 AVFILES dhcpcd[2046]: DHCP_ACK received from (192.168.0.1) Jul 22 09:31:21 AVFILES kernel: mdcmd (467): clear Jul 22 09:31:56 AVFILES kernel: end_request: I/O error, dev sda, sector 60977 Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60945) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60946) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60947) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60948) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60949) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60950) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60951) failed Jul 22 09:31:56 AVFILES kernel: FAT: Directory bread(block 60952) failed Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507648 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507649 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507650 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507651 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507652 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507653 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507654 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507655 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507648 Jul 22 11:04:39 AVFILES kernel: Buffer I/O error on device sda1, logical block 507649 Jul 22 11:04:39 AVFILES unmenu[4827]: gawk: ./17-unmenu-syslog.awk:1: fatal: can't read sourcefile `./17-unmenu-syslog.awk' (Input/output error) Jul 22 11:04:50 AVFILES unmenu[4827]: gawk: ./16-unmenu-syslog.awk:1: fatal: can't read sourcefile `./16-unmenu-syslog.awk' (Input/output error) Jul 22 11:05:03 AVFILES unmenu[4827]: sh: ./200-unmenu-smarthistory.cgi: Input/output error Jul 22 11:05:38 AVFILES kernel: printk: 326 messages suppressed. Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507648 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507649 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507650 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507651 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507652 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507653 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507654 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507655 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507648 Jul 22 11:05:38 AVFILES kernel: Buffer I/O error on device sda1, logical block 507649 Jul 22 11:05:39 AVFILES unmenu[4827]: gawk: ./17-unmenu-syslog.awk:1: fatal: can't read sourcefile `./17-unmenu-syslog.awk' (Input/output error) Jul 22 11:05:47 AVFILES unmenu[4827]: gawk: ./25-unmenu-dupe_files.awk:1: fatal: can't read sourcefile `./25-unmenu-dupe_files.awk' (Input/output error) Looks a lot like your thumb-drive is dying.
Archived
This topic is now archived and is closed to further replies.