December 2, 200619 yr Hi, I ran a parity check on my drives since it's been about 3 or 4 weeks since it last ran. When the parity check finished it says completed with 0 errors. However if I look at the columns where it indicates Reads, Writes, etc the column for Errors shows 189 errors for disk 3. Does this mean I have a problem with the files on my disk? What do I do now?
December 2, 200619 yr When the parity check finished it says completed with 0 errors. However if I look at the columns where it indicates Reads, Writes, etc the column for Errors shows 189 errors for disk 3. What that means is that the parity is doing it's job and that you have one disk going bad. The parity info should recover whatever is not correct on that one drive. I would look critically at the drive with the errors and also examine the syslogs. Could be a loose or bad cable but you might consider replacing it. So far you should not have lost data as the parity is intact.
December 3, 200619 yr Author where do I find the syslogs and is there something I can run that will scan the drive to see if it's okay?
December 7, 200619 yr When it says "Last checked on <date & time>, finding 0 errors", it means in this case, it found 0 sync errors. A sync error is where the data of a 4K parity block does not match the generated parity of the corresponding 4K blocks from the data drives. The counter does not take into account the number of read errors encountered during the parity check - these are reported in the Errors column (as you have noted). When a read error is encountered during a parity check, the s/w will generate the data that should be there, and then write that data to the disk which which had the error. If you run a parity check and the Errors counter(s) increment, then you should run another parity check. If this time the Errors counter(s) do not increment, then the disk is "probably" ok. If the Errors counter(s) still increments, you should consider replacing the disk. To view the log, telnet into the server and type: cat /var/log/syslog
December 11, 200619 yr Author when I type in cat /var/log/syslog I get the following which is probably not what I should be getting. Tower login: root Linux 2.4.33. root@Tower:~# cat /var/log/syslog Dec 10 04:40:01 Tower syslogd 1.4.1: restart. Dec 10 08:58:27 Tower emhttp[959]: shcmd (51): /usr/sbin/hdparm -y /dev/hdb >/de v/null Dec 10 09:35:52 Tower smbd[4945]: [2006/12/10 09:35:52, 0] lib/util_sock.c:read_ socket_data(384) Dec 10 09:35:52 Tower smbd[4945]: read_socket_data: recv failure for 4. Error = Connection timed out Dec 10 13:02:27 Tower emhttp[959]: shcmd (52): /usr/sbin/hdparm -y /dev/hdb >/de v/null root@Tower:~# cat /var/log/syslog Dec 10 04:40:01 Tower syslogd 1.4.1: restart. Dec 10 08:58:27 Tower emhttp[959]: shcmd (51): /usr/sbin/hdparm -y /dev/hdb >/de v/null Dec 10 09:35:52 Tower smbd[4945]: [2006/12/10 09:35:52, 0] lib/util_sock.c:read_ socket_data(384) Dec 10 09:35:52 Tower smbd[4945]: read_socket_data: recv failure for 4. Error = Connection timed out Dec 10 13:02:27 Tower emhttp[959]: shcmd (52): /usr/sbin/hdparm -y /dev/hdb >/de v/null root@Tower:~# cat
December 11, 200619 yr Actually those "errors" are fine (i.e., they are not disk errors). You can send the entire log to [email protected] and I'll take a look.
January 3, 200719 yr Author Actually those "errors" are fine (i.e., they are not disk errors). You can send the entire log to [email protected] and I'll take a look. Tom, I reran my syslog but it appears to be empty. Is the log cleared out when you restart the system? Do I need to rerun a parity check, then grab the syslogs before I stop by system? When you say the errors are fine, what exactly do you mean? Every time I run a parity check disk 3 always comes back with those errors. What's causing this.
January 3, 200719 yr The system log is on a ram drive so it's contents are lost after a reset or power cycle. In your case, you should re-run a parity-check and then look at the syslog again.
January 10, 200719 yr Author The system log is on a ram drive so it's contents are lost after a reset or power cycle. In your case, you should re-run a parity-check and then look at the syslog again. FYI I sent you my log file. Lots of errors in there. I'm hoping it's as simple as reseating the drives and cables but please let me know what you think might be the problem. Thanks!
January 16, 200719 yr Author Tom, Still haven't heard a word about my parity errors. I emailed you my syslog as you requested. Any news?? I'd really like to get this sorted out before expanding my array.
January 17, 200719 yr Author Hello, I ran a parity check and received the following errors in my syslog. Hopefully you can shed some light on it. Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476296/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476304/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476312/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476320/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476328/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476336/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476344/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476352/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476360/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476368/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476376/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476384/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476392/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476400/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476408/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476416/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476424/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476432/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476440/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476448/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476456/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476464/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476472/3, count: 1, uptodate 0 . Jan 8 20:39:51 Tower kernel: md3: read error! Jan 8 20:39:51 Tower kernel: end_read_request 198476480/3, count: 1, uptodate 0 . Jan 8 20:40:05 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:05 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:05 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:06 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:06 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:06 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:08 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:08 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:08 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:09 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:09 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:09 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:10 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:10 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:10 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:10 Tower kernel: scsi2: ERROR on channel 0, id 1, lun 0, CDB: 0x28 00 0b e3 4d 07 00 00 c8 00 Jan 8 20:40:10 Tower kernel: Current sd08:41: sns = 70 3 Jan 8 20:40:10 Tower kernel: ASC=11 ASCQ= 4 Jan 8 20:40:10 Tower kernel: Raw sense data:0x70 0x00 0x03 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 0x00 0x11 0x04 0x00 0x00 0x00 0x00 Jan 8 20:40:10 Tower kernel: I/O error: dev 08:41, sector 199445704 Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445704/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445712/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445720/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445728/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445736/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445744/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445752/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445760/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445768/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445776/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445784/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445792/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445800/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445808/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445816/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445824/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445832/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445840/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445848/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445856/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445864/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445872/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445880/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445888/3, count: 1, uptodate 0 . Jan 8 20:40:10 Tower kernel: md3: read error! Jan 8 20:40:10 Tower kernel: end_read_request 199445896/3, count: 1, uptodate 0 . Jan 8 20:40:11 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:11 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:11 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:12 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:12 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:12 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:13 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:13 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:13 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:15 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:15 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:15 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:16 Tower kernel: ata2: translated ATA stat/err 0x51/40 to SCSI SK/A SC/ASCQ 0x3/11/04 Jan 8 20:40:16 Tower kernel: ata2: status=0x51 { DriveReady SeekComplete Error } Jan 8 20:40:16 Tower kernel: ata2: error=0x40 { UncorrectableError } Jan 8 20:40:16 Tower kernel: scsi2: ERROR on channel 0, id 1, lun 0, CDB: 0x28 00 0b e3 66 3f 00 00 c8 00 Jan 8 20:40:16 Tower kernel: Current sd08:41: sns = 70 3 Jan 8 20:40:16 Tower kernel: ASC=11 ASCQ= 4 Jan 8 20:40:16 Tower kernel: Raw sense data:0x70 0x00 0x03 0x00 0x00 0x00 0x00 0x0a 0x00 0x00 0x00 0x00 0x11 0x04 0x00 0x00 0x00 0x00 Jan 8 20:40:16 Tower kernel: I/O error: dev 08:41, sector 199452160 Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452160/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452168/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452176/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452184/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452192/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452200/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452208/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452216/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452224/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452232/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452240/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452248/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452256/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452264/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452272/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452280/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452288/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452296/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452304/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452312/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452320/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452328/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452336/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452344/3, count: 1, uptodate 0 . Jan 8 20:40:16 Tower kernel: md3: read error! Jan 8 20:40:16 Tower kernel: end_read_request 199452352/3, count: 1, uptodate 0 . Jan 8 21:31:27 Tower emhttp[959]: shcmd (152): /usr/sbin/hdparm -y /dev/hdb >/d ev/null Jan 8 22:09:27 Tower emhttp[959]: shcmd (153): /usr/sbin/hdparm -y /dev/sde >/d ev/null Jan 8 22:12:06 Tower kernel: md: sync done. time=40550sec rate=7708K/sec Jan 8 22:12:06 Tower kernel: md: writing superblock to /boot/config/super.dat Jan 8 22:12:06 Tower kernel: md: recovery thread sync completion status: 0 Jan 8 23:05:27 Tower emhttp[959]: shcmd (154): /usr/sbin/hdparm -y /dev/sdc >/d ev/null Jan 8 23:13:27 Tower emhttp[959]: shcmd (155): /usr/sbin/hdparm -y /dev/sdb >/d ev/null Jan 8 23:13:27 Tower emhttp[959]: shcmd (156): /usr/sbin/hdparm -y /dev/sdd >/d ev/null root@Tower:~#
January 18, 200719 yr Those error messages represent aborted I/O operations as a result of disk read errors on data disk3.
January 18, 200719 yr Author Those error messages represent aborted I/O operations as a result of disk read errors on data disk3. Which means what exactly? Does that mean I have a problem with my drive, cables, etc. Is there something I should be looking at fixing? I'd really like to understand the error. I am reframing from copying data to that drive because I don't know what to make of it. Any help would greatly be appreciated.
January 21, 200719 yr Yes, read errors being reported on disk3 indicates something is wrong with disk3. If you are using a high quality power supply, and good cables, then it's the hard drive itself. Eventually it will probably experience a write-error, in which case the s/w will disable it. Worse, if another hard drives fails, it may not get rebuilt successfully because this drive's read errors will prevent reconstruction of those corresponding sectors. One thing you can do is replace it with another hard drive, and move it to another controller. If the read errors follow the drive (that is, they still happen when attached to another controller) you can be pretty certain the hard drive is flaky.
January 27, 200719 yr Author Okay thanks Tom. Now that was the type of help I was looking for. That gives me something more to work with.
Archived
This topic is now archived and is closed to further replies.