I ran a parity check (nocorrect) yesterday and it found 41 sync errors but showed no errors on any disk.  I then ran a parity check (correct) and it resulted in 41 sync errors (corrected).  Just in case I ran it again and I am still getting 41 errors corrected.  I'm not sure what my next step should be.  Please let me know what information you need in order to help me determine the cause of this.

 

Running version 4.7

 

 

Here the last few lines shown in unmenu:

 

Apr 2 04:06:49 Cube kernel: ata5: SError: { UnrecovData 10B8B Dispar Handshk } (Errors)

Apr 2 04:06:49 Cube kernel: ata5.00: failed command: READ DMA EXT (Minor Issues)

Apr 2 04:06:49 Cube kernel: ata5.00: cmd 25/00:00:4f:8a:85/00:04:7d:00:00/e0 tag 0 dma 524288 in (Drive related)

Apr 2 04:06:49 Cube kernel: res 50/00:00:4e:8a:85/00:00:7d:00:00/e0 Emask 0x10 (ATA bus error) (Errors)

Apr 2 04:06:49 Cube kernel: ata5.00: status: { DRDY } (Drive related)

Apr 2 04:06:49 Cube kernel: ata5: hard resetting link (Minor Issues)

Apr 2 04:06:49 Cube kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) (Drive related)

Apr 2 04:06:49 Cube kernel: ata5.00: configured for UDMA/133 (Drive related)

Apr 2 04:06:49 Cube kernel: ata5: EH complete (Drive related)

Apr 2 04:13:11 Cube ntpd[1453]: synchronized to 65.55.21.13, stratum 2

Apr 2 04:47:07 Cube sSMTP[5199]: Sent mail for root@localhost (221 pacmmta51 closing connection) uid=0 username=root outbytes=1547

Apr 2 04:47:25 Cube ntpd[1453]: no servers reachable

syslog.txt

It always says "corrected" even if it is a NOCORRECT parity check.

 

The first monthly check found the errors.  The second, (assuming you did a correcting check) found them again and fixed them.

A third should show no errors. (again, assuming you did not perform a NOCORRECT check the second time)

 

It is because the NOCORRECT check was never properly shown in the unRAID interface as "detected" instead of "corected"

 

It is a bug in the message returned when a NOCORRECT check is performed.  It should just say "detected"

 

Joe L.

I appreciate your time, but I may have not been clear in my description. This is the second CORRECT check that I have run and it still says it fixed 41 errors. I don't know what I should do next.  Those entries about ATA5 from unmenu look like I may have some disk errors or something to me, but I don't really know the proper way to attack the problem.

Should I run another CORRECT parity check?  2 in a row have claimed to fix 41 errors.  This server has been up and running for over a year with no problems.

I went ahead and ran another check. It isn't even done yet and has corrected 97 this time. I have no idea what to do.

I went ahead and ran another check. It isn't even done yet and has corrected 97 this time. I have no idea what to do.

To me, it indicates that you have a hardware issue.

 

It could be memory, a disk, a power supply,  a disk controller, etc.

 

Step 1.  Look in the syslog for errors, clues to what might be happening.

 

Unfortunately, there is no simple way to determine which disk is returning inconsistent results.  It could be any one of them.

 

Step 2. If nothing shows in the syslog, run a memory test.  If memory is an issue, nothing else can fix it.

 

Joe L.

There are several references to ATA5 errors in the log so in an effort to determine if it is the drive or the controller I swapped drive 5 and 6.  I was afraid to change too much at once (baby steps).  I ran a NOCORRECT scan and this time it found no errors but the log is still showing ATA5 errors.  Can you tell by the log if ATA5 has a different drive on it now?  I'm not even sure that swapping 5 and 6 changed the affected drive.  I'm sorry for the basic question, but I can't tell for sure.  If it is a different drive now and still giving the errors then obviously it is the controller.  The problem then is I don't know which controller it is.  The main board has 4 ports and I have a PCI card that has 2 drives on it (1 data and the cache drive).

 

I have attached a log from before and after swapping the drives.

syslog-2012-04-02_before_switch.txt

syslog-2012-04-04_after_switch.txt

In your first log file you will see the following section:

 

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919552

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919560

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919568

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919576

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919584

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919592

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919600

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919608

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919616

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919624

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919632

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919640

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919648

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919656

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919664

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919672

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919680

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919688

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919696

Apr  2 02:52:18 Cube kernel: md: parity incorrect: 1697919704

 

these are the blocks which had parity issues.  As you can see they are all in a fairly narrow range.  There appear to be 21 listed not 41. 

 

Run another correcting parity check and see if you get the same block numbers in error.  Then run another correcting check and see if the numbers reappear or shift to different blocks or go away. 

 

Read the "how to trouble shoot recurring parity errors" section of this portion of the FAQ:

 

http://lime-technology.com/wiki/index.php/FAQ#Hard_Drives

 

there was a long discussion about bad drives causing this sort of issue here (I was able to give some specific examples):

 

http://lime-technology.com/forum/index.php?topic=11515.0

 

Since your errors all appear to be in a short range of blocks you stand a good chance of being able to identify the disk (if that is the problem) by running repeated md5sum tests of each of the drives (the wiki contains a sample script).  In my case there was one drive that the "dd | md5sum" command would often return a different value on (say 1 out of 3 times) so this was why I could run a couple of good parity checks, then the problem would reappear on the next check and then disappear...  Note: the SMART reports gave no indications that anything was wrong.

 

Regards,

 

Stephen

 

I will do this as soon as I get home.  Is there any way to see what disk is involved in those ata5 errors?  I have spare disks on hand and would love to just get back up and running. 

OK, I think I have good and bad news.  The good is that the parity checks have found no more errors.  The bad is that I am still getting those ATA5.0 errors no matter which drive is connected to which port.  If I am thinking correctly that means it is a controller (I will try new cables first to make sure it isn't just a cable).  At this point I really need help determining which controller it is.  Is there something in the log that will help determine which controller is causing these ATA5.0 entries?  If not is there a way to know for certain which drive is attached to it at a given time so I can figure it out that way?  If it is the PCI card I am in luck, but if it is the mainboard I don't see that I have a choice other than to replace the board with something else.  If it were to come to replacing this mainboard as long as I know which disk is parity and cache, I should just be able to move all the other drives over to the new board correct?  The only other option I can see is that if we know which controller, I could copy everything from that drive share and remove the drive, init the array and copy back over with that slot empty.

Based on the following from your syslog, I'm going to guess that it is disk1 that has the errors.

Note that ata5 is then scsi 5:0:0:0 which then is sd:5:0:0:0. (as it uses the SCSI driver interface in the kernel)

Apr  3 22:55:38 Cube kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Apr  3 22:55:38 Cube kernel: ata5.00: ATA-8: WDC WD20EARS-00MVWB0, 51.0AB51, max UDMA/133

Apr  3 22:55:38 Cube kernel: ata5.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA

Apr  3 22:55:38 Cube kernel: ata5.00: configured for UDMA/133

Apr  3 22:55:38 Cube kernel: scsi 5:0:0:0: Direct-Access    ATA      WDC WD20EARS-00M 51.0 PQ: 0 ANSI: 5

Apr  3 22:55:38 Cube kernel: sd 5:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)

Apr  3 22:55:38 Cube kernel: sd 5:0:0:0: [sdc] Write Protect is off

Apr  3 22:55:38 Cube kernel: sd 5:0:0:0: [sdc] Mode Sense: 00 3a 00 00

Apr  3 22:55:38 Cube kernel: sd 5:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Apr  3 22:55:38 Cube kernel:  sdc: sdc1

Apr  3 22:55:38 Cube kernel: sd 5:0:0:0: [sdc] Attached SCSI disk

 

Apr  3 22:55:39 Cube emhttp: pci-0000:02:00.0-scsi-0:0:0:0 host5 (sdc) WDC_WD20EARS-00MVWB0_WD-WCAZA073986

Apr  3 22:55:39 Cube kernel: md: import disk1: [8,32] (sdc) WDC WD20EARS-00M WD-WCAZA0739864 size: 195351455

 

I see the disk initialize as a 3GB SATA link.  Perhaps the drive has a jumper to set it to the slower 1.5GB speed.  It might help if changing the SATA cable does not.

 

If you've changed drives around, the original syslog might not be pointing to the same disk, but you'll at least be able to figure it out based on these hints.  Please repeat this analysis by looking for the equivalent lines in a current syslog.

 

 

Joe L.

 

The erros in that old log seem to be CRC errors, pointing to a bad cable, or possibly a bad port on the disk controller

Apr  3 23:32:06 Cube kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x780100 action 0x6

Apr  3 23:32:06 Cube kernel: ata5.00: irq_stat 0x08000000

Apr  3 23:32:06 Cube kernel: ata5: SError: { UnrecovData 10B8B Dispar BadCRC Handshk }

Apr  3 23:32:06 Cube kernel: ata5.00: failed command: READ DMA EXT

Apr  3 23:32:06 Cube kernel: ata5.00: cmd 25/00:18:57:c0:55/00:02:0b:00:00/e0 tag 0 dma 274432 in

Apr  3 23:32:06 Cube kernel:          res 50/00:00:56:c0:55/00:00:0b:00:00/e0 Emask 0x10 (ATA bus error)

Apr  3 23:32:06 Cube kernel: ata5.00: status: { DRDY }

Apr  3 23:32:06 Cube kernel: ata5: hard resetting link

Thank you so much.  I just checked on the scan I started this morning before leaving for work and got what looks like the same errors:

 

Apr  6 13:25:35 Cube kernel: ata5.00: exception Emask 0x10 SAct 0x0 SErr 0x580100 action 0x6 (Errors)

Apr  6 13:25:35 Cube kernel: ata5.00: irq_stat 0x08000000 (Drive related)

Apr  6 13:25:35 Cube kernel: ata5: SError: { UnrecovData 10B8B Dispar Handshk } (Errors)

Apr  6 13:25:35 Cube kernel: ata5.00: failed command: READ DMA EXT (Minor Issues)

Apr  6 13:25:35 Cube kernel: ata5.00: cmd 25/00:d8:af:37:04/00:02:6b:00:00/e0 tag 0 dma 372736 in (Drive related)

Apr  6 13:25:35 Cube kernel:          res 50/00:00:ae:37:04/00:00:6b:00:00/e0 Emask 0x10 (ATA bus error) (Errors)

Apr  6 13:25:35 Cube kernel: ata5.00: status: { DRDY } (Drive related)

Apr  6 13:25:35 Cube kernel: ata5: hard resetting link (Minor Issues)

Apr  6 13:25:35 Cube kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) (Drive related)

Apr  6 13:25:35 Cube kernel: ata5.00: configured for UDMA/133 (Drive related)

Apr  6 13:25:35 Cube kernel: ata5: EH complete (Drive related)

 

In this most recent log it looks like these errors are still involving sdb as well, but there is a different drive attached so I think that backs up what you are thinking.  I apologize in advance for the stupid question, but how do I know for sure which controller this is.....can I go by the serial number of the drive attached?  Also if I get that problem fixed and run a scan....if I get no errors is it safe to assume those disk errors on sdc were related?

syslog-2012-04-06-3.txt

Based on the following from your syslog, I'm going to guess that it is disk1 that has the errors.

Note that ata5 is then scsi 5:0:0:0 which then is sd:5:0:0:0. (as it uses the SCSI driver interface in the kernel)

 

Is this always the case?

 

I am really new to all of this.  In the following lines is ata5 sdc or sdb (do I look at the entry before or after)?

 

Apr  6 07:51:52 Cube kernel:  sdb: sdb1

Apr  6 07:51:52 Cube kernel: sd 2:0:0:0: [sdb] Attached SCSI disk

Apr  6 07:51:52 Cube kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Apr  6 07:51:52 Cube kernel: ata5.00: ATA-8: WDC WD20EARS-00MVWB0, 51.0AB51, max UDMA/133

Apr  6 07:51:52 Cube kernel: ata5.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA

Apr  6 07:51:52 Cube kernel: ata5.00: configured for UDMA/133

Apr  6 07:51:52 Cube kernel: scsi 5:0:0:0: Direct-Access    ATA      WDC WD20EARS-00M 51.0 PQ: 0 ANSI: 5

Apr  6 07:51:52 Cube kernel: sd 5:0:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)

Apr  6 07:51:52 Cube kernel: sd 5:0:0:0: [sdc] Write Protect is off

Apr  6 07:51:52 Cube kernel: sd 5:0:0:0: [sdc] Mode Sense: 00 3a 00 00

Apr  6 07:51:52 Cube kernel: sd 5:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Apr  6 07:51:52 Cube kernel:  sdc:

Apr  6 07:51:52 Cube kernel: ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

Apr  6 07:51:52 Cube kernel: ata6.00: ATA-8: WDC WD20EADS-00R6B0, 01.00A01, max UDMA/133

Apr  6 07:51:52 Cube kernel: ata6.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32), AA

I think you look for the one after.

 

Some-times it is difficult to tell, as the lines in the syslog are intermingled as the drives initialize, but in this case, it seemed pretty consistent.

If it is still the same port, but a different disk drive.. I would suspect the cable first, and then the port itself on the disk controller.

If you've bundles the cables to the disks, it could be cross-talk from one cable to another, especially if bundled with any of the power cables.

 

 

Thanks again Joe.  I have looked through a few logs now and I think I know which port so when I get off work I will swap the cable out with a new one and with any luck maybe that will work.  If not, I guess I may be looking for a mainboard.  I really appreciate everyone's input.  I thought I had at least some understanding of unRAID until this came up....now I realize I am clueless.

Unfortunately it wasn't the cable.  I did notice though that there is one open port on the PCI card that I had forgotten about so I'm going to try that tonight.

That did the trick.