Intermittent link errors on secondary SATA interface only

[mattkhan]

I have an unraid 6.1.6 server which uses an ASRock Z87 Extreme6 mobo. This mobo has 10 sata ports delivered via 3 interfaces

 

00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

 

the intel one covers ata1-6, the 1st ASM1062 covers ata7-8 and the last one covers ata9-10

 

I have 2 drive cages with 5 slots in each and the ASM ports are spread across the 2 cages. The Intel controller is rock solid, no issues whatsoever. It seems like I hit problems whenever I use an ASM controller, the latest example is during a preclear and occurred as soon as preclear moved from preread to zero'ing

 

Jan  5 19:58:06 zalaga-unraid kernel: ata8.00: exception Emask 0x10 SAct 0x7fffffdf SErr 0x400000 action 0x6 frozen
Jan  5 19:58:06 zalaga-unraid kernel: ata8.00: irq_stat 0x08000000, interface fatal error
Jan  5 19:58:06 zalaga-unraid kernel: ata8: SError: { Handshk }
Jan  5 19:58:06 zalaga-unraid kernel: ata8.00: failed command: WRITE FPDMA QUEUED
Jan  5 19:58:06 zalaga-unraid kernel: ata8.00: cmd 61/40:00:80:4e:6d/05:00:01:00:00/40 tag 0 ncq 688128 out
Jan  5 19:58:06 zalaga-unraid kernel:         res 40/00:1c:40:5e:6d/00:00:01:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:58:06 zalaga-unraid kernel: ata8.00: status: { DRDY }
Jan  5 19:58:06 zalaga-unraid kernel: ata8.00: failed command: WRITE FPDMA QUEUED
Jan  5 19:58:06 zalaga-unraid kernel: ata8.00: cmd 61/40:08:c0:53:6d/05:00:01:00:00/40 tag 1 ncq 688128 out
Jan  5 19:58:06 zalaga-unraid kernel:         res 40/00:1c:40:5e:6d/00:00:01:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:58:06 zalaga-unraid kernel: ata8.00: status: { DRDY }
.
.
. repeats for a bit until 
.
.
n  5 19:58:06 zalaga-unraid kernel: ata8: hard resetting link
Jan  5 19:58:07 zalaga-unraid kernel: ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jan  5 19:58:07 zalaga-unraid kernel: ata8.00: configured for UDMA/133
Jan  5 19:58:07 zalaga-unraid kernel: ata8: EH complete
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: exception Emask 0x10 SAct 0x7fffffef SErr 0x400000 action 0x6 frozen
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: irq_stat 0x08000000, interface fatal error
Jan  5 19:58:41 zalaga-unraid kernel: ata8: SError: { Handshk }
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: failed command: WRITE FPDMA QUEUED
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: cmd 61/40:00:00:d5:0b/05:00:02:00:00/40 tag 0 ncq 688128 out
Jan  5 19:58:41 zalaga-unraid kernel:         res 40/00:14:80:df:0b/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: status: { DRDY }
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: failed command: WRITE FPDMA QUEUED
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: cmd 61/40:08:40:da:0b/05:00:02:00:00/40 tag 1 ncq 688128 out
Jan  5 19:58:41 zalaga-unraid kernel:         res 40/00:14:80:df:0b/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: status: { DRDY }
.
.
. carries on until
.
.
.
Jan  5 19:58:41 zalaga-unraid kernel: ata8.00: status: { DRDY }
Jan  5 19:58:41 zalaga-unraid kernel: ata8: hard resetting link
Jan  5 19:58:42 zalaga-unraid kernel: ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jan  5 19:58:42 zalaga-unraid kernel: ata8.00: configured for UDMA/133
Jan  5 19:58:42 zalaga-unraid kernel: ata8: EH complete
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: exception Emask 0x10 SAct 0xff0 SErr 0x400000 action 0x6 frozen
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: irq_stat 0x08000000, interface fatal error
Jan  5 19:58:52 zalaga-unraid kernel: ata8: SError: { Handshk }
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: failed command: WRITE FPDMA QUEUED
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: cmd 61/40:20:80:88:3b/05:00:02:00:00/40 tag 4 ncq 688128 out
Jan  5 19:58:52 zalaga-unraid kernel:         res 40/00:5c:40:ad:3b/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: status: { DRDY }
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: failed command: WRITE FPDMA QUEUED
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: cmd 61/40:28:c0:8d:3b/05:00:02:00:00/40 tag 5 ncq 688128 out
Jan  5 19:58:52 zalaga-unraid kernel:         res 40/00:5c:40:ad:3b/00:00:02:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: status: { DRDY }
.
.
. and again
.
.
Jan  5 19:58:52 zalaga-unraid kernel: ata8.00: status: { DRDY }
Jan  5 19:58:52 zalaga-unraid kernel: ata8: hard resetting link
Jan  5 19:58:53 zalaga-unraid kernel: ata8: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jan  5 19:58:53 zalaga-unraid kernel: ata8.00: configured for UDMA/133
Jan  5 19:58:53 zalaga-unraid kernel: ata8: EH complete
Jan  5 19:59:41 zalaga-unraid kernel: ata8: limiting SATA link speed to 3.0 Gbps
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: exception Emask 0x10 SAct 0x77ffffff SErr 0x400000 action 0x6 frozen
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: irq_stat 0x08000000, interface fatal error
Jan  5 19:59:41 zalaga-unraid kernel: ata8: SError: { Handshk }
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: failed command: WRITE FPDMA QUEUED
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: cmd 61/40:00:00:8d:19/05:00:03:00:00/40 tag 0 ncq 688128 out
Jan  5 19:59:41 zalaga-unraid kernel:         res 40/00:cc:40:10:1a/00:00:03:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: status: { DRDY }
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: failed command: WRITE FPDMA QUEUED
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: cmd 61/40:08:40:92:19/05:00:03:00:00/40 tag 1 ncq 688128 out
Jan  5 19:59:41 zalaga-unraid kernel:         res 40/00:cc:40:10:1a/00:00:03:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: status: { DRDY }
.
.
. until...
.
.
Jan  5 19:59:41 zalaga-unraid kernel:         res 40/00:cc:40:10:1a/00:00:03:00:00/40 Emask 0x10 (ATA bus error)
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: status: { DRDY }
Jan  5 19:59:41 zalaga-unraid kernel: ata8: hard resetting link
Jan  5 19:59:41 zalaga-unraid kernel: ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Jan  5 19:59:41 zalaga-unraid kernel: ata8.00: configured for UDMA/133
Jan  5 19:59:41 zalaga-unraid kernel: ata8: EH complete

 

and now it's fine and continues charging along zero'ing at 150MB/s

 

The same behaviour has been seen after; swapping the drive cage, changing the drive, swapping sata cables

 

It seems the variable left is the mobo. It's not obvious to me how it could be the PSU as the server is otherwise completely idle when this happens.

 

Any ideas would be appreciated on how to debug (or fix!) this further.

[mattkhan]

anyone with any thoughts on this?

[Frank1940]

I am certainly no expert in this area but I would suggest that you get a complete Diagnostics file by going to 'Tools' tab, and clicking on the 'Diagnostics' icon.  It will generate a file that you will download to your computer.  Then attach that file to a new post in this thread.  That will give the real experts much, much more information to investigate your problem.

[mattkhan]

fair point

 

the diagnostics I posted in http://lime-technology.com/forum/index.php?topic=45255.msg432055#msg432055 cover from the same time period