RD errors, 150+ Callbacks Suppressed

[silasfelinus]

After replacing a string of older hard drives, removing one drive and reducing my array (possibly introducing some duplicate files in the process of trying to save data), and re-mapping cables (19 disk array, lots of cables), the system has finally been running without hard drive errors reported on Main, but today I noticed these in the log after a bunch of dockers refused to load via Traefik:

Log is repeating these errors, with a long block of the rd error messages ending in the format: "rd 2044759, flush 0, corrupt 0, gen 0"

Nov 13 09:41:54 alexandria kernel: BTRFS error (device sdm1: state EAL): bdev /dev/sdm1 errs: wr 44, rd 2044759, flush 0, corrupt 0, gen 0
Nov 13 09:41:56 alexandria kernel: vethaf9fa70: renamed from eth0
Nov 13 09:41:56 alexandria kernel: br-de846b490fb5: port 13(vethbc0943c) entered disabled state
Nov 13 09:41:56 alexandria  avahi-daemon[7877]: Interface vethbc0943c.IPv6 no longer relevant for mDNS.
Nov 13 09:41:56 alexandria  avahi-daemon[7877]: Leaving mDNS multicast group on interface vethbc0943c.IPv6 with address fe80::705a:8ff:fe30:bca5.
Nov 13 09:41:56 alexandria kernel: br-de846b490fb5: port 13(vethbc0943c) entered disabled state
Nov 13 09:41:56 alexandria kernel: device vethbc0943c left promiscuous mode
Nov 13 09:41:56 alexandria kernel: br-de846b490fb5: port 13(vethbc0943c) entered disabled state
Nov 13 09:41:56 alexandria  avahi-daemon[7877]: Withdrawing address record for fe80::705a:8ff:fe30:bca5 on vethbc0943c.
Nov 13 09:41:57 alexandria kernel: br-de846b490fb5: port 13(veth64d52cd) entered blocking state
Nov 13 09:41:57 alexandria kernel: br-de846b490fb5: port 13(veth64d52cd) entered disabled state
Nov 13 09:41:57 alexandria kernel: device veth64d52cd entered promiscuous mode
Nov 13 09:41:57 alexandria kernel: br-de846b490fb5: port 13(veth64d52cd) entered blocking state
Nov 13 09:41:57 alexandria kernel: br-de846b490fb5: port 13(veth64d52cd) entered forwarding state
Nov 13 09:41:57 alexandria kernel: eth0: renamed from veth304c008
Nov 13 09:41:57 alexandria kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth64d52cd: link becomes ready
Nov 13 09:41:58 alexandria kernel: btrfs_dev_stat_print_on_error: 142 callbacks suppressed
Nov 13 09:41:58 alexandria kernel: BTRFS error (device sdm1: state EAL): bdev /dev/sdm1 errs: wr 44, rd 2044902, flush 0, corrupt 0, gen 0


Thank you for advise! Diagnostics attached
 

alexandria-diagnostics-20231113-0934.zip

[JorgeB]

Nov 10 19:09:36 alexandria  emhttpd: import 31 cache device: (sdm) SPCC_Solid_State_Disk_YTAK230100803

 

This device dropped offline, check/replace cables and post new diags after array start.

[silasfelinus]

Relocated ssd to a new spot on array cables next to the other ssds. It appeared on reboot, no errors reported on Main and all drives accounted for, but now log shows new I/O errors, possibly after triggering Move

 

Quote

Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#23 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#23 CDB: opcode=0x28 28 00 23 8f 5c 38 00 00 20 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597816 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#24 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#24 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#24 CDB: opcode=0x28 28 00 23 8f 5c 60 00 00 10 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597856 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#25 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#25 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#25 CDB: opcode=0x28 28 00 23 8f 5c 70 00 00 10 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597872 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#26 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#26 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#26 CDB: opcode=0x28 28 00 23 8f 5c 88 00 00 08 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597896 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#27 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#27 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#27 CDB: opcode=0x28 28 00 23 8f 5c 98 00 00 08 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597912 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#28 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#28 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#28 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#28 CDB: opcode=0x28 28 00 23 8f 5c b0 00 00 10 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597936 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#29 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#29 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#29 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#29 CDB: opcode=0x28 28 00 23 8f 5c c8 00 00 08 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597960 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: ata3: EH complete
Nov 13 11:32:01 alexandria Plugin Auto Update: Checking for available plugin updates
Nov 13 11:32:01 alexandria Docker Auto Update: Community Applications Docker Autoupdate running
Nov 13 11:32:01 alexandria Docker Auto Update: Checking for available updates
Nov 13 11:32:07 alexandria Plugin Auto Update: Checking for language updates
Nov 13 11:32:07 alexandria Plugin Auto Update: Community Applications Plugin Auto Update finished
Nov 13 11:32:49 alexandria Docker Auto Update: No updates will be installed

Nov 13 11:10:29 alexandria kernel: ata3.00: status: { DRDY }
Nov 13 11:10:29 alexandria kernel: ata3.00: failed command: READ FPDMA QUEUED
Nov 13 11:10:29 alexandria kernel: ata3.00: cmd 60/10:c8:70:5c:8f/00:00:23:00:00/40 tag 25 ncq dma 8192 in
Nov 13 11:10:29 alexandria kernel:         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 13 11:10:29 alexandria kernel: ata3.00: status: { DRDY }
Nov 13 11:10:29 alexandria kernel: ata3.00: failed command: READ FPDMA QUEUED
Nov 13 11:10:29 alexandria kernel: ata3.00: cmd 60/08:d0:88:5c:8f/00:00:23:00:00/40 tag 26 ncq dma 4096 in
Nov 13 11:10:29 alexandria kernel:         res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 13 11:10:29 alexandria kernel: ata3.00: status: { DRDY }
Nov 13 11:10:29 alexandria kernel: ata3.00: failed command: READ FPDMA QUEUED
Nov 13 11:10:29 alexandria kernel: ata3.00: cmd 60/08:d8:98:5c:8f/00:00:23:00:00/40 tag 27 ncq dma 4096 in
Nov 13 11:10:29 alexandria kernel:         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 13 11:10:29 alexandria kernel: ata3.00: status: { DRDY }
Nov 13 11:10:29 alexandria kernel: ata3.00: failed command: READ FPDMA QUEUED
Nov 13 11:10:29 alexandria kernel: ata3.00: cmd 60/10:e0:b0:5c:8f/00:00:23:00:00/40 tag 28 ncq dma 8192 in
Nov 13 11:10:29 alexandria kernel:         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 13 11:10:29 alexandria kernel: ata3.00: status: { DRDY }
Nov 13 11:10:29 alexandria kernel: ata3.00: failed command: READ FPDMA QUEUED
Nov 13 11:10:29 alexandria kernel: ata3.00: cmd 60/08:e8:c8:5c:8f/00:00:23:00:00/40 tag 29 ncq dma 4096 in
Nov 13 11:10:29 alexandria kernel:         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 13 11:10:29 alexandria kernel: ata3.00: status: { DRDY }
Nov 13 11:10:29 alexandria kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Nov 13 11:10:29 alexandria kernel: ata3.00: cmd 61/40:f0:80:bc:e1/00:00:15:00:00/40 tag 30 ncq dma 32768 out
Nov 13 11:10:29 alexandria kernel:         res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 13 11:10:29 alexandria kernel: ata3.00: status: { DRDY }
Nov 13 11:10:29 alexandria kernel: ata3.00: failed command: READ FPDMA QUEUED
Nov 13 11:10:29 alexandria kernel: ata3.00: cmd 60/08:f8:10:5c:8f/00:00:23:00:00/40 tag 31 ncq dma 4096 in
Nov 13 11:10:29 alexandria kernel:         res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 13 11:10:29 alexandria kernel: ata3.00: status: { DRDY }
Nov 13 11:10:29 alexandria kernel: ata3: hard resetting link
Nov 13 11:10:29 alexandria kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Nov 13 11:10:29 alexandria kernel: ata3.00: configured for UDMA/133
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#20 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#20 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#20 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#20 CDB: opcode=0x28 28 00 23 8f 5b d8 00 00 08 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597720 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#21 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#21 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#21 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#21 CDB: opcode=0x28 28 00 23 8f 5b e8 00 00 08 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597736 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#22 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#22 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#22 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#22 CDB: opcode=0x28 28 00 23 8f 5c 20 00 00 10 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597792 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#23 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#23 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#23 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#23 CDB: opcode=0x28 28 00 23 8f 5c 38 00 00 20 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597816 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#24 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#24 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#24 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#24 CDB: opcode=0x28 28 00 23 8f 5c 60 00 00 10 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597856 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#25 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#25 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#25 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#25 CDB: opcode=0x28 28 00 23 8f 5c 70 00 00 10 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597872 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#26 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#26 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#26 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#26 CDB: opcode=0x28 28 00 23 8f 5c 88 00 00 08 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597896 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#27 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#27 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#27 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#27 CDB: opcode=0x28 28 00 23 8f 5c 98 00 00 08 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597912 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#28 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#28 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#28 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#28 CDB: opcode=0x28 28 00 23 8f 5c b0 00 00 10 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597936 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#29 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#29 Sense Key : 0x5 [current] 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#29 ASC=0x21 ASCQ=0x4 
Nov 13 11:10:29 alexandria kernel: sd 4:0:0:0: [sdd] tag#29 CDB: opcode=0x28 28 00 23 8f 5c c8 00 00 08 00
Nov 13 11:10:29 alexandria kernel: I/O error, dev sdd, sector 596597960 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 13 11:10:29 alexandria kernel: ata3: EH complete
Nov 13 11:32:01 alexandria Plugin Auto Update: Checking for available plugin updates
Nov 13 11:32:01 alexandria Docker Auto Update: Community Applications Docker Autoupdate running
Nov 13 11:32:01 alexandria Docker Auto Update: Checking for available updates
Nov 13 11:32:07 alexandria Plugin Auto Update: Checking for language updates
Nov 13 11:32:07 alexandria Plugin Auto Update: Community Applications Plugin Auto Update finished
Nov 13 11:32:49 alexandria Docker Auto Update: No updates will be installed
Nov 13 11:33:10 alexandria webGUI: Successful login user root from fe80::729f:a1b:a061:3faf
 

alexandria-diagnostics-20231113-1129.zip

[silasfelinus]

I haven't had an error since those initial IO's, so I've marked that as solved. I'll admit feeling a bit chagrined if it was a cable issue all along, but I'm so hoping that's all it was, thanks!

[JorgeB]

Keep monitoring for any more ATA errors, those are not normal.