Given the following disk log of one disk in one of my ZFS Cache Pools I'm trying to ascertain what is actually happening - I'm concerned it might be failing, but I'm also concerned something else is causing this.  One thing I am also concerned about, considering this is an SSD, is that its "spinning down".  After seeing this I did set the cache pool to "Never" on Spin Down Delay to see if that has any effect.

 

Any pointers or info would be greatly appreciated.

 

 

Jan 27 11:13:12 catan kernel: sd 2:0:8:0: [sdi] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
Jan 27 11:13:12 catan kernel: sd 2:0:8:0: [sdi] Write Protect is off
Jan 27 11:13:12 catan kernel: sd 2:0:8:0: [sdi] Mode Sense: 46 00 10 08
Jan 27 11:13:12 catan kernel: sd 2:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jan 27 11:13:12 catan kernel: sdi: sdi1
Jan 27 11:13:12 catan kernel: sd 2:0:8:0: [sdi] Attached SCSI disk
Jan 27 11:21:27 catan emhttpd: online: SATA_SSD_22111451213917 (sdi) 512 1000215216
Jan 27 11:21:27 catan emhttpd: import 34 pool device: (sdi) SATA_SSD_22111451213917
Jan 27 11:21:29 catan emhttpd: read SMART /dev/sdi
Jan 27 11:22:02 catan kernel: sd 2:0:8:0: [sdi] Synchronizing SCSI cache
Jan 27 11:22:02 catan kernel: sd 2:0:8:0: [sdi] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK
Jan 27 11:22:17 catan kernel: sd 2:0:12:0: [sdi] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
Jan 27 11:22:17 catan kernel: sd 2:0:12:0: [sdi] Write Protect is off
Jan 27 11:22:17 catan kernel: sd 2:0:12:0: [sdi] Mode Sense: 46 00 10 08
Jan 27 11:22:17 catan kernel: sd 2:0:12:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jan 27 11:22:17 catan kernel: sdi: sdi1
Jan 27 11:22:17 catan kernel: sd 2:0:12:0: [sdi] Attached SCSI disk
Jan 27 11:22:37 catan emhttpd: import 34 pool device: (sdi) SATA_SSD_22111451213917
Jan 27 11:22:48 catan emhttpd: shcmd (133): /usr/sbin/zpool import -f -m -N -o autoexpand=on  -d /dev/sdc1 -d /dev/sdd1 -d /dev/sde1 -d /dev/sdb1 -d /dev/sdi1 -d /dev/sdf1 -d /dev/sdh1 12046042128853122756 cachetwo
Jan 27 11:22:49 catan emhttpd:    /dev/sdi1  ONLINE       0     0     0
Jan 27 12:00:26 catan root: /mnt/cachetwo: 1.25 TiB (1376806121472 bytes) trimmed on /dev/sdc1, /dev/sdd1, /dev/sde1, /dev/sdb1, /dev/sdi1, /dev/sdf1, /dev/sdh1
Jan 27 16:00:26 catan root: /mnt/cachetwo: 1.25 TiB (1377107742720 bytes) trimmed on /dev/sdc1, /dev/sdd1, /dev/sde1, /dev/sdb1, /dev/sdi1, /dev/sdf1, /dev/sdh1
Jan 27 20:00:24 catan root: /mnt/cachetwo: 1.26 TiB (1380435308544 bytes) trimmed on /dev/sdc1, /dev/sdd1, /dev/sde1, /dev/sdb1, /dev/sdi1, /dev/sdf1, /dev/sdh1
Jan 27 21:50:40 catan kernel: sd 2:0:12:0: [sdi] tag#181 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=0s
Jan 27 21:50:40 catan kernel: sd 2:0:12:0: [sdi] tag#181 CDB: opcode=0x28 28 00 04 0d 45 68 00 00 08 00
Jan 27 21:50:40 catan kernel: I/O error, dev sdi, sector 67978600 op 0x0:(READ) flags 0x100 phys_seg 1 prio class 2
Jan 27 21:50:40 catan kernel: zio pool=cachetwo vdev=/dev/sdi1 error=5 type=1 offset=34803994624 size=4096 flags=1572992
Jan 27 21:50:40 catan kernel: sd 2:0:12:0: [sdi] tag#182 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=0s
Jan 27 21:50:40 catan kernel: sd 2:0:12:0: [sdi] tag#182 CDB: opcode=0x28 28 00 00 00 0a 10 00 00 10 00
Jan 27 21:50:40 catan kernel: I/O error, dev sdi, sector 2576 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 27 21:50:40 catan kernel: zio pool=cachetwo vdev=/dev/sdi1 error=5 type=1 offset=270336 size=8192 flags=721089
Jan 27 21:50:40 catan kernel: sd 2:0:12:0: [sdi] tag#183 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=0s
Jan 27 21:50:40 catan kernel: sd 2:0:12:0: [sdi] tag#183 CDB: opcode=0x28 28 00 3b 9e 0e 10 00 00 10 00
Jan 27 21:50:40 catan kernel: I/O error, dev sdi, sector 1000214032 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 27 21:50:40 catan kernel: zio pool=cachetwo vdev=/dev/sdi1 error=5 type=1 offset=512108535808 size=8192 flags=721089
Jan 27 21:50:40 catan kernel: sd 2:0:12:0: [sdi] tag#184 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=0s
Jan 27 21:50:40 catan kernel: sd 2:0:12:0: [sdi] tag#184 CDB: opcode=0x28 28 00 3b 9e 10 10 00 00 10 00
Jan 27 21:50:40 catan kernel: I/O error, dev sdi, sector 1000214544 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
Jan 27 21:50:40 catan kernel: zio pool=cachetwo vdev=/dev/sdi1 error=5 type=1 offset=512108797952 size=8192 flags=721089
Jan 27 21:50:46 catan kernel: sd 2:0:12:0: [sdi] Synchronizing SCSI cache
Jan 27 21:50:46 catan kernel: sd 2:0:12:0: [sdi] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK
Jan 27 21:51:01 catan kernel: sd 2:0:13:0: [sdi] 1000215216 512-byte logical blocks: (512 GB/477 GiB)
Jan 27 21:51:01 catan kernel: sd 2:0:13:0: [sdi] Write Protect is off
Jan 27 21:51:01 catan kernel: sd 2:0:13:0: [sdi] Mode Sense: 46 00 10 08
Jan 27 21:51:01 catan kernel: sd 2:0:13:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jan 27 21:51:01 catan kernel: sdi: sdi1
Jan 27 21:51:01 catan kernel: sd 2:0:13:0: [sdi] Attached SCSI disk
Jan 27 22:06:04 catan emhttpd: spinning down /dev/sdi
Jan 27 22:06:04 catan emhttpd: sdspin /dev/sdi down: 1

 

Please post the diagnostics.
 

Attached.  Note this is after a reboot, with the cachetwo pool in a normal state

catan-diagnostics-20250128-2246.zip

Post new ones before rebooting if it happens again.

10 hours ago, JorgeB said:

Post new ones before rebooting if it happens again.

I tried, it locked up and wouldn't generate the file.

You can enable the syslog server and post that if it happens again.

syslog is attached.

syslog

Jan 31 01:34:31 catan kernel: hpsa 0000:03:00.0: Acknowledging event: 0x80000012 (HP SSD Smart Path configuration change)
Jan 31 01:34:31 catan kernel: hpsa 0000:03:00.0: scsi 1:0:8:0: updated Direct-Access     ATA      SATA SSD         PHYS DRV SSDSmartPathCap- En- Exp=1

 

Looks more like a power/connection issue, could also be controller related, I would recommend using an onboard SATA port instead, with new cables.

 

Unrelated to the issue, but there appears to be a docker container constantly restarting, check their uptimes.