I'm experiencing recurrent drive failures in the first vdev of my ZFS pool on a Unraid server. Has anyone encountered a similar issue or have suggestions to resolve it?

 

root@Zeus:~# zpool status
  pool: dumpster
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Apr 21 19:50:36 2024
        3.31T scanned at 86.8G/s, 3.31T issued at 86.8G/s, 76.7T total
        0B resilvered, 4.31% done, no estimated completion time
config:

        NAME        STATE     READ WRITE CKSUM
        dumpster    ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdg1    ONLINE       0     0     0
            sdd1    ONLINE       0     0     0
            sdc1    ONLINE       0     0     0
            sdf1    ONLINE       0     0     2
          raidz1-1  ONLINE       0     0     0
            sde1    ONLINE       0     0     0
            sdm1    ONLINE       0     0     0
            sdl1    ONLINE       0     0     0
            sdj1    ONLINE       0     0     0
          raidz1-2  ONLINE       0     0     0
            sdh1    ONLINE       0     0     0
            sdi1    ONLINE       0     0     0
            sdn1    ONLINE       0     0     0
            sdo1    ONLINE       0     0     0
        cache
          nvme0n1   ONLINE       0     0     0

errors: No known data errors

/////// TWO MINUTES LATER //////

Apr 21 19:52:42 Zeus kernel: sd 5:2:6:0: [sdf] tag#3887 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
Apr 21 19:52:44 Zeus kernel: sd 5:2:6:0: SCSI device is removed
Apr 21 19:52:44 Zeus kernel: zio pool=dumpster vdev=/dev/sdf1 error=5 type=1 offset=654300413952 size=28672 flags=40080ca0
Apr 21 19:52:44 Zeus kernel: zio pool=dumpster vdev=/dev/sdf1 error=5 type=1 offset=654493540352 size=12288 flags=1808a0
Apr 21 19:52:44 Zeus kernel: sd 5:2:6:0: [sdf] Synchronizing SCSI cache
Apr 21 19:52:44 Zeus kernel: sd 5:2:6:0: [sdf] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK
Apr 21 19:52:44 Zeus kernel: megaraid_sas 0000:42:00.0: scanning for scsi5...
Apr 21 19:52:44 Zeus kernel: megaraid_sas 0000:42:00.0: 62950 (767058765s/0x0001/FATAL) - JBOD 06 for PD 22(e0x10/s6) is now OFFLINE
Apr 21 19:52:48 Zeus kernel: zio pool=dumpster vdev=/dev/sdf1 error=5 type=5 offset=0 size=0 flags=100480
Apr 21 19:52:48 Zeus kernel: zio pool=dumpster vdev=/dev/sdf1 error=5 type=5 offset=0 size=0 flags=100480
Apr 21 19:52:55 Zeus kernel: scsi 5:2:6:0: Direct-Access     ATA      WDC  WUH722020AL W108 PQ: 0 ANSI: 6
Apr 21 19:52:55 Zeus kernel: sd 5:2:6:0: Attached scsi generic sg6 type 0
Apr 21 19:52:55 Zeus kernel: sd 5:2:6:0: [sdf] 39063650304 512-byte logical blocks: (20.0 TB/18.2 TiB)
Apr 21 19:52:55 Zeus kernel: sd 5:2:6:0: [sdf] 4096-byte physical blocks
Apr 21 19:52:55 Zeus kernel: sd 5:2:6:0: [sdf] Write Protect is off
Apr 21 19:52:55 Zeus kernel: sd 5:2:6:0: [sdf] Mode Sense: 9b 00 10 08
Apr 21 19:52:55 Zeus kernel: sd 5:2:6:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
Apr 21 19:52:56 Zeus kernel: sdf: sdf1
Apr 21 19:52:56 Zeus kernel: sd 5:2:6:0: [sdf] Attached SCSI disk

/////// new zpool status //////

root@Zeus:~# zpool status
  pool: dumpster
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Apr 21 19:50:36 2024
        76.6T scanned at 0B/s, 76.2T issued at 702M/s, 76.7T total
        0B resilvered, 99.26% done, no estimated completion time
config:

        NAME        STATE     READ WRITE CKSUM
        dumpster    DEGRADED     0     0     0
          raidz1-0  DEGRADED     0     0     0
            sdg1    ONLINE       0     0     0
            sdd1    ONLINE       0     0     0
            sdc1    ONLINE       0     0     0
            sdf1    REMOVED      0     0     0
          raidz1-1  ONLINE       0     0     0
            sde1    ONLINE       0     0     0
            sdm1    ONLINE       0     0     0
            sdl1    ONLINE       0     0     0
            sdj1    ONLINE       0     0     0
          raidz1-2  ONLINE       0     0     0
            sdh1    ONLINE       0     0     0
            sdi1    ONLINE       0     0     0
            sdn1    ONLINE       0     0     0
            sdo1    ONLINE       0     0     0
        cache
          nvme0n1   ONLINE       0     0     0

errors: No known data errors
root@Zeus:~#

 

 

The first time this happened I thought it was a wire issue. Then it happened again so I switched out the raid controller then the backplane. Eventually I said screw it and started from square one. Purchased 4x new Ultrastar DC HC560's (20TB) drives. Wiped all the drives, rebuilt everything from the ground up. Transferred all the files back on (around 75TB). Then it started doing this again.

 

I ran zfs for about 3 maybe 4 years on my unraid server without any problems at all. This only started happening after 6.12 was released and I imported the zpools into unraid for native integration. Not sure if that has anything to do with it or if its complete coincidental.

 

Prior to adding the new drives, the 4x 18tb drives were the ones in the first vdev. Funny enough, even though they're the same 4x drives, now that they are in the 2nd vdev i haven't had a single issue with them. 

 

Any thoughts on what might be up?

Please post the diagnostics.
 

zeus-diagnostics-20240421-1840.zip

17 hours ago, JorgeB said:

Please post the diagnostics.

 

thank you

The drive dropped offline and then reconnected, don't see anything that suggests a software issue, try swapping cables/slots for that drive (or the whole vdev) with another one, then see where the problem follows.

I've swapped cables and slots a number of times now but I'll give it another shot. I ordered some new cables from a different brand; maybe those will make the difference.