Hi all.

I have following Problem:

I currently adding drives to an existing ZFS pool (Raid Z 2)

Wile adding the last Disk the pools stops because of read errors of a existing drive. I did a smart short test, result is fine. Long test is currently working.
I ordered alrdy a new one, but what is the best way here. Should i "zpool clear" here to finish the expansion or should i directly replace the drive? I Replaced alrdy the cables just in case. I moved the server to a new Case (Jonsbo N5) and thought may i damaged a cable or something.

dmesg | grep -i 'sdh\|disk\|error':

root@Tower:~# dmesg | grep -i 'sdh\|disk\|error'

[ 0.005060] RAMDISK: [mem 0x7dba3000-0x7fffffff]

[ 5.167048] scsi 2:0:0:0: Direct-Access SanDisk Cruzer Force 1.00 PQ: 0 ANSI: 6

[ 5.182183] sd 2:0:0:0: [sda] Attached SCSI removable disk

[ 67.307763] sd 3:0:3:0: [sde] Attached SCSI disk

[ 67.309678] sd 3:0:0:0: [sdb] Attached SCSI disk

[ 67.309709] sd 3:0:2:0: [sdd] Attached SCSI disk

[ 67.309793] sd 3:0:1:0: [sdc] Attached SCSI disk

[ 67.349296] sd 4:0:2:0: [sdh] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)

[ 67.349305] sd 4:0:2:0: [sdh] 4096-byte physical blocks

[ 67.357360] sd 4:0:2:0: [sdh] Write Protect is off

[ 67.357365] sd 4:0:2:0: [sdh] Mode Sense: 9b 00 10 08

[ 67.358154] sd 4:0:2:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA

[ 67.384486] sdh: sdh1

[ 67.384566] sd 4:0:2:0: [sdh] Attached SCSI disk

[ 67.395365] sd 4:0:1:0: [sdg] Attached SCSI disk

[ 67.402654] sd 4:0:5:0: [sdk] Attached SCSI disk

[ 67.403624] sd 4:0:4:0: [sdj] Attached SCSI disk

[ 67.411663] sd 4:0:6:0: [sdl] Attached SCSI disk

[ 67.413179] sd 4:0:3:0: [sdi] Attached SCSI disk

[ 67.491856] sd 4:0:7:0: [sdm] Attached SCSI disk

[ 67.929025] sd 4:0:0:0: [sdf] Attached SCSI disk

[ 398.850914] sd 4:0:2:0: [sdh] tag#6892 CDB: opcode=0x8a 8a 00 00 00 00 01 bc 23 a0 d0 00 00 00 b0 00 00

[ 399.732843] sd 4:0:2:0: [sdh] tag#6024 CDB: opcode=0x8a 8a 00 00 00 00 01 bc 23 a0 38 00 00 00 90 00 00

[ 399.732880] sd 4:0:2:0: [sdh] tag#6851 CDB: opcode=0x88 88 00 00 00 00 00 d2 a1 e9 78 00 00 00 08 00 00

[ 404.797835] sd 4:0:2:0: [sdh] tag#6854 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=1s

[ 404.797841] sd 4:0:2:0: [sdh] tag#6854 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00

[ 404.797843] I/O error, dev sdh, sector 7814036992 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2

[ 406.042244] sd 4:0:2:0: [sdh] tag#6858 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=37s

[ 406.042245] sd 4:0:2:0: [sdh] tag#6857 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=37s

[ 406.042251] sd 4:0:2:0: [sdh] tag#6857 CDB: opcode=0x8a 8a 00 00 00 00 01 bc 23 a0 38 00 00 00 90 00 00

[ 406.042251] sd 4:0:2:0: [sdh] tag#6858 CDB: opcode=0x88 88 00 00 00 00 00 d2 a1 e9 78 00 00 00 08 00 00

[ 406.042255] I/O error, dev sdh, sector 7451418680 op 0x1:(WRITE) flags 0x0 phys_seg 5 prio class 2

[ 406.042255] I/O error, dev sdh, sector 3533826424 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

[ 406.042260] zio pool=vault vdev=/dev/sdh1 error=5 type=2 offset=3815126331392 size=73728 flags=2148533418

[ 406.042260] zio pool=vault vdev=/dev/sdh1 error=5 type=1 offset=1809319096320 size=4096 flags=3145856

[ 412.297993] sd 4:0:2:0: [sdh] tag#6889 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=7s

[ 412.298000] sd 4:0:2:0: [sdh] tag#6889 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00

[ 412.298004] I/O error, dev sdh, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

[ 412.298008] Buffer I/O error on dev sdh, logical block 976754624, async page read

[ 413.542329] sd 4:0:2:0: [sdh] tag#6893 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=7s

[ 413.542334] sd 4:0:2:0: [sdh] tag#6893 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 bc 50 00 00 00 10 00 00

[ 413.542337] I/O error, dev sdh, sector 7814036560 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

[ 413.542342] zio pool=vault vdev=/dev/sdh1 error=5 type=1 offset=4000786685952 size=8192 flags=1245377

[ 413.542363] sd 4:0:2:0: [sdh] tag#6894 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=7s

[ 413.542366] sd 4:0:2:0: [sdh] tag#6894 CDB: opcode=0x88 88 00 00 00 00 00 00 00 02 50 00 00 00 10 00 00

[ 413.542368] I/O error, dev sdh, sector 592 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

[ 413.542371] zio pool=vault vdev=/dev/sdh1 error=5 type=1 offset=270336 size=8192 flags=1245377

[ 413.542378] sd 4:0:2:0: [sdh] tag#6895 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=7s

[ 413.542381] sd 4:0:2:0: [sdh] tag#6895 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 ba 50 00 00 00 10 00 00

[ 413.542383] I/O error, dev sdh, sector 7814036048 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

[ 413.542386] zio pool=vault vdev=/dev/sdh1 error=5 type=1 offset=4000786423808 size=8192 flags=1245377

[ 414.811984] sd 4:0:2:0: [sdh] tag#6898 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=7s

[ 414.811989] sd 4:0:2:0: [sdh] tag#6898 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00

[ 414.811995] I/O error, dev sdh, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2

[ 414.812001] zio pool=vault vdev=/dev/sdh1 error=5 type=5 offset=0 size=0 flags=2098304

[ 421.042072] sd 4:0:2:0: [sdh] tag#6062 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=52s

[ 421.042083] sd 4:0:2:0: [sdh] tag#6062 CDB: opcode=0x8a 8a 00 00 00 00 01 bc 23 a0 d0 00 00 00 b0 00 00

[ 421.042086] I/O error, dev sdh, sector 7451418832 op 0x1:(WRITE) flags 0x0 phys_seg 6 prio class 2

[ 421.042091] zio pool=vault vdev=/dev/sdh1 error=5 type=2 offset=3815126409216 size=90112 flags=2148533418

[ 426.042087] sd 4:0:2:0: [sdh] tag#6065 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=9s

[ 426.042092] sd 4:0:2:0: [sdh] tag#6065 CDB: opcode=0x88 88 00 00 00 00 01 d1 c0 be 00 00 00 00 08 00 00

[ 426.042094] I/O error, dev sdh, sector 7814036992 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

[ 426.042099] Buffer I/O error on dev sdh, logical block 976754624, async page read

[ 661.474553] mce: [Hardware Error]: Machine check events logged

[ 661.474559] [Hardware Error]: Corrected error, no action required.

[ 661.474567] [Hardware Error]: CPU:0 (19:21:2) MC18_STATUS[-|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|-]: 0x9c2040000000011b

[ 661.474585] [Hardware Error]: Error Addr: 0x000000016eaf2900

[ 661.474590] [Hardware Error]: IPID: 0x0000009600150f00, Syndrome: 0x36a408000a800d01

[ 661.474599] [Hardware Error]: Unified Memory Controller Ext. Error Code: 0

[ 661.474614] [Hardware Error]: cache level: L3/GEN, tx: GEN, mem-tx: RD

tower-diagnostics-20260102-1315.zip

Edited January 2Jan 2 by Kasjo

Looks more like a power/connection issue, I would replace the cables for that disk, then clear the errors to see if the expansion finishes, and a scrub will be run after the expansion finishes. Check for no more errors.

Yeah ok, i will do this then. Cables are replaced. Pool starts now to resilvering.

***Update:

Ok it stops again for errors here:

I guess so the drive is dead then? Better wait for the replacement then.

As the N5 case has a backplane, i will shutdown the server now and swap the HDD to another slot, just to see if its a Drive or backplane problem. If the error still persists on this drive i think its dead and need a replace.

Faulted again and followed the drive. Im replacing it. Thx for the help.
Can i simply replace it while the expansion is in progress?

Edited January 2Jan 2 by Kasjo

1 hour ago, Kasjo said:

Can i simply replace it while the expansion is in progress?

You can, the expansion will be paused until the replacement is done, and then continue.