Hello, having issues with the primary ZFS pool on a running system (not restart related). [Reviewed other threads, couldn't find exact troubleshooting. But happy to delete and go to another thread if directions are elsewhere] My media_cache zfs pool is 4 mirrors of SSDs. Noticed the system had unmounted the pool. Terminal reports (zpool import) that it's faulted/corrupted data. Immediately standing out is the missing mirror-3. Also, sdm1 appears twice. I'm hoping there's a fix that preserves data, since the pool is transitory and not backed up. Again, happy to review a resource that might be helpful. The link from the screen shot wasn't particularly optimistic.

zion-diagnostics-20251216-1601.zip

2 hours ago, Unready said:

My media_cache zfs pool is 4 mirrors of SSDs.

I'm afraid there are 3 mirrors and 2 striped devices; one of those is corrupt, so the pool is likely gone, post the output from:

zdb -l /dev/sdX1 | grep txg | head -n 1

Replace X with the identifier for all pool members

The pool was originally configured as 4 mirrored pairs.

There's an 8th drive not listed in the zfs import. "sdn1" (seen below, Line 2 in the main tab details excerpt) isn't present in the zfs import results. (Instead, "sdm1" showed up twice also re-pasted below.)

I ran zdb -l for all 8 known devices, only 7 results returned. I confirmed it was "sdn1."

root@Zion:~# zdb -l /dev/sdk1 | grep txg | head -n 1
zdb -l /dev/sdp1 | grep txg | head -n 1
zdb -l /dev/sdc1 | grep txg | head -n 1
zdb -l /dev/sdq1 | grep txg | head -n 1
zdb -l /dev/sdg1 | grep txg | head -n 1
zdb -l /dev/sdm1 | grep txg | head -n 1
zdb -l /dev/sdo1 | grep txg | head -n 1
zdb -l /dev/sdn1 | grep txg | head -n 1
    txg: 4342827
    txg: 4342827
    txg: 4342827
    txg: 4342827
    txg: 4342827
    txg: 4342827
    txg: 4342827
root@Zion:~# zdb -l /dev/sdn1 | grep txg | head -n 1
root@Zion:~# 

My fingers-crossed theory is that somehow it got sdm1 confused for sdn1, which is the actual mirror for sdk1. So it thinks the data is corrupted. Hope I'm making sense. Thanks again.

15 minutes ago, Unready said:

The pool was originally configured as 4 mirrored pairs.

Yep, but something went wrong at pool time creating, since two of the devices are striped without any redundancy.

Post the full zdb output for sdn1: zdb -l /dev/sdn1

Also the output from blkid /dev/sdn1

Sure! There wasn't much output from either.

root@Zion:~# zdb -l /dev/sdn1
cannot open '/dev/sdn1': No such file or directory
root@Zion:~# blkid /dev/sdn1
root@Zion:~# 

12 hours ago, Unready said:

cannot open '/dev/sdn1': No such file or directory

This suggests the device is missing the partition; please post output from fdisk -l /dev/sdn

Sure, copied below. I wasn't sure if it was meant to be "sdn1" so ran it both ways.

root@Zion:~# fdisk -l /dev/sdn
fdisk: cannot open /dev/sdn: Input/output error
root@Zion:~# fdisk -l /dev/sdn1
fdisk: cannot open /dev/sdn1: No such file or directory

It's not just the partition missing; it's the device, post current diagnostics.

Diagnostics attached. Thanks again!

zion-diagnostics-20251227-1348.zip

Dec 25 04:40:01 Zion kernel: critical medium error, dev sdn, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2

Dec 25 04:40:01 Zion kernel: sd 13:0:3:0: [sdn] tag#7128 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s

Dec 25 04:40:01 Zion kernel: sd 13:0:3:0: [sdn] tag#7128 Sense Key : 0x3 [current]

Dec 25 04:40:01 Zion kernel: sd 13:0:3:0: [sdn] tag#7128 ASC=0x11 ASCQ=0x0

Dec 25 04:40:01 Zion kernel: sd 13:0:3:0: [sdn] tag#7128 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00

Dec 25 04:40:01 Zion kernel: critical medium error, dev sdn, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

Dec 25 04:40:01 Zion kernel: Buffer I/O error on dev sdn, logical block 0, async page read

Dec 25 04:40:03 Zion kernel: sd 13:0:3:0: [sdn] tag#7156 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=0s

Dec 25 04:40:03 Zion kernel: sd 13:0:3:0: [sdn] tag#7156 Sense Key : 0x3 [current]

Dec 25 04:40:03 Zion kernel: sd 13:0:3:0: [sdn] tag#7156 ASC=0x11 ASCQ=0x0

Dec 25 04:40:03 Zion kernel: sd 13:0:3:0: [sdn] tag#7156 CDB: opcode=0x88 88 00 00 00 00 00 00 00 00 00 00 00 00 20 00 00

Dec 25 04:40:03 Zion kernel: critical medium error, dev sdn, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2

It's logged as a device problem, but replace both cables for that device and post new diags.

Hello, awaiting some new cabling after the holidays. Still planning on updating. Thanks for the continued help.

Also, a question. Since this cache is mirrored, shouldn't I be able to remove the indicated drive, and the system should account for it using its mirror? I haven't tested yet, but I'd like to confirm my understanding.

7 hours ago, Unready said:

Since this cache is mirrored

It's not totally mirrored, as mentioned before; two of the devices are striped vdevs.