I got tired of cache disks failing (about once a year) and restoring from appdata and VM backups, some of which always failed to work, so I made a ZFS cache pool of three mirrored disks.

I just got finished setting everything back up, when some dockers stopped working today. I was unable to stop the failed dockers, so I killed the docker service and shut down the array, but due to active processes on the cache pool which did not respond to kill -9 i had to hard reset the server. Starting up again the cache pool of three disks is unmountable. All three disks complete SMART short tests without error.

Is there anything I can do to save the pool?

Please post the diagnostics and the output from zpool import

root@Shrek:~# zpool import

pool: cachepool

id: 624089324922130285

state: ONLINE

action: The pool can be imported using its name or numeric identifier.

config:

cachepool ONLINE

mirror-0 ONLINE

sdi1 ONLINE

sdh1 ONLINE

sdl1 ONLINE

shrek-diagnostics-20250705-1203.zip

Post the output from

zpool import cachepool -o readonly=on

If that fails, and it will likely will, post the output from

zdb -l /dev/sdi

zdb -l /dev/sdh

zdb -l /dev/sdl

root@Shrek:~# zpool import cachepool -o readonly=on

cannot import 'cachepool': I/O error

Destroy and re-create the pool from

a backup source.

root@Shrek:~# zdb -l /dev/sdi

failed to unpack label 0

failed to unpack label 1

failed to unpack label 2

failed to unpack label 3

root@Shrek:~# zdb -l /dev/sdh

failed to unpack label 0

failed to unpack label 1

failed to unpack label 2

failed to unpack label 3

root@Shrek:~# zdb -l /dev/sdl

failed to unpack label 0

failed to unpack label 1

failed to unpack label 2

failed to unpack label 3

root@Shrek:~#

Sorry, wrong commands, should be:

zdb -l /dev/sdi1

zdb -l /dev/sdh1

zdb -l /dev/sdl1

root@Shrek:~# zdb -l /dev/sdi1

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'cachepool'

state: 0

txg: 613982

pool_guid: 624089324922130285

errata: 0

hostname: 'Shrek'

top_guid: 3293130853435037943

guid: 11309557061343029751

vdev_children: 1

vdev_tree:

type: 'mirror'

id: 0

guid: 3293130853435037943

whole_disk: 0

metaslab_array: 132

metaslab_shift: 34

ashift: 12

asize: 8001558413312

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 11309557061343029751

path: '/dev/sdi1'

whole_disk: 0

DTL: 96

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 15152360535842760272

path: '/dev/sdh1'

whole_disk: 0

DTL: 88

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 17771728439540138945

path: '/dev/sdl1'

whole_disk: 0

DTL: 97

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

labels = 0 1 2 3

root@Shrek:~# zdb -l /dev/sdh1

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'cachepool'

state: 0

txg: 613982

pool_guid: 624089324922130285

errata: 0

hostname: 'Shrek'

top_guid: 3293130853435037943

guid: 15152360535842760272

vdev_children: 1

vdev_tree:

type: 'mirror'

id: 0

guid: 3293130853435037943

whole_disk: 0

metaslab_array: 132

metaslab_shift: 34

ashift: 12

asize: 8001558413312

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 11309557061343029751

path: '/dev/sdi1'

whole_disk: 0

DTL: 96

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 15152360535842760272

path: '/dev/sdh1'

whole_disk: 0

DTL: 88

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 17771728439540138945

path: '/dev/sdl1'

whole_disk: 0

DTL: 97

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

labels = 0 1 2 3

root@Shrek:~# zdb -l /dev/sdl1

------------------------------------

LABEL 0

------------------------------------

version: 5000

name: 'cachepool'

state: 0

txg: 613982

pool_guid: 624089324922130285

errata: 0

hostname: 'Shrek'

top_guid: 3293130853435037943

guid: 17771728439540138945

vdev_children: 1

vdev_tree:

type: 'mirror'

id: 0

guid: 3293130853435037943

whole_disk: 0

metaslab_array: 132

metaslab_shift: 34

ashift: 12

asize: 8001558413312

is_log: 0

create_txg: 4

children[0]:

type: 'disk'

id: 0

guid: 11309557061343029751

path: '/dev/sdi1'

whole_disk: 0

DTL: 96

create_txg: 4

children[1]:

type: 'disk'

id: 1

guid: 15152360535842760272

path: '/dev/sdh1'

whole_disk: 0

DTL: 88

create_txg: 4

children[2]:

type: 'disk'

id: 2

guid: 17771728439540138945

path: '/dev/sdl1'

whole_disk: 0

DTL: 97

create_txg: 4

features_for_read:

com.delphix:hole_birth

com.delphix:embedded_data

com.klarasystems:vdev_zaps_v2

labels = 0 1 2 3

1 minute ago, MHDFreefall said:

txg: 613982

This is the same for all 3 devices, which is good, but not sure why it's failing to mount, possibly some metadata corruption, you can try this, it will try to rewind the pool a few txgs, so if it works you may lose the last few seconds of data:

zpool import -fF -o readonly=on cachepool

If it still fails, post the output from:

tail -n 200 /proc/spl/kstat/zfs/dbgmsg

root@Shrek:~# zpool import -fF -o readonly=on cachepool

cannot import 'cachepool': I/O error

Destroy and re-create the pool from

a backup source.

dbgmsg.txt

So the problem does appear to be corrupt metadata:

1751711042 ffff88829ab09100 spa_misc.c:429:spa_load_note(): spa_load(cachepool, config trusted): spa_load_verify found 1 metadata errors and 1 data errors

1751711042 ffff88829ab09100 spa_misc.c:415:spa_load_failed(): spa_load(cachepool, config trusted): FAILED: spa_load_verify failed [error=5]

You can try to disable ZFS data verification, note that doing this may return corrupt data, but possibly still allow you to backup most of it, then reformat the pool:

echo 0 >/sys/module/zfs/parameters/spa_load_verify_data

echo 0 >/sys/module/zfs/parameters/spa_load_verify_metadata

Then try again importing the pool read-only

zpool import -o readonly=on cachepool

If it works, start the array, pool will still show unmountable on the GUI, but the data should be under /mnt/cachepool, then backup what you can to another disk/pool, when done reboot to reset the ZFS settings.

Great, disk mounted! Thanks a lot for the help. Even if there is an error somewhere most of the files should be ok.

Is there any reason to not trust the disks? I will run a long SMART test for a while to see if any errors pop up. Do you know what can have caused the corrupt data if not disk failure?

I guess there is no advantage to having a three disk mirror when the file system fails. I will make the disks into a two disk mirrored pool, and set up some kind of periodic backup to the last disk in order to have a ready to go but slightly out of date mirror if the cache pool fails again.

Do I need to do anything to turn ZFS data verification back on, or will it revert after the next reboot?

8 minutes ago, MHDFreefall said:

Is there any reason to not trust the disks?

I would not suspect the disks, but if it happens again, there may be an underlying hardware issue, like bad RAM or a controller/firmware issue.

7 minutes ago, MHDFreefall said:

or will it revert after the next reboot?

This.

Perfect, thanks again. Procedure saved to my "unraidhowto.txt" for future reference.