Hello,

After long inactivity i founds some time to tinker with my server again. Sadly, this kicked of a series of unlucky events which may have lead to data loss for me.

This is a request for help to maybe still salvage some data. My ZFS knowledge is very limited and I'm most likely missing something.

I have a very simple setup with two drives and a SSD as cache pool. The cache hosts the appdata of my containers. I had automated backups of appdata to the array running for a bit but it stopped working end of last year. I am solely to blame for not fixing backups in time but life sometimes just gets in the way. The server was mostly idle/unused.

Short summary of what happened:

• The system was running fine

• I decided to finally tackle backups.

• I noticed some warnings, checked dmesg and noticed some errors regarding squashfs and xz, sounded like the USB drive was fried

• Got a new drive, put unraid on it and restored my config there

• System booted without errors in dmesgbut now my cache pool shows an error: "Unmountable: wrong or no filesystem"

I searched for the error and most post led me to believe that my data is gone now. Some things i have tried from other posts:

root@Tower:~# zpool list
no pools available
root@Tower:~# zpool import
pool: cache
id: 7139977111747743935
state: ONLINE
status: Some supported features are not enabled on the pool.
(Note that they may be intentionally disabled if the
'compatibility' property is set.)
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit 'zpool upgrade'.
config:
cache ONLINE
sdb1 ONLINE
root@Tower:~# zpool import cache
cannot import 'cache': I/O error
Destroy and re-create the pool from
a backup source.
root@Tower:~# zpool import -d /dev/disk/by-id/ata-Crucial_CT250MX200SSD1_1529101BF7ED
pool: cache
id: 7139977111747743935
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
config:
cache UNAVAIL insufficient replicas
ata-Crucial_CT250MX200SSD1_1529101BF7ED UNAVAIL invalid labelAny advice would be appreciated.

tower-diagnostics-20250604-1654.zip

Edited June 4, 20251 yr by buxel

Try importing the pool in read-only mode, but it typically doesn't work with the error:

zpool import -o readonly=on cache

You are right, same result

root@Tower:~# zpool import -o readonly=on cache
cannot import 'cache': I/O error
        Destroy and re-create the pool from
        a backup source.

(I forgot to attach the diagnostics before. Edited my post and added them now)

I came across zpool clear in this thread: https://forum.proxmox.com/threads/zfs-i-o-error.33670/

SMART reports no errors for the drive. Are there any non-obvious consequences of running it on my pool?

Edited June 4, 20251 yr by buxel

AFAIK zpool clear on works with an imported pool.

As a latch ditch effort, try: zpool import -FX cache

zpool clear does not find the pool. I think it needs to be imported for that?
zpool import -FX cache took a while and then crashed the server.

A few more questions / straws

• Is there any way i can get a better understanding what is going wrong? I/O error is pretty generic. Googling was inconclusive, ranging from ZFS bugs to dying hardware.

• I understand an SSD can die but the timing is just odd and it has not been used much. SMART is also fine.

• Here is where my lack of knowledge becomes obvious: could i mount the ZFS filesystem somehow manually "outside" the pool?

• Does "invalid label" refer to the disk label?

  # zpool import -d /dev/disk/by-id/ata-Crucial_CT250MX200SSD1_1529101BF7ED
  config:
          cache                                      UNAVAIL  insufficient replicas
            ata-Crucial_CT250MX200SSD1_1529101BF7ED  UNAVAIL  invalid label

Edited June 4, 20251 yr by buxel

11 hours ago, buxel said:

Is there any way i can get a better understanding what is going wrong?

Difficult to say for sure, you may still be able to recover the data using a file recovery app, like UFS explorer, the free trial should show if it can recover anything.

And if you like more advance diagnose of what happened, I would suggest posting on the zfs mailing list or opening an issue on OpenZFS' GitHub

Thanks for the tip, i will give it a try.

I managed to pull some data from the drive.

I do not recommend any of this (because evidently, I don't know what I'm doing ;) ) but will post it for any poor soul in a similar situation.

Source: https://serverfault.com/questions/1001813/zpool-fails-to-import-raidz3-pool-despite-sufficient-replicas-available

# Disable ZFS data verification. Flags will be reset after reboot
echo 0 >/sys/module/zfs/parameters/spa_load_verify_data
echo 0 >/sys/module/zfs/parameters/spa_load_verify_metadata
# Mount the pool readonly
zpool import -o readonly=on cache -f