VERIFY3 zfs pool corrupted

[Radzu]

A few days ago something happened and I couldn't log in to my server so I turned it off hard.

Then I had problems with one of my VMs.

Finally after reboot the server was stuck on Mountunig array.

 

I connected the monitor and saw an error while mounting:

Unraid login: VERIFY3(size <= SPA_MAXBLOCKSIZE) failed (24155136 <= 16777216)

PANIC at all.c:188:abd_alloc()

 

Even though I have many pools, I finally found out that the problem is with the ZFS pool consisting of 2x 1TB disks.

 

I saw that many people had a similar problem and many of them solved the problem by restoring data from a backup.
I made the assumption (now I see that it was wrong) that if the pool is in RAID1, it protects it.

 

I tried removing the disks from the server one at a time, but both disks produce the same error.

 

Now I'm a bit out of ideas.

 

I've seen a lot of advice on mounting a drive as read-only, but my knowledge is too poor to understand how to boot the system for these commands to work.

 

I know that there is some mode that can be forced at system startup.

I know that Array can be run in Maintenance Mode.

 

When disabled array tries to execute the command:

zpool import -o readonly=on nvme

I have the same symptoms as when staring array (VERIFY3 ...).

 

Does anyone have an idea how I can recover my data?

 

Diagnostics were done before starting array, as it is not possible later.

 

 

 

unraid-diagnostics-20240502-0010.zip

[JorgeB]

See if it mounts read only, with the array stopped type

zpool import -o readonly=on nvme

 

If it works start the array, the pool will show unmountable in the GUI but the data will be under /mnt/nvme, backup and re-format, and keep in mind that a mirror is not a backup, it just provides redundancy if a device fails, it won't protect against file system corruption, or file deletion, or ransomware, etc.

[Radzu]

Hello after a long weekend.

 

As I wrote in my first post, the "zpool import -o readonly=on nvme" command causes hang and does not mount the disk so data cannot be copied.

 

I am a bit devastated by the information that I bought two disks specifically to protect against data loss, and a simple system error resulted in the inability to access both disks.

Once I restore the server to its original state, I will certainly not use ZFS pools again.

 

I connected one of the drives to another computer and am trying to recover data with some programs, but without purchasing a license in this way it is unlikely to be recovered.

 

 

 

 

[JorgeB]

51 minutes ago, Radzu said:

As I wrote in my first post, the "zpool import -o readonly=on nvme" command causes hang and does not mount the disk so data cannot be copied.

Sorry, missed that, in that case only a file recovery util like UFS explorer may be able to help.

 

 

 

 

[Radzu]

I purchased a license for UFS Standard Recovery and recovered most of the files.

I would like to avoid a similar situation in the future. Is using btrfs safer?
Additionally, I plan to configure Easy Backup to Array.

[JorgeB]

41 minutes ago, Radzu said:

Is using btrfs safer?

Typically zfs is considered more reliable than btrfs, but filesystem corruption can still happen, especially if there are any hardware issues, and a mirror is no substitute for backups.