PANIC on one of my cache pools - blkptr error

[Dragonwyntir]

Hello fine folks,

 

This afternoon, my Unraid server seized, becoming unresponsive even at the terminal.  I restarted it, and it came back up, but when it begins to start the array, it hits one of my cache pools and stops there, never finishing mounting.  Unfortunately that has a few critical items on it, so I am not sure how to proceed.  I set it up as a ZFS pool a little while ago, with three 1TB SSDs, configured as "raidz".  The first of the three disks seems to be where it stops mounting, but that may be anecdotal in the GUI.  I do see in the attributes of that disk that *Reallocated sector count* has a raw value of 1, while the others are at 0.

I've attached the most recent logs - hopefully someone has an idea on how I can get this back up and running.  Of course I'm leaving out of town for four days starting tomorrow at 11AM, so the timing is *chef's kiss* - happy to plug away at it tonight if anyone has any thoughts.

 

Thanks everyone for reading!

dragonvault-syslog-20230917-0026.zip

[Dragonwyntir]

Well damn, I came across another person with the exact same issue. 

Starting array after upgrading to 6.12.1 stuck at Mounting ZFS Cache Pool (VERIFY3 failed) - General Support - Unraid

I'm not sure if I'll be able to follow the same steps, but I don't currently have a backup of what's on that cache pool, so I'd love to get any pointers about getting everything off of there before blowing it all away.  And should I stay away from ZFS for the cache pool and head back to btrfs?

[JorgeB]

Try importing the pool read only:

zpool import -o readonly=on ssd_pool

 

[Dragonwyntir]

Hi Jorge,

 

That worked - the array is up and I can poke around now (including seeing what is on the ssd_pool) - it shows that the pool is Unmountable in the Main Unraid tab, and it gives me options to format all three.

[JorgeB]

18 hours ago, Dragonwyntir said:

it shows that the pool is Unmountable in the Main Unraid tab, and it gives me options to format all three.

That's normal, but you can copy what you need somewhere else then re-format the pool.

[Dragonwyntir]

Hi Jorge,

 

Good thinking on this, but I might be missing a step somewhere.  I was able to back up (copy) the data off of the pool, and I'm now ready to format the pool.  When I try to do that (check "Yes I want to do this" and click "Format"), it thinks for awhile and then errors out.  Do I need to take the pool out of read-only mode?

[JorgeB]

5 hours ago, Dragonwyntir said:

Do I need to take the pool out of read-only mode?

You need to export the pool first:

 

zpool export ssd_pool

 

[Dragonwyntir]

Hi Jorge,

 

Thanks for your help on this.  I was able to get the data backed up, the pool reformatted and then move the data back.  Really appreciate the responses!

 

Question: I moved back to btrfs for now because of how potentially catastrophic that was - is that me being too cautious?

[JorgeB]

13 hours ago, Dragonwyntir said:

I moved back to btrfs for now because of how potentially catastrophic that was - is that me being too cautious?

It's a valid option, though zfs is usually considered more robust, if you also have issues with btrfs it may indicate a hardware issue.