Hello! My system randomly locked up earlier today during a large file transfer. The WebUI was completely non-responsive and while I was able to SSH into the server, restart/shutdown commands failed to execute. I hard shutdown the system using the power button.

 

After rebooting, I started the array in maintenance mode only to encounter my single pool device (2x 500GB SSD in ZFS Mirror) locked and reporting the wrong encryption key. The array itself was successfully unlocked. Subsequent WebUI reboots have not resolved the issue.

 

Diagnostics are attached.

diagnostics.zip

Edited November 13, 20241 yr by shiruken

need other data... It looks like you have encryption setup... zfs commands some data captured in your daig. Thanks will review shortly.

When dealing with ZFS encryption issues after a hard shutdown, the system might be failing to recognize or properly handle the encryption key due to data corruption or a key management glitch. Here’s a step-by-step approach to troubleshoot and hopefully resolve the locked pool issue:

 

Steps to Troubleshoot and Fix

Check Pool Status in Maintenance Mode:

 

zpool status -v

This command should give you detailed information about the pool’s health. Look for any errors, particularly those related to encryption or data corruption.

 

Verify Encryption Key:

Re-enter the encryption key manually to ensure it’s correct. Use the following command to attempt unlocking:

 

zfs load-key -a

If this returns any specific errors about the key, it could indicate corruption in the encryption metadata. If it succeeds, you should be able to access the pool normally.

 

Check for Corruption with ZFS Scrub:

Run a scrub operation on the ZFS pool to identify and attempt to repair any corrupted data. You can do this with:

 

zpool scrub <poolname>
zpool status <poolname>

If it finds and corrects errors, you should be able to unlock and access the pool afterward.

 

Reboot and Unlock via CLI:

Sometimes the WebUI has caching issues or doesn’t fully recognize the pool’s unlocked state. Reboot the server and use the CLI to attempt unlocking again:

 

zfs mount -a

 

Review System Logs:

Check system logs for errors related to the ZFS pool or encryption during the previous shutdown or boot. Use:

Look for any entries indicating disk errors, encryption issues, or kernel errors. These logs may help diagnose if there are underlying hardware or configuration issues.

 

Consider Backup and Recreation:

If issues persist, consider recreating the pool. Before doing this, ensure you have a backup. If the data is still inaccessible but critical, consider reaching out to data recovery services specialized in ZFS with encryption.

 

Preventive Measures

After resolving the issue, consider these steps to avoid future problems:

Battery Backup (UPS): Protect the server from power issues that can cause hard shutdowns.

Regular ZFS Scrubs: Schedule scrubs to detect and address data issues before they escalate.

Backup Key Files: If possible, store your ZFS encryption keys securely in case of issues like this in the future.

ZFS encryption is robust but complex, so any corruption or misconfiguration can lead to situations like this. Taking the above steps should help stabilize your system and protect against future lockups.

 

Unfortunately the output of zpool status is "no pools available".

https://docs.unraid.net/unraid-os/manual/security/data-encryption/

you need to mount them first, then...
zfs mount -a

Q/A Lots of weirdness in the logs. That may have hurt you more in the long run...

 

Nov 13 14:14:39 Gondolin emhttpd: shcmd (4): modprobe zfs

Nov 13 14:14:39 Gondolin kernel: ZFS: Loaded module v2.1.15-1, ZFS pool version 5000, ZFS filesystem version 5

Systems up at this point...


Not sure why this would be happening unless you told the array to stop:
Nov 13 14:22:03 Gondolin emhttpd: Sync filesystems...
Nov 13 14:22:03 Gondolin emhttpd: shcmd (441): sync
Nov 13 14:22:03 Gondolin emhttpd: shcmd (442): /usr/sbin/zfs unmount -a
Nov 13 14:22:03 Gondolin emhttpd: shcmd (444): rm -f /boot/config/plugins/dynamix/mover.cron
Nov 13 14:22:03 Gondolin emhttpd: shcmd (445): /usr/local/sbin/update_cron
Nov 13 14:22:03 Gondolin emhttpd: Unmounting disks...
...
Nov 13 14:25:04 Gondolin emhttpd: shcmd (675): /usr/sbin/zfs mount -a

*You may have deleted import metadata with the recycling bin plugin... used in encryption...

Nov 13 14:30:33 Gondolin Recycle Bin: Stopping Recycle Bin
Nov 13 14:30:33 Gondolin emhttpd: Stopping Recycle Bin...
Nov 13 14:30:33 Gondolin unassigned.devices: Unmounting All Devices...
Nov 13 14:30:33 Gondolin emhttpd: shcmd (698): /etc/rc.d/rc.samba stop
Nov 13 14:30:33 Gondolin wsdd2[7218]: 'Terminated' signal received.
Nov 13 14:30:33 Gondolin wsdd2[7218]: terminating.
Nov 13 14:30:33 Gondolin emhttpd: shcmd (699): rm -f /etc/avahi/services/smb.service
Nov 13 14:30:33 Gondolin avahi-daemon[7400]: Files changed, reloading.
Nov 13 14:30:33 Gondolin avahi-daemon[7400]: Service group file /services/smb.service vanished, removing services.
Nov 13 14:30:34 Gondolin emhttpd: Stopping mover...
Nov 13 14:30:34 Gondolin emhttpd: shcmd (701): /usr/local/sbin/mover stop
Nov 13 14:30:34 Gondolin root: mover: not running
Nov 13 14:30:34 Gondolin emhttpd: Sync filesystems...
Nov 13 14:30:34 Gondolin emhttpd: shcmd (702): sync
Nov 13 14:30:34 Gondolin emhttpd: shcmd (703): /usr/sbin/zfs unmount -a
Nov 13 14:30:34 Gondolin emhttpd: shcmd (704): umount /mnt/user0
Nov 13 14:30:34 Gondolin emhttpd: shcmd (705): rmdir /mnt/user0
Nov 13 14:30:34 Gondolin emhttpd: shcmd (706): umount /mnt/user
Nov 13 14:30:34 Gondolin emhttpd: shcmd (707): rmdir /mnt/user
Nov 13 14:30:34 Gondolin emhttpd: shcmd (709): rm -f /boot/config/plugins/dynamix/mover.cron
Nov 13 14:30:34 Gondolin emhttpd: shcmd (710): /usr/local/sbin/update_cron
Nov 13 14:30:34 Gondolin emhttpd: Unmounting disks...
Nov 13 14:30:34 Gondolin emhttpd: shcmd (711): umount /mnt/disk1
Nov 13 14:30:34 Gondolin kernel: XFS (dm-0): Unmounting Filesystem
Nov 13 14:30:34 Gondolin emhttpd: shcmd (712): rmdir /mnt/disk1
Nov 13 14:30:34 Gondolin emhttpd: shcmd (713): umount /mnt/disk2
Nov 13 14:30:34 Gondolin kernel: XFS (dm-1): Unmounting Filesystem
Nov 13 14:30:34 Gondolin emhttpd: shcmd (714): rmdir /mnt/disk2
Nov 13 14:30:34 Gondolin emhttpd: shcmd (715): umount /mnt/disk3
Nov 13 14:30:34 Gondolin kernel: XFS (dm-2): Unmounting Filesystem

Then it looks like you messed with the superdata that is used on the disks:Nov 13 14:30:35 Gondolin kernel: mdcmd (37): stop 
Nov 13 14:30:35 Gondolin kernel: md1p1: stopping
Nov 13 14:30:35 Gondolin kernel: md2p1: stopping
Nov 13 14:30:35 Gondolin kernel: md3p1: stopping
Nov 13 14:30:35 Gondolin kernel: md: unRAID driver removed
Nov 13 14:30:35 Gondolin emhttpd: shcmd (721): /sbin/modprobe md-mod super=/boot/config/super.dat

*then a new config and import disk? also could have borke encryption keys...
Nov 13 14:30:35 Gondolin emhttpd: import 30 cache device: (sdb) Samsung_SSD_860_EVO_500GB_S59UNJ0N704041B
Nov 13 14:30:35 Gondolin emhttpd: import 31 cache device: (sdc) Samsung_SSD_860_EVO_500GB_S59UNJ0N704046F

Nov 13 14:31:48 Gondolin root: Fix Common Problems Version 2024.11.05
Nov 13 14:31:48 Gondolin root: Fix Common Problems: Other Warning: Array is not started

So yes something is definitely messed up... a more experienced user may be required here...

Did you run the umount commands manualy?
I'm seeing quite a bit of zfs unmount anything in user scritps as you had a weird go file commented out...?

Based on your logs and system details, it looks like the ZFS pool and possibly encryption keys may have been affected by file system or metadata corruption following the hard shutdown. Here’s a more targeted troubleshooting process that could help resolve these issues...

 

Troubleshooting Steps

Check ZFS Encryption Keys and Keyfile:

If you are using a keyfile for encryption, verify it is still intact and accessible.

 

zfs load-key -a

If you encounter an error related to the key, it might mean the keyfile or metadata associated with the encryption could have been affected.

 

Re-import the ZFS Pool:

Since zpool status shows "no pools available," it may be necessary to re-import the pool. This can help if the pool state was corrupted or incorrectly unmounted.

zpool import

if you know the poolname:
 

zpool import -f <poolname>

 

if encypted:
 

zfs load-key <dataset>
zfs mount -a

 

Inspect the super.dat File and Check Disk Assignments:

It appears Unraid’s super.dat configuration file may have been manipulated or reloaded. Confirm that disk assignments and cache configurations are correct in the Unraid GUI, as discrepancies here can prevent pools from mounting correctly.

Go to Tools > New Config in the Unraid GUI, reassign disks (based on prior setup), and start the array, being cautious not to alter encryption or disk order.

Outside of this i'm not sure how to help you recover

Quote

Did you run the umount commands manualy?

I did not. The only commands I tried when the system initially locked up were "poweroff" and "shutdown", neither of which worked. After rebooting and encountering the error, I tried "zpool status" and "zfs list".

 

Unfortunately none of the zfs commands do anything. When the array is mounted, the pool devices both have an "Unmountable: Unsupported or no file system" error message.

I hate to say it. you may be out of luck on the current disk drive data and would need to reformat them zfs mirror with encryption to get them back to a working state...

I've never been in that error area where zfs commands failed or dealt with zfs encryption. these commands our form oracles zfs docs. A more experienced user will need to assist. good luck.

You may need to 3rd party boot a different os to recover the data:

https://www.klennet.com/zfs-recovery/

https://ubuntuforums.org/showthread.php?t=2477381

Yeah, that's what I figured.

 

I'm also worried that the abruptness indicates some kind of hardware failure that reformatting won't address.

If you type:

 

cryptsetup luksOpen /dev/sdb1 sdb1

 

Does it ask for the passphrase?

Quote

Does it ask for the passphrase?

 

Yes. But entering the passphrase results in a "No key available with this passphrase" message.

Edited November 14, 20241 yr by shiruken

That suggests the LUKS headers are damaged, if that is the case the pool is most likely unrecoverable, unless you made a backup of them before this issue, I recommend using encryption only if you really need, it's just another layer that can cause issue.

BTW, try the other pool device, in case that one opens, assuming the pool was redundant.

No luck on the other pool device either.

 

Guess I'll be reformatting! Just for future reference, would it be prudent to use this script to backup the LUKS headers? It seems like that would have potentially allowed me to recover from this error?

I think it might have, though I don't use encryption, so never had to test it.

Something else I noticed in the the uploaded diagnostic logs were segmentation faults:

 

Nov 13 14:26:11 Gondolin kernel: rc.diskinfo[13950]: segfault at 152a38204824 ip 00000000008db485 sp 00007ffd9c015e00 error 4 in php[600000+3b3000] likely on CPU 0 (core 0, socket 0)
Nov 13 14:26:11 Gondolin kernel: Code: 0f 1f 44 00 00 8b 47 04 83 f8 1a 75 0d f6 47 11 02 74 0e 48 8b 7f 08 8b 47 04 a9 10 fc ff ff 74 78 48 8b 7b 18 48 85 ff 74 10 <f6> 47 04 40 75 0a 83 2f 01 75 05 e8 2b 75 fc ff 48 83 c3 20 49 39
Nov 13 14:26:43 Gondolin kernel: rc.diskinfo[16116]: segfault at 146741a04824 ip 00000000008db485 sp 00007ffefde51af0 error 4 in php[600000+3b3000] likely on CPU 0 (core 0, socket 0)
Nov 13 14:26:43 Gondolin kernel: Code: 0f 1f 44 00 00 8b 47 04 83 f8 1a 75 0d f6 47 11 02 74 0e 48 8b 7f 08 8b 47 04 a9 10 fc ff ff 74 78 48 8b 7b 18 48 85 ff 74 10 <f6> 47 04 40 75 0a 83 2f 01 75 05 e8 2b 75 fc ff 48 83 c3 20 49 39
Nov 13 14:27:05 Gondolin kernel: php-fpm[12750]: segfault at 14ac698be740 ip 00000000008b19db sp 00007ffdf30559a0 error 6 in php-fpm[600000+3c0000] likely on CPU 3 (core 3, socket 0)

 

I'm guessing that's indicative of some kind of hardware failure?

I'm having trouble reformatting the drives. I've created a completely new pool and selected ZFS - encrypted as the filesystem but when I start the array and select the option to format the unmountable drives, it errors out saying that the encryption key is wrong.

13 hours ago, shiruken said:

 

I'm guessing that's indicative of some kind of hardware failure?

Could be, you should run memtest.

 

11 hours ago, shiruken said:

I'm having trouble reformatting the drives. I've created a completely new pool and selected ZFS - encrypted as the filesystem but when I start the array and select the option to format the unmountable drives, it errors out saying that the encryption key is wrong.

Try memtest first, wrong key error could also be a RAM issue.