- Minor
I’m experiencing a recurring issue where my NVMe drive, used as a ZFS cache pool, intermittently fails. This causes the kernel to disable the device, ZFS to suspend the pool, and Unraid to enter a looping state where it cannot cleanly unmount user shares or datasets. This loop ultimately forces me to perform a hard reset, resulting in an unclean shutdown.
Key Details:
Initial NVMe Drive Failure:
The NVMe drive starts by showing timeout errors (I/O timeout) in the kernel logs, followed by repeated reset attempts. Eventually, the kernel disables the device, reporting errors like Device not ready; aborting reset and Disabling device after reset failure.
Once the device is disabled, ZFS encounters uncorrectable I/O errors (error=5), which causes it to suspend the affected pool.
ZFS Pool Suspension:
After the drive failure, ZFS reports that the cache pool is suspended and cannot process I/O. Logs show errors such as:
WARNING: Pool 'cache' has encountered an uncorrectable I/O failure and has been suspended.
Repeated messages about specific datasets being unavailable, e.g., cannot open 'cache': pool I/O is currently suspended.
Unmounting Loop and Forced Shutdown:
Following the failure, Unraid attempts to unmount user shares and ZFS datasets but fails repeatedly with "resource busy" errors:
umount: /mnt/user: target is busy
rmdir: failed to remove '/mnt/user': Device or resource busy
This looping behavior prevents the system from shutting down cleanly. I am forced to perform a hard reset, leading to an unclean shutdown.
Behavior After Reboot:
Rebooting temporarily resolves the issue, with the NVMe drive and the ZFS pool working normally for some time before the problem repeats.
After stopping the Array, this is what keeps looping:
Jan 27 18:53:39 [SERVER_NAME] root: cannot unmount '/mnt/[DATASET_1]': pool or dataset is busy
Jan 27 18:53:39 [SERVER_NAME] emhttpd: shcmd (816026): exit status: 1
Jan 27 18:53:39 [SERVER_NAME] emhttpd: shcmd (816027): umount /mnt/user
Jan 27 18:53:39 [SERVER_NAME] root: umount: /mnt/user: target is busy.
Jan 27 18:53:39 [SERVER_NAME] emhttpd: shcmd (816027): exit status: 32
Jan 27 18:53:39 [SERVER_NAME] emhttpd: shcmd (816028): rmdir /mnt/user
Jan 27 18:53:39 [SERVER_NAME] root: rmdir: failed to remove '/mnt/user': Device or resource busy
Jan 27 18:53:39 [SERVER_NAME] emhttpd: shcmd (816028): exit status: 1
Jan 27 18:53:39 [SERVER_NAME] emhttpd: shcmd (816031): /usr/local/sbin/update_cron
Jan 27 18:53:39 [SERVER_NAME] emhttpd: Retry unmounting user share(s)...
Jan 27 18:53:44 [SERVER_NAME] emhttpd: shcmd (816032): /usr/sbin/zfs unmount -a
Jan 27 18:53:44 [SERVER_NAME] root: cannot open '[POOL_NAME]': pool I/O is currently suspended
Jan 27 18:53:44 [SERVER_NAME] root: cannot open '[POOL_NAME]/[DATASET_2]': pool I/O is currently suspended
Jan 27 18:53:44 [SERVER_NAME] root: cannot open '[POOL_NAME]/[DATASET_3]': pool I/O is currently suspended
Jan 27 18:53:44 [SERVER_NAME] root: cannot open '[POOL_NAME]/[DATASET_4]': pool I/O is currently suspended
Jan 27 18:53:44 [SERVER_NAME] root: cannot unmount '/mnt/[DATASET_1]': pool or dataset is busy
Jan 27 18:53:44 [SERVER_NAME] emhttpd: shcmd (816032): exit status: 1
Jan 27 18:53:44 [SERVER_NAME] emhttpd: shcmd (816033): umount /mnt/user
Jan 27 18:53:44 [SERVER_NAME] root: umount: /mnt/user: target is busy.
Jan 27 18:53:44 [SERVER_NAME] emhttpd: shcmd (816033): exit status: 32
Jan 27 18:53:44 [SERVER_NAME] emhttpd: shcmd (816034): rmdir /mnt/user
Jan 27 18:53:44 [SERVER_NAME] root: rmdir: failed to remove '/mnt/user': Device or resource busy
Jan 27 18:53:44 [SERVER_NAME] emhttpd: shcmd (816034): exit status: 1
Jan 27 18:53:44 [SERVER_NAME] emhttpd: shcmd (816037): /usr/local/sbin/update_cron
Jan 27 18:53:44 [SERVER_NAME] emhttpd: Retry unmounting user share(s)...
Jan 27 18:53:49 [SERVER_NAME] emhttpd: shcmd (816038): /usr/sbin/zfs unmount -a
Jan 27 18:53:49 [SERVER_NAME] root: cannot open '[POOL_NAME]': pool I/O is currently suspended
Jan 27 18:53:49 [SERVER_NAME] root: cannot open '[POOL_NAME]/[DATASET_2]': pool I/O is currently suspended
Jan 27 18:53:49 [SERVER_NAME] root: cannot open '[POOL_NAME]/[DATASET_3]': pool I/O is currently suspended
Jan 27 18:53:49 [SERVER_NAME] root: cannot open '[POOL_NAME]/[DATASET_4]': pool I/O is currently suspended
Jan 27 18:53:49 [SERVER_NAME] root: cannot unmount '/mnt/[DATASET_1]': pool or dataset is busy
Jan 27 18:53:49 [SERVER_NAME] emhttpd: shcmd (816038): exit status: 1
Jan 27 18:53:49 [SERVER_NAME] emhttpd: shcmd (816039): umount /mnt/user
Jan 27 18:53:49 [SERVER_NAME] root: umount: /mnt/user: target is busy.
Jan 27 18:53:49 [SERVER_NAME] emhttpd: shcmd (816039): exit status: 32
Jan 27 18:53:49 [SERVER_NAME] emhttpd: shcmd (816040): rmdir /mnt/user
Jan 27 18:53:49 [SERVER_NAME] root: rmdir: failed to remove '/mnt/user': Device or resource busy
Jan 27 18:53:49 [SERVER_NAME] emhttpd: shcmd (816040): exit status: 1
Jan 27 18:53:50 [SERVER_NAME] emhttpd: shcmd (816043): /usr/local/sbin/update_cron
Jan 27 18:53:50 [SERVER_NAME] emhttpd: Retry unmounting user share(s)...
I don't know if it is a bad NVMe or what, but either way I think Unraid needs to handle this error better to be able to gracefully reset/shutdown. I'm still looking into this issue, and will report back if it is a hardware (NVMe) problem.
Smart test (short) shows no errors with the NVMe.
Self-test status: No self-test in progress
Num Test_Description Status Power_on_Hours Failing_LBA NSID Seg SCT Code
0 Short Completed without error 3684 - - - - -
1 Short Completed without error 3342 - - - - -
2 Short Aborted: Self-test command 0 - - - - -
3 Extended Completed without error 0 - - - - -
4 Short Completed without error 0 - - - - -
5 Short Aborted: Controller Reset 0 - - - - -
6 Short Completed without error 0 - - - - -
7 Short Aborted: Self-test command 0 - - - - -
8 Short Aborted: Self-test command 0 - - - - -
9 Short Aborted: Controller Reset 0 - - - - -
10 Short Aborted: Self-test command 0 - - - - -
11 Short Aborted: Self-test command 0 - - - - -
12 Short Aborted: Controller Reset 0 - - - - -
13 Short Aborted: Self-test command 0 - - - - -
14 Short Completed without error 0 - - - - -