December 14, 20241 yr Ok, the exact same thing just happened again. sdn (a device in my zpool) was spontaneously added as a hot spare in my zfs pool after a reboot -> Instant kaboom to my boot USB. Since I haven't rebooted I was able to use the Flash Backup button to download my configs luckily. I have not rebooted yet. I actually don't really know how to recover from this either, since the same drive is somehow in both the zpool itself and the hot spare group. I'm guessing if I do 'zpool remove zfs /dev/sdn1' that the drive will be removed from the actual pool itself as well as the hot spare group, which is not something I am comfortable with. Here's the output of `zpool status`: root@Tower:~# zpool status pool: zfs state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 16:17:34 with 0 errors on Tue Dec 3 16:17:35 2024 config: NAME STATE READ WRITE CKSUM zfs ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdj1 ONLINE 0 0 0 sdg1 ONLINE 0 0 0 sdh1 ONLINE 0 0 0 sdi ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 sdk1 ONLINE 0 0 0 sdl1 ONLINE 0 0 0 sdm1 ONLINE 0 0 0 sdn1 ONLINE 0 0 0 spares sdn FAULTED corrupted data errors: 1 data errors, use '-v' for a list Here's the same wipefs command being run as last time: Dec 13 18:35:19 Tower emhttpd: mounting /mnt/zfs Dec 13 18:35:19 Tower emhttpd: shcmd (120): mkdir -p /mnt/zfs Dec 13 18:35:19 Tower emhttpd: shcmd (121): /usr/sbin/zpool import -f -N -o autoexpand=on -d /dev/sdj1 -d /dev/sdg1 -d /dev/sdh1 -d /dev/sdi1 -d /dev/sdk1 -d /dev/sdl1 -d /dev/sdm1 -d /dev/sdn1 17062760427266059542 zfs Dec 13 18:35:21 Tower root: error: /webGui/include/InitCharts.php: wrong csrf_token Dec 13 18:35:21 Tower root: error: /webGui/include/InitCharts.php: wrong csrf_token Dec 13 18:35:21 Tower root: error: /webGui/include/InitCharts.php: wrong csrf_token Dec 13 18:35:21 Tower root: error: /webGui/include/InitCharts.php: wrong csrf_token Dec 13 18:35:26 Tower emhttpd: shcmd (122): /usr/sbin/zfs set mountpoint=/mnt/zfs zfs Dec 13 18:35:26 Tower emhttpd: shcmd (123): /usr/sbin/zfs set atime=off zfs Dec 13 18:35:27 Tower emhttpd: shcmd (124): /usr/sbin/zfs mount zfs Dec 13 18:35:27 Tower emhttpd: shcmd (125): /usr/sbin/zpool set autotrim=off zfs Dec 13 18:35:27 Tower emhttpd: shcmd (126): /usr/sbin/zfs set compression=on zfs Dec 13 18:35:27 Tower emhttpd: /mnt/zfs root profile: raidz1 Dec 13 18:35:27 Tower emhttpd: /mnt/zfs root groups: 2 Dec 13 18:35:27 Tower emhttpd: /mnt/zfs root width: 4 Dec 13 18:35:27 Tower emhttpd: /usr/sbin/zpool status -PL zfs 2>&1 Dec 13 18:35:27 Tower emhttpd: pool: zfs Dec 13 18:35:27 Tower emhttpd: state: ONLINE Dec 13 18:35:27 Tower emhttpd: status: One or more devices has experienced an error resulting in data Dec 13 18:35:27 Tower emhttpd: #011corruption. Applications may be affected. Dec 13 18:35:27 Tower emhttpd: action: Restore the file in question if possible. Otherwise restore the Dec 13 18:35:27 Tower emhttpd: #011entire pool from backup. Dec 13 18:35:27 Tower emhttpd: see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A Dec 13 18:35:27 Tower emhttpd: scan: scrub repaired 0B in 16:17:34 with 0 errors on Tue Dec 3 16:17:35 2024 Dec 13 18:35:27 Tower emhttpd: config: Dec 13 18:35:27 Tower emhttpd: #011NAME STATE READ WRITE CKSUM Dec 13 18:35:27 Tower emhttpd: #011zfs ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 raidz1-0 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdj1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdg1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdh1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdi1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 raidz1-1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdk1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdl1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdm1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdn1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011spares Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdn1 FAULTED corrupted data Dec 13 18:35:27 Tower emhttpd: shcmd (127): /sbin/wipefs -af /dev/sda1 Dec 13 18:35:27 Tower root: /dev/sda1: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20 Dec 13 18:35:27 Tower root: /dev/sda1: 1 byte was erased at offset 0x00000000 (vfat): eb Dec 13 18:35:27 Tower root: /dev/sda1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa Dec 13 18:35:27 Tower emhttpd: writing MBR on disk (sda) with partition 1 offset 64, erased: 0 Dec 13 18:35:28 Tower emhttpd: re-reading (sda) partition table Dec 13 18:35:28 Tower emhttpd: error: mkmbr, 2196: Device or resource busy (16): ioctl BLKRRPART: /dev/sda Dec 13 18:35:28 Tower emhttpd: shcmd (128): udevadm settle Dec 13 18:35:33 Tower emhttpd: shcmd (129): /usr/sbin/zpool replace -f zfs /dev/sdn1 /dev/sda1 Dec 13 18:35:33 Tower root: cannot open '/dev/sda1': Device or resource busy Dec 13 18:35:33 Tower root: cannot replace /dev/sdn1 with /dev/sda1: /dev/sda1 is busy, or device removal is in progress Dec 13 18:35:33 Tower emhttpd: shcmd (129): exit status: 255 Dec 13 18:35:33 Tower emhttpd: errors: 1 data errors, use '-v' for a list Dec 13 18:35:33 Tower emhttpd: shcmd (130): /usr/sbin/zfs mount -a Dec 13 18:35:34 Tower emhttpd: shcmd (131): sync Dec 13 18:35:34 Tower emhttpd: shcmd (132): mkdir /mnt/user0 Dec 13 18:35:34 Tower emhttpd: shcmd (133): /usr/local/bin/shfs /mnt/user0 -disks 14 -o default_permissions,allow_other,noatime Dec 13 18:35:34 Tower shfs: FUSE library version 3.12.0 Dec 13 18:35:34 Tower emhttpd: shcmd (134): mkdir /mnt/user Dec 13 18:35:34 Tower emhttpd: shcmd (135): /usr/local/bin/shfs /mnt/user -disks 15 -o default_permissions,allow_other,noatime -o remember=330 Dec 13 18:35:34 Tower shfs: FUSE library version 3.12.0 Dec 13 18:35:34 Tower emhttpd: shcmd (138): /usr/local/sbin/update_cron Dec 13 18:35:35 Tower emhttpd: Unregistered Flash device error (ENOFLASH3) Dec 13 18:35:37 Tower root: Delaying execution of fix common problems scan for 10 minutes This is not something that can wait until 7.0.0 to be fixed, this needs a patch on 6.12 ASAP. Edited December 14, 20241 yr by sazrocks
December 14, 20241 yr Community Expert as stated ealeir in this thread. Quote It's a bug, but it's a very corner case because it requires that the user has a pool with an invalid device, but like mentioned, it should not happen with v7. It would have been better if you made a new topic post...
December 14, 20241 yr Community Expert 7 hours ago, sazrocks said: the exact same thing just happened again. sdn (a device in my zpool) was spontaneously added as a hot spare in my zfs pool after a reboot Syslog-previous, shows that you already had a spare in the pool, before this boot, and it wasn't correctly assigned to the pool in the GUI: Dec 13 17:56:31 Tower emhttpd: #011NAME STATE READ WRITE CKSUM Dec 13 17:56:31 Tower emhttpd: #011zfs ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /r..1/...-0 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /dev/sde1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /dev/sdb1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /dev/sdc1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /dev/sdd1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /r..1/...-1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /dev/sdf1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /dev/sdh1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /dev/sdi1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011 /dev/sdj1 ONLINE 0 0 0 Dec 13 17:56:31 Tower emhttpd: #011spares Dec 13 17:56:31 Tower emhttpd: #011 /dev/sdn1 AVAIL You cannot do that, if you want to use a spare, it must be assigned to the pool in the GUI. sdn was this disk that was unassigned at the time: Dec 13 17:54:09 Tower emhttpd: TOSHIBA_HDWG180_X040A052FBEG (sdn) 512 15628053168 Dec 13 17:56:35 Tower unassigned.devices: Disk with ID 'TOSHIBA_HDWG180_X040A052FBEG (dev2)' is not set to auto mount. At the next boot, the devices IDs changed, as they can do, causing the issue, so while this is a bug, it won't happen if you have a spare correctly assigned to the pool.
December 14, 20241 yr Community Expert Now to try to remove the device, sdn1 is also a raidz member, but since that cannot be removed, it should either remove the spare or fail to remove the device, try without /dev/ first zpool remove zfs sdn1
December 16, 20241 yr @JorgeBOnce again thank you for your help. I found your excellent post on adding a hot spare correctly (so that unraid/the UI recognizes the drive) and followed it, so hopefully this doesn't come up again. Sent you some beer money.
December 16, 20241 yr Community Expert 4 hours ago, sazrocks said: Once again thank you for your help. I found your excellent post on adding a hot spare correctly (so that unraid/the UI recognizes the drive) and followed it Please note that spares don't automatically kick in yet if a pool disk fails, that why they are currently not supported with v7, it can give the user a false sense of security, with v6.12 you can still use them, but will need to manually start the replacement. 4 hours ago, sazrocks said: Sent you some beer money. Thanks!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.