sazrocks

Running v7.0.1, just ran into this exact issue... and this thread. I guess I'll try to make sure I close my tabs after I'm done with them.

@JorgeBOnce again thank you for your help. I found your excellent post on adding a hot spare correctly (so that unraid/the UI recognizes the drive) and followed it, so hopefully this doesn't come up again. Sent you some beer money.

Here is my diagnostics file: tower-diagnostics-20241213-2004.zip

Ok, the exact same thing just happened again. sdn (a device in my zpool) was spontaneously added as a hot spare in my zfs pool after a reboot -> Instant kaboom to my boot USB. Since I haven't rebooted I was able to use the Flash Backup button to download my configs luckily. I have not rebooted yet. I actually don't really know how to recover from this either, since the same drive is somehow in both the zpool itself and the hot spare group. I'm guessing if I do 'zpool remove zfs /dev/sdn1' that the drive will be removed from the actual pool itself as well as the hot spare group, which is not something I am comfortable with. Here's the output of `zpool status`: root@Tower:~# zpool status pool: zfs state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 16:17:34 with 0 errors on Tue Dec 3 16:17:35 2024 config: NAME STATE READ WRITE CKSUM zfs ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 sdj1 ONLINE 0 0 0 sdg1 ONLINE 0 0 0 sdh1 ONLINE 0 0 0 sdi ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 sdk1 ONLINE 0 0 0 sdl1 ONLINE 0 0 0 sdm1 ONLINE 0 0 0 sdn1 ONLINE 0 0 0 spares sdn FAULTED corrupted data errors: 1 data errors, use '-v' for a list Here's the same wipefs command being run as last time: Dec 13 18:35:19 Tower emhttpd: mounting /mnt/zfs Dec 13 18:35:19 Tower emhttpd: shcmd (120): mkdir -p /mnt/zfs Dec 13 18:35:19 Tower emhttpd: shcmd (121): /usr/sbin/zpool import -f -N -o autoexpand=on -d /dev/sdj1 -d /dev/sdg1 -d /dev/sdh1 -d /dev/sdi1 -d /dev/sdk1 -d /dev/sdl1 -d /dev/sdm1 -d /dev/sdn1 17062760427266059542 zfs Dec 13 18:35:21 Tower root: error: /webGui/include/InitCharts.php: wrong csrf_token Dec 13 18:35:21 Tower root: error: /webGui/include/InitCharts.php: wrong csrf_token Dec 13 18:35:21 Tower root: error: /webGui/include/InitCharts.php: wrong csrf_token Dec 13 18:35:21 Tower root: error: /webGui/include/InitCharts.php: wrong csrf_token Dec 13 18:35:26 Tower emhttpd: shcmd (122): /usr/sbin/zfs set mountpoint=/mnt/zfs zfs Dec 13 18:35:26 Tower emhttpd: shcmd (123): /usr/sbin/zfs set atime=off zfs Dec 13 18:35:27 Tower emhttpd: shcmd (124): /usr/sbin/zfs mount zfs Dec 13 18:35:27 Tower emhttpd: shcmd (125): /usr/sbin/zpool set autotrim=off zfs Dec 13 18:35:27 Tower emhttpd: shcmd (126): /usr/sbin/zfs set compression=on zfs Dec 13 18:35:27 Tower emhttpd: /mnt/zfs root profile: raidz1 Dec 13 18:35:27 Tower emhttpd: /mnt/zfs root groups: 2 Dec 13 18:35:27 Tower emhttpd: /mnt/zfs root width: 4 Dec 13 18:35:27 Tower emhttpd: /usr/sbin/zpool status -PL zfs 2>&1 Dec 13 18:35:27 Tower emhttpd: pool: zfs Dec 13 18:35:27 Tower emhttpd: state: ONLINE Dec 13 18:35:27 Tower emhttpd: status: One or more devices has experienced an error resulting in data Dec 13 18:35:27 Tower emhttpd: #011corruption. Applications may be affected. Dec 13 18:35:27 Tower emhttpd: action: Restore the file in question if possible. Otherwise restore the Dec 13 18:35:27 Tower emhttpd: #011entire pool from backup. Dec 13 18:35:27 Tower emhttpd: see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A Dec 13 18:35:27 Tower emhttpd: scan: scrub repaired 0B in 16:17:34 with 0 errors on Tue Dec 3 16:17:35 2024 Dec 13 18:35:27 Tower emhttpd: config: Dec 13 18:35:27 Tower emhttpd: #011NAME STATE READ WRITE CKSUM Dec 13 18:35:27 Tower emhttpd: #011zfs ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 raidz1-0 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdj1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdg1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdh1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdi1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 raidz1-1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdk1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdl1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdm1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdn1 ONLINE 0 0 0 Dec 13 18:35:27 Tower emhttpd: #011spares Dec 13 18:35:27 Tower emhttpd: #011 /dev/sdn1 FAULTED corrupted data Dec 13 18:35:27 Tower emhttpd: shcmd (127): /sbin/wipefs -af /dev/sda1 Dec 13 18:35:27 Tower root: /dev/sda1: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20 Dec 13 18:35:27 Tower root: /dev/sda1: 1 byte was erased at offset 0x00000000 (vfat): eb Dec 13 18:35:27 Tower root: /dev/sda1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa Dec 13 18:35:27 Tower emhttpd: writing MBR on disk (sda) with partition 1 offset 64, erased: 0 Dec 13 18:35:28 Tower emhttpd: re-reading (sda) partition table Dec 13 18:35:28 Tower emhttpd: error: mkmbr, 2196: Device or resource busy (16): ioctl BLKRRPART: /dev/sda Dec 13 18:35:28 Tower emhttpd: shcmd (128): udevadm settle Dec 13 18:35:33 Tower emhttpd: shcmd (129): /usr/sbin/zpool replace -f zfs /dev/sdn1 /dev/sda1 Dec 13 18:35:33 Tower root: cannot open '/dev/sda1': Device or resource busy Dec 13 18:35:33 Tower root: cannot replace /dev/sdn1 with /dev/sda1: /dev/sda1 is busy, or device removal is in progress Dec 13 18:35:33 Tower emhttpd: shcmd (129): exit status: 255 Dec 13 18:35:33 Tower emhttpd: errors: 1 data errors, use '-v' for a list Dec 13 18:35:33 Tower emhttpd: shcmd (130): /usr/sbin/zfs mount -a Dec 13 18:35:34 Tower emhttpd: shcmd (131): sync Dec 13 18:35:34 Tower emhttpd: shcmd (132): mkdir /mnt/user0 Dec 13 18:35:34 Tower emhttpd: shcmd (133): /usr/local/bin/shfs /mnt/user0 -disks 14 -o default_permissions,allow_other,noatime Dec 13 18:35:34 Tower shfs: FUSE library version 3.12.0 Dec 13 18:35:34 Tower emhttpd: shcmd (134): mkdir /mnt/user Dec 13 18:35:34 Tower emhttpd: shcmd (135): /usr/local/bin/shfs /mnt/user -disks 15 -o default_permissions,allow_other,noatime -o remember=330 Dec 13 18:35:34 Tower shfs: FUSE library version 3.12.0 Dec 13 18:35:34 Tower emhttpd: shcmd (138): /usr/local/sbin/update_cron Dec 13 18:35:35 Tower emhttpd: Unregistered Flash device error (ENOFLASH3) Dec 13 18:35:37 Tower root: Delaying execution of fix common problems scan for 10 minutes This is not something that can wait until 7.0.0 to be fixed, this needs a patch on 6.12 ASAP.

Can this be linked in the OP or added to the "recommended posts" for this thread? I've been trying to solve this issue for days while trying to get backblaze to backup my photos share (with 1M+ files), and this post solved it perfectly. Thank you so much for posting this.

Right, but I'm trying to understand why wipefs would ever be run automatically, much less on the unraid boot USB, regardless of the state of the zfs pool. Secondly, I have no idea how sdk became a hot spare for zfs; I am certain I setup the hot spare to be one of the 2 extra 8TB drives I have in the system, not one of my array disks. I get that 7.0.0 has additional protections against this, but there is something fundamentally wrong with the logic of some unRAID component here that is trying to run wipefs at all.

Actually, I did some digging in the backup from before the upgrade and found this in the syslog from a diagnostics snapshot I had taken on Nov 15th (idk what issue I was looking at, it wasn't serious then): Nov 15 18:37:34 Tower emhttpd: shcmd (58): /usr/sbin/zfs set mountpoint=/mnt/zfs zfs Nov 15 18:37:38 Tower emhttpd: shcmd (59): /usr/sbin/zfs set atime=off zfs Nov 15 18:37:41 Tower emhttpd: shcmd (60): /usr/sbin/zfs mount zfs Nov 15 18:37:42 Tower emhttpd: shcmd (61): /usr/sbin/zpool set autotrim=off zfs Nov 15 18:37:49 Tower emhttpd: shcmd (62): /usr/sbin/zfs set compression=on zfs Nov 15 18:37:55 Tower emhttpd: /mnt/zfs root profile: /r..1/... Nov 15 18:37:55 Tower emhttpd: /mnt/zfs root groups: 1 Nov 15 18:37:55 Tower emhttpd: /mnt/zfs root width: 4 Nov 15 18:37:55 Tower emhttpd: /usr/sbin/zpool status -PL zfs 2>&1 Nov 15 18:37:55 Tower emhttpd: pool: zfs Nov 15 18:37:55 Tower emhttpd: state: ONLINE Nov 15 18:37:55 Tower emhttpd: scan: scrub in progress since Sun Nov 3 00:00:01 2024 Nov 15 18:37:55 Tower emhttpd: 10.2T scanned at 27.5G/s, 8.51T issued at 6.86K/s, 26.0T total Nov 15 18:37:55 Tower emhttpd: 0B repaired, 32.71% done, no estimated completion time Nov 15 18:37:55 Tower emhttpd: config: Nov 15 18:37:55 Tower emhttpd: NAME STATE READ WRITE CKSUM Nov 15 18:37:55 Tower emhttpd: zfs ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /r..1/...-0 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /dev/sdb1 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /dev/sde1 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /dev/sdc1 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /dev/sdh1 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: spares Nov 15 18:37:55 Tower emhttpd: /dev/sdk1 FAULTED corrupted data Nov 15 18:37:55 Tower emhttpd: shcmd (64): /sbin/wipefs -a /dev/sda Nov 15 18:37:55 Tower root: wipefs: error: /dev/sda: probing initialization failed: Device or resource busy Nov 15 18:37:55 Tower emhttpd: shcmd (64): exit status: 1 Nov 15 18:37:55 Tower emhttpd: errors: No known data errors Nov 15 18:37:55 Tower emhttpd: shcmd (65): /usr/sbin/zfs mount -a Nov 15 18:37:55 Tower emhttpd: shcmd (66): sync Nov 15 18:37:55 Tower emhttpd: shcmd (67): mkdir /mnt/user0 Nov 15 18:37:55 Tower emhttpd: shcmd (68): /usr/local/bin/shfs /mnt/user0 -disks 14 -o default_permissions,allow_other,noatime Nov 15 18:37:55 Tower shfs: FUSE library version 3.12.0 Nov 15 18:37:55 Tower emhttpd: shcmd (69): mkdir /mnt/user Nov 15 18:37:55 Tower emhttpd: shcmd (70): /usr/local/bin/shfs /mnt/user -disks 15 -o default_permissions,allow_other,noatime -o remember=330 Nov 15 18:37:55 Tower shfs: FUSE library version 3.12.0 Nov 15 18:37:56 Tower emhttpd: shcmd (72): /usr/local/sbin/update_cron This was on 6.12.4. What looks to me to be happening here is: 1. sdk (one of my array drives) is already a "zfs spare" despite being a 4TB drive (zpool is all 8TB drives) and not being compatible with that in anyway, and that this occurred before the upgrade to 6.12.14. 2. wipefs tries to run to nuke the usb, but by (apparent) luck, the USB drive is already busy and so wipefs fails. This makes me theorize that something happened in the way the array is loaded between 6.12.4 such that when the wipefs command runs, the USB is no longer busy and happily wipes itself. But the root is that wipefs is being called at all, so there's a problem that has originated in 6.12.4 or earlier here. @JorgeBI don't think these diagnostics are anonymized (is there an easy way to check?) so I'm not super comfortable posting the full zip publicly, what's the best way for me to send them to you?

I’ll go digging in the backups this evening, but I had another idea; what if I tried to induce the same bug by adding an invalid drive to the pool as a spare? Is there a way I could add something like the array parity drive as a “zfs hot spare” without actually wiping the drive? My knowledge of ZFS is unfortunately not crazy in depth so I’m not sure how possible/safe this is.

No, and I’m kicking myself for not having done so, that probably would have helped track down the root cause here. What I can say is I had an 8TB drive setup as a spare when I was on 6.12.4 (the drive is still plugged in and recognized in 6.12.14), but then when on 6.12.14, somehow zpool status showed one of my 4TB array disks as being the spare instead of that 8TB drive. What I do have is several flash backups from before and after the upgrade (but before mounting disks). Is there anything relevant from those that I could pull which would be useful for debugging purposes?

Just ran into this on 6.12.14 upgrading from 6.12.4, this is absolutely mindblowing that this continues to be an issue. In my case my ZFS pool had somehow picked up one of my unRAID array drives as a "spare" (which of course it did not like), and so unraid decides to immediately wipefs itself out of existence as soon as I mount the array. I am very lucky I made a flash backup immediately prior to updating otherwise I would be in some trouble right now. I am very concerned because I don't understand what scenario could possibly exist where wipefs should be called on any physical device attached to the system without extreme levels of user involvement and consent. This worked to solve the issue, thank you.

Does this include unRAID 7.0 and other such future major versions?

Been running for about 48 hours now with this fix, zero memory leaks that I can detect. Thanks so much for posting it here!

Unfortunately the drives are both in a ZFS pool, so I can't do this. Should I run a scrub as the equivelant?

Looking at the smart reports now, they now say "Past" for the seek error rate row. How can it go from "Failing" to "Past"? TOSHIBA_HDWG180_X040A052FBEG-20231015-1832.txt TOSHIBA_HDWG180_X040A0CEFBEG-20231015-1831.txt

I can't seem to get an extended or short test to even run on either of the drives, it tries and then a few seconds later says "Completed: unknown failure". The SMART error log is empty for both drives.