October 23, 20241 yr I honestly have no clue if this is a bug or if there's something really weird with my setup .. suddenly. I haven't really changed anything, but now after every boot, the Unraid flash drive gets wipefs'd and then Unraid runs in emulated mode. Everything works, until next boot and nothing gets stored if I try to save settings (I save them manually from a 1 day old backup directly in the config files if required for my debugging). This is probably the interesting bit of the logs from the drive itself - this is output from first start of the array under "Maintenance mode" which works fine, drive still accessible and all. Then when I start the array the operational way, it will start wipefs: BOOT: Oct 23 20:35:01 Odin kernel: sd 7:0:0:0: [sdb] 125313283 512-byte logical blocks: (64.2 GB/59.8 GiB) Oct 23 20:35:01 Odin kernel: sd 7:0:0:0: [sdb] Write Protect is off Oct 23 20:35:01 Odin kernel: sd 7:0:0:0: [sdb] Mode Sense: 43 00 00 00 Oct 23 20:35:01 Odin kernel: sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Oct 23 20:35:01 Odin kernel: sdb: sdb1 Oct 23 20:35:01 Odin kernel: sd 7:0:0:0: [sdb] Attached SCSI removable disk Starting maintenance mode: Oct 23 20:36:03 Odin emhttpd: Samsung_Flash_Drive_0391923080011130-0:0 (sdb) 512 125313283 Oct 23 20:36:04 Odin emhttpd: import flash device: sdb Oct 23 20:36:04 Odin emhttpd: read SMART /dev/sdb Starting normal mode: Oct 23 20:40:30 Odin emhttpd: read SMART /dev/sdb Oct 23 20:40:33 Odin emhttpd: Samsung_Flash_Drive_0391923080011130-0:0 (sdb) 512 125313283 Oct 23 20:40:33 Odin emhttpd: import flash device: sdb Oct 23 20:40:34 Odin emhttpd: read SMART /dev/sdb Oct 23 20:43:04 Odin emhttpd: shcmd (550): /sbin/wipefs -af /dev/sdb1 Oct 23 20:43:04 Odin root: /dev/sdb1: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20 Oct 23 20:43:04 Odin root: /dev/sdb1: 1 byte was erased at offset 0x00000000 (vfat): eb Oct 23 20:43:04 Odin root: /dev/sdb1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa Oct 23 20:43:04 Odin emhttpd: writing MBR on disk (sdb) with partition 1 offset 64, erased: 0 Oct 23 20:43:05 Odin emhttpd: re-reading (sdb) partition table Oct 23 20:43:05 Odin emhttpd: error: mkmbr, 2196: Device or resource busy (16): ioctl BLKRRPART: /dev/sdb Oct 23 20:43:10 Odin emhttpd: shcmd (552): /usr/sbin/cryptsetup luksFormat /dev/sdb1 --key-file=/root/keyfile Oct 23 20:43:10 Odin root: Cannot use device /dev/sdb1 which is in use (already mapped or mounted). What I have tried: rolled back to Unraid version 6.12.11 2 different kernels (unraid original + intel gpu kernel) disabled all plugins (-all-, even my own, including purging of configs) disabled dockers (VM's I don't use and are disabled). different USB ports (both USB 2.0) different USB drives enabled and disabled "destructive mode" in "Unassigned devices" plugin tried to use both the windows usb creator on my work laptop, the real manual way on my computer including the usb creator for linux (which didn't like 64GB drive), but doesn't matter, all methods behaved the same. Most of these are a desperation of making it work, but it seems like it's software related beyond user related inputs. I have no idea at this point, and don't see anything logic in the logs either (I have attached the diagnostic file). Once I shutdown the system, I must recreate the USB flash drive again, copying my backup back onto it, running make_boot script etc. before I can start it. Reboots will just hang with an empty flash drive. I have NO idea why it want to run wipefs on the flash drive, it makes absolutely no sense when the drive is even working as expected. Edited October 24, 20241 yr by FlamongOle removed diagnostic file
October 23, 20241 yr Community Expert Not sure why or at boot that would clear the flash drive... something in the go file or a start script? can you try another flash drive? with trail mode on the same hardware? be sure to use a USB2.0 port and not a 3.0 port. What volume format is the flash drive? (vfat/nfs/hfs?) as unraid requires the usb as a volume fat32 format. I have encountered many issues with the unraid creator tool and usualy have to format my flash drive beforehand: So i usual use this windows tool to format my usb as a fat32 volume https://www.bleepingcomputer.com/download/hp-usb-disk-storage-format-tool/ CALL the Volume "UNRAID" !!! <This is also required... then extract the latest stable unraid on the flash drive and run the and run the make bootable. 9as we need some flagsset for mbr/uefi and teh boot partition https://docs.unraid.net/unraid-os/download_list/ Somewhere it files sytem that was the flash drive was erased per your log of [sdb] Write Protect is off Ar you using a usb hard drive / bzoverley? As it looks like your using a usb HD and have the USB HD bee the boot volume and formatted for the unraid array...
October 23, 20241 yr Author 5 minutes ago, bmartino1 said: Not sure why or at boot that would clear the flash drive... something in the go file or a start script? can you try another flash drive? with trail mode on the same hardware? be sure to use a USB2.0 port and not a 3.0 port. What volume format is the flash drive? (vfat/nfs/hfs?) as unraid requires the usb as a volume fat32 format. I have encountered many issues with the unraid creator tool and usualy have to format my flash drive beforehand: So i usual use this windows tool to format my usb as a fat32 volume https://www.bleepingcomputer.com/download/hp-usb-disk-storage-format-tool/ CALL the Volume "UNRAID" !!! <This is also required... then extract the latest stable unraid on the flash drive and run the and run the make bootable. 9as we need some flagsset for mbr/uefi and teh boot partition https://docs.unraid.net/unraid-os/download_list/ Somewhere it files sytem that was the flash drive was erased per your log of [sdb] Write Protect is off Ar you using a usb hard drive / bzoverley? As it looks like your using a usb HD and have the USB HD bee the boot volume and formatted for the unraid array... Everything you just asked, is already answered. But I can clarify that, yes, it is FAT32, drive called uppercase UNRAID. No, it's not an HDD. It's not a problem that I can boot it, it's that Unraid runs wipefs after everything has booted up, meaning: it can't boot next time! Startup scripts are not available once the plugins are all gone/disabled. Yes, I have something small in the "go" file which has been identical since I started using Unraid. Nothing that should run or activate wipefs anyhow.
October 23, 20241 yr Community Expert Do you have other settings enabled with the unassigned devices or plugins to auto wipe? Do you have start the array automatic at boot? As I would recommend useing the same flash drive for the UUID. Install fresh and copy the key file in teh config file over only and start fresh. Clearly their a config or edit something in a backup or remake progress that is telling the unraid array to format that volume as it is detect as /dev/sdb? was /dev/sdb originally a drive in the array? Did other hardware change? as this still seems like the array is puliing that mount point "/dev/sdb" by a pre-made setting and formatting it per the config file. can you zip and post your backup for testing? - PM
October 23, 20241 yr Author 1 minute ago, bmartino1 said: Do you have other settings enabled with the unassigned devices or plugins to auto wipe? Do you have start the array automatic at boot? As I would recommend useing the same flash drive for the UUID. Install fresh and copy the key file in teh config file over only and start fresh. Clearly their a config or edit something in a backup or remake progress that is telling the unraid array to format that volume as it is detect as /dev/sdb? was /dev/sdb originally a drive in the array? Did other hardware change? as this still seems like the array is puliing that mount point "/dev/sdb" by a pre-made setting and formatting it per the config file. can you zip and post your backup for testing? - PM No way I'm posting my backup. But again, this is already answered. Please read my initial post. It clearly says I have tried multiple drives and ports. That means that /dev/sdb was not the same between the swaps. About all my points in the first post is a fresh install, just different configs. Without plugins, there's no configuration to be done in the plugin you suggest even. As I said, I removed and purged all of it. This is a system which ran without issue for so long, and suddenly started to act up. It feels weird if this is hardware related at all. But I don't understand why wipefs would run at all, it makes no sense.
October 23, 20241 yr Community Expert Its not a plug its a config file within the config folder. you should be booting a fresh unraid not a backup as the test. if a fresh downald no back or copy of your config folder. is still delting, then its something on your hardware like impmi that is calling for a erasure.
October 23, 20241 yr Author 1 hour ago, bmartino1 said: Its not a plug its a config file within the config folder. you should be booting a fresh unraid not a backup as the test. if a fresh downald no back or copy of your config folder. is still delting, then its something on your hardware like impmi that is calling for a erasure. Unassigned devices is by fact a plugin, even though it's probably "included by standard" by now: /config/plugins/unassigned.devices/ Absolutely everything under /config/plugins/ was removed for one test, including plugins-removed and plugins-error. I have been going through every single file, and there's nothing there that can cause this problem. I even tried one more time now after cleaning up some additional old share configs. It's just doing the same over and over, wiping the flash drive, but only the one UNRAID is on, and only after everything has mounted. Plugins, dockers, VMs can all be ruled out. IPMI is a plugin too, and I don't see how that should by itself call for a "wipefs", makes no sense. Yes, I do use IPMI.
October 23, 20241 yr Community Expert a leftover config from unassigned preclear adds data into a config else where that could be casusing this... as example... their is a config file that auto starts the array... This is not by default from a fresh unraid trial version of unraid... (deleting a plugin or unistall doesn't always remove any pre-configured settings it has affected...) so at unraid grub boot mode boot into safe mode no plugins (with no other edits default stuff only!)... Further... IPMI is actual a separate chip firmware running on the motherboard, the plugins you are reefing to connects to that... There could be a misconfiguration there... *Just another example of a misunderstanding cause by asumptions... regardless, you have a config setting somewhere that is clear... And the undesired result seems to be happening at array start / startup, cause by trying to restore... So DON'T and test by doing a fresh installation! so no restroed backup configs to test that, i'm not sure how eles to assist you. maybe another user as its celar your miss understanding me... good luck.
October 24, 20241 yr Community Expert Solution This looks like a bug, there was a similar one before that would do that if you tried to start the array with a degraded zfs pool, that was fixed and you don't have a degraded pool, but you do have one with an unavailable spare, and that device is now being used in another pool, try manually removing the spare form the server pool, and then retest.
October 24, 20241 yr Author 5 hours ago, JorgeB said: This looks like a bug, there was a similar one before that would do that if you tried to start the array with a degraded zfs pool, that was fixed and you don't have a degraded pool, but you do have one with an unavailable spare, and that device is now being used in another pool, try manually removing the spare form the server pool, and then retest. Now, that looks more promising. I will try to do that right away as I do have a spare drive in a zfs pool. I even see it as "UNAVAIL" too, so something there happened. I will report back as soon as I can confirm this. Thanks! Edited October 24, 20241 yr by FlamongOle
October 24, 20241 yr Community Expert You can have spares, the problem was the spare being unavailable and used by another pool, though in v7 probably the flash would not be wiped, since it includes better protection for that.
October 24, 20241 yr Author 4 hours ago, JorgeB said: You can have spares, the problem was the spare being unavailable and used by another pool, though in v7 probably the flash would not be wiped, since it includes better protection for that. It was not used by another pool, it was just unavailable for some reason. It remained on the pool I assigned it too. Anyway, I disabled it now and hope I rather can enable it safer in the future (v7). It's not a crisis and will do for the time being as a cold spare. Thanks anyway for pointing it out. It's a nasty bug regardless.
October 24, 20241 yr Community Expert 3 minutes ago, FlamongOle said: It was not used by another pool You removed the diags, but pretty sure that the spare device that appeared unavailable, sdb IIRC, was assigned to the first pool.
October 24, 20241 yr Author 4 hours ago, JorgeB said: You removed the diags, but pretty sure that the spare device that appeared unavailable, sdb IIRC, was assigned to the first pool. Might be, I couldn't see anything in the diagnostic about exactly that. Anyway, the Unraid team got 3 different diagnostics from me. But most important is that it now works.
October 25, 20241 yr Community Expert I still had the diags on my home PC, this was the problem pool: Oct 23 20:43:04 Odin emhttpd: pool: server Oct 23 20:43:04 Odin emhttpd: state: ONLINE Oct 23 20:43:04 Odin emhttpd: config: Oct 23 20:43:04 Odin emhttpd: #011NAME STATE READ WRITE CKSUM Oct 23 20:43:04 Odin emhttpd: #011server ONLINE 0 0 0 Oct 23 20:43:04 Odin emhttpd: #011 /m..r/...-0 ONLINE 0 0 0 Oct 23 20:43:04 Odin emhttpd: #011 /dev/mapper/sdc1 ONLINE 0 0 0 Oct 23 20:43:04 Odin emhttpd: #011 /dev/mapper/sdd1 ONLINE 0 0 0 Oct 23 20:43:04 Odin emhttpd: #011spares Oct 23 20:43:04 Odin emhttpd: #011 /dev/sdf1 UNAVAIL And sdf is assigned to the backups pool: Oct 23 20:43:01 Odin emhttpd: pool: backups Oct 23 20:43:01 Odin emhttpd: state: ONLINE Oct 23 20:43:01 Odin emhttpd: status: Some supported and requested features are not enabled on the pool. Oct 23 20:43:01 Odin emhttpd: #011The pool can still be used, but some features are unavailable. Oct 23 20:43:01 Odin emhttpd: action: Enable all features using 'zpool upgrade'. Once this is done, Oct 23 20:43:01 Odin emhttpd: #011the pool may no longer be accessible by software that does not support Oct 23 20:43:01 Odin emhttpd: #011the features. See zpool-features(7) for details. Oct 23 20:43:01 Odin emhttpd: scan: scrub repaired 0B in 04:33:55 with 0 errors on Fri Apr 19 03:15:01 2024 Oct 23 20:43:01 Odin emhttpd: config: Oct 23 20:43:01 Odin emhttpd: #011NAME STATE READ WRITE CKSUM Oct 23 20:43:01 Odin emhttpd: #011backups ONLINE 0 0 0 Oct 23 20:43:01 Odin emhttpd: #011 /dev/mapper/sdf1 ONLINE 0 0 0
November 29, 20241 yr On 10/24/2024 at 1:09 AM, JorgeB said: This looks like a bug, there was a similar one before that would do that if you tried to start the array with a degraded zfs pool Just ran into this on 6.12.14 upgrading from 6.12.4, this is absolutely mindblowing that this continues to be an issue. In my case my ZFS pool had somehow picked up one of my unRAID array drives as a "spare" (which of course it did not like), and so unraid decides to immediately wipefs itself out of existence as soon as I mount the array. I am very lucky I made a flash backup immediately prior to updating otherwise I would be in some trouble right now. I am very concerned because I don't understand what scenario could possibly exist where wipefs should be called on any physical device attached to the system without extreme levels of user involvement and consent. On 10/24/2024 at 1:09 AM, JorgeB said: try manually removing the spare form the server pool, and then retest. This worked to solve the issue, thank you.
November 29, 20241 yr Community Expert 1 hour ago, sazrocks said: Just ran into this on 6.12.14 upgrading from 6.12.4, this is absolutely mindblowing that this continues to be an issue. This should only happen if you have a spare incorrectly configured, did you save the diags before removing it?
November 29, 20241 yr 6 hours ago, JorgeB said: This should only happen if you have a spare incorrectly configured, did you save the diags before removing it? No, and I’m kicking myself for not having done so, that probably would have helped track down the root cause here. What I can say is I had an 8TB drive setup as a spare when I was on 6.12.4 (the drive is still plugged in and recognized in 6.12.14), but then when on 6.12.14, somehow zpool status showed one of my 4TB array disks as being the spare instead of that 8TB drive. What I do have is several flash backups from before and after the upgrade (but before mounting disks). Is there anything relevant from those that I could pull which would be useful for debugging purposes?
November 29, 20241 yr Community Expert 10 minutes ago, sazrocks said: What I do have is several flash backups from before and after the upgrade (but before mounting disks) If it's before mounting the disks I'm afraid it won't help, but if you have at least a syslog, or syslog-previous, saved with the disks mounted, it might.
November 29, 20241 yr 42 minutes ago, JorgeB said: If it's before mounting the disks I'm afraid it won't help, but if you have at least a syslog, or syslog-previous, saved with the disks mounted, it might. I’ll go digging in the backups this evening, but I had another idea; what if I tried to induce the same bug by adding an invalid drive to the pool as a spare? Is there a way I could add something like the array parity drive as a “zfs hot spare” without actually wiping the drive? My knowledge of ZFS is unfortunately not crazy in depth so I’m not sure how possible/safe this is.
November 29, 20241 yr 1 hour ago, JorgeB said: If it's before mounting the disks I'm afraid it won't help, but if you have at least a syslog, or syslog-previous, saved with the disks mounted, it might. Actually, I did some digging in the backup from before the upgrade and found this in the syslog from a diagnostics snapshot I had taken on Nov 15th (idk what issue I was looking at, it wasn't serious then): Nov 15 18:37:34 Tower emhttpd: shcmd (58): /usr/sbin/zfs set mountpoint=/mnt/zfs zfs Nov 15 18:37:38 Tower emhttpd: shcmd (59): /usr/sbin/zfs set atime=off zfs Nov 15 18:37:41 Tower emhttpd: shcmd (60): /usr/sbin/zfs mount zfs Nov 15 18:37:42 Tower emhttpd: shcmd (61): /usr/sbin/zpool set autotrim=off zfs Nov 15 18:37:49 Tower emhttpd: shcmd (62): /usr/sbin/zfs set compression=on zfs Nov 15 18:37:55 Tower emhttpd: /mnt/zfs root profile: /r..1/... Nov 15 18:37:55 Tower emhttpd: /mnt/zfs root groups: 1 Nov 15 18:37:55 Tower emhttpd: /mnt/zfs root width: 4 Nov 15 18:37:55 Tower emhttpd: /usr/sbin/zpool status -PL zfs 2>&1 Nov 15 18:37:55 Tower emhttpd: pool: zfs Nov 15 18:37:55 Tower emhttpd: state: ONLINE Nov 15 18:37:55 Tower emhttpd: scan: scrub in progress since Sun Nov 3 00:00:01 2024 Nov 15 18:37:55 Tower emhttpd: 10.2T scanned at 27.5G/s, 8.51T issued at 6.86K/s, 26.0T total Nov 15 18:37:55 Tower emhttpd: 0B repaired, 32.71% done, no estimated completion time Nov 15 18:37:55 Tower emhttpd: config: Nov 15 18:37:55 Tower emhttpd: NAME STATE READ WRITE CKSUM Nov 15 18:37:55 Tower emhttpd: zfs ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /r..1/...-0 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /dev/sdb1 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /dev/sde1 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /dev/sdc1 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: /dev/sdh1 ONLINE 0 0 0 Nov 15 18:37:55 Tower emhttpd: spares Nov 15 18:37:55 Tower emhttpd: /dev/sdk1 FAULTED corrupted data Nov 15 18:37:55 Tower emhttpd: shcmd (64): /sbin/wipefs -a /dev/sda Nov 15 18:37:55 Tower root: wipefs: error: /dev/sda: probing initialization failed: Device or resource busy Nov 15 18:37:55 Tower emhttpd: shcmd (64): exit status: 1 Nov 15 18:37:55 Tower emhttpd: errors: No known data errors Nov 15 18:37:55 Tower emhttpd: shcmd (65): /usr/sbin/zfs mount -a Nov 15 18:37:55 Tower emhttpd: shcmd (66): sync Nov 15 18:37:55 Tower emhttpd: shcmd (67): mkdir /mnt/user0 Nov 15 18:37:55 Tower emhttpd: shcmd (68): /usr/local/bin/shfs /mnt/user0 -disks 14 -o default_permissions,allow_other,noatime Nov 15 18:37:55 Tower shfs: FUSE library version 3.12.0 Nov 15 18:37:55 Tower emhttpd: shcmd (69): mkdir /mnt/user Nov 15 18:37:55 Tower emhttpd: shcmd (70): /usr/local/bin/shfs /mnt/user -disks 15 -o default_permissions,allow_other,noatime -o remember=330 Nov 15 18:37:55 Tower shfs: FUSE library version 3.12.0 Nov 15 18:37:56 Tower emhttpd: shcmd (72): /usr/local/sbin/update_cron This was on 6.12.4. What looks to me to be happening here is: 1. sdk (one of my array drives) is already a "zfs spare" despite being a 4TB drive (zpool is all 8TB drives) and not being compatible with that in anyway, and that this occurred before the upgrade to 6.12.14. 2. wipefs tries to run to nuke the usb, but by (apparent) luck, the USB drive is already busy and so wipefs fails. This makes me theorize that something happened in the way the array is loaded between 6.12.4 such that when the wipefs command runs, the USB is no longer busy and happily wipes itself. But the root is that wipefs is being called at all, so there's a problem that has originated in 6.12.4 or earlier here. @JorgeBI don't think these diagnostics are anonymized (is there an easy way to check?) so I'm not super comfortable posting the full zip publicly, what's the best way for me to send them to you? Edited November 29, 20241 yr by sazrocks
November 29, 20241 yr Community Expert 10 minutes ago, sazrocks said: Nov 15 18:37:55 Tower emhttpd: /dev/sdk1 FAULTED corrupted data This was the problem, as long as you remove that it won't happen, and with 7.0.0 it should not happen, even with a degraded spare, there are more protections.
November 29, 20241 yr 3 minutes ago, JorgeB said: This was the problem, as long as you remove that it won't happen, and with 7.0.0 it should not happen, even with a degraded spare, there are more protections. Right, but I'm trying to understand why wipefs would ever be run automatically, much less on the unraid boot USB, regardless of the state of the zfs pool. Secondly, I have no idea how sdk became a hot spare for zfs; I am certain I setup the hot spare to be one of the 2 extra 8TB drives I have in the system, not one of my array disks. I get that 7.0.0 has additional protections against this, but there is something fundamentally wrong with the logic of some unRAID component here that is trying to run wipefs at all. Edited November 29, 20241 yr by sazrocks
November 29, 20241 yr Community Expert 1 hour ago, sazrocks said: Right, but I'm trying to understand why wipefs would ever be run automatically It's a bug, but it's a very corner case because it requires that the user has a pool with an invalid device, but like mentioned, it should not happen with v7.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.