robobub

Thanks for your help, I will definitely try that out when I get home. I was hoping to leave the AC off during the day and only boot up the server at night for backups on my trip, but this unclean shutdown definitely makes it difficult. Here's ps aux/top/pstree/lsof/zpool status outputs. $ zpool list no pools available ps_aux.txt pstree.txt top.txt lsof2.txt

But as with the /mnt/user error, /mnt/cache_zfs is already unmounted, as you can see from the outputs from findmnt ,lsof /mnt/cache*, and find /mnt/. zpool status shows no pools are available anymore. Docker and VM services were successfully shut down at this point of array stopping, and nothing was showing up in htop/top, but I did try disabling them and starting/stopping the array and the same issue occurs. I just left for a weekend trip so I'll be holding off on trying safe mode. Though in theory I could try it with Wireguard/Tailscale being part of the core unRAID and you can change the default grub boot option. I'm sure it's some weird plugin/configuration, but I mean everything except tmpfs is unmounted, and I can halt with no data loss, it's just for some reason unRAID won't shut down and marks it as an unclean shutdown. htop/top/ps doesn't show anything obvious in terms of plugins or processes (is that output in diagnostics?).

UPDATE: in this version change inotifywait becomes a crucial part of the shutdown process. Previously I had some rare issues with that process preventing shutdown from a different plugin and had it killed on a shutdown script, which caused the shutdown process to stop and never recover. I recently upgraded to 6.12.15 from 6.12.10 to try to fix some zfs issues (migrating from my own zfs management to unRAID's) and now it refuses to stop the array causing unclean shutdown/reboots. I had run into this message plenty of times before in the past due to other issues, but to my surprise, all the disks and zfs are actually already unmounted. From the timestamps it looks like from stop array request to unmounting the last disk is around 2 minutes. Then unRAID just sits there with this message in the GUI. Attached diagnostics and some of my own attempts and investigating. How can I figure out why it is unhappy? EDIT: ps/top/lsof outputs here $ findmnt TARGET SOURCE FSTYPE OPTIONS / rootfs rootfs rw,size=16391148k,nr_inodes=4097787,inode64 ├─/proc proc proc rw,relatime ├─/sys sysfs sysfs rw,relatime │ ├─/sys/fs/fuse/connections fusectl fusectl rw,relatime │ └─/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime ├─/run tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=131072k,mode=755,inode64 │ └─/run/docker/netns/default nsfs[net:[4026531840]] nsfs rw ├─/boot /dev/sda1 vfat rw,noatime,nodiratime,fmask=0177,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,flush,errors=remount-ro ├─/lib /dev/loop0 squashfs ro,relatime,errors=continue │ └─/lib overlay overlay rw,relatime,lowerdir=/lib,upperdir=/var/local/overlay/lib,workdir=/var/local/overlay-work/lib ├─/usr /dev/loop1 squashfs ro,relatime,errors=continue │ └─/usr overlay overlay rw,relatime,lowerdir=/usr,upperdir=/var/local/overlay/usr,workdir=/var/local/overlay-work/usr ├─/dev devtmpfs devtmpfs rw,relatime,size=8192k,nr_inodes=4097787,mode=755,inode64 │ ├─/dev/pts devpts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 │ └─/dev/shm tmpfs tmpfs rw,relatime,inode64 ├─/hugetlbfs hugetlbfs hugetlbfs rw,relatime,pagesize=2M ├─/var/log tmpfs tmpfs rw,relatime,size=131072k,mode=755,inode64 └─/mnt rootfs[/mnt] rootfs rw,size=16391148k,nr_inodes=4097787,inode64 ├─/mnt/disks tmpfs tmpfs rw,relatime,size=1024k,inode64 ├─/mnt/remotes tmpfs tmpfs rw,relatime,size=1024k,inode64 ├─/mnt/addons tmpfs tmpfs rw,relatime,size=1024k,inode64 └─/mnt/rootshare tmpfs tmpfs rw,relatime,size=1024k,inode64 $ tail /var/log/syslog Jul 10 13:04:09 t1001 root: cannot unmount '/mnt/cache_zfs': pool or dataset is busy Jul 10 13:04:25 t1001 emhttpd: shcmd (160): umount /mnt/user Jul 10 13:04:25 t1001 root: umount: /mnt/user: target is busy. Jul 10 13:04:25 t1001 emhttpd: shcmd (160): exit status: 32 Jul 10 13:04:25 t1001 emhttpd: shcmd (161): rmdir /mnt/user Jul 10 13:04:25 t1001 root: rmdir: failed to remove '/mnt/user': Device or resource busy Jul 10 13:04:25 t1001 emhttpd: shcmd (161): exit status: 1 Jul 10 13:04:25 t1001 emhttpd: shcmd (163): rm -f /boot/config/plugins/dynamix/mover.cron Jul 10 13:04:25 t1001 emhttpd: shcmd (164): /usr/local/sbin/update_cron Jul 10 13:04:25 t1001 emhttpd: Retry unmounting user share(s)... Jul 10 13:04:27 t1001 kernel: script (5153): drop_caches: 3 $ ## the error messages above actually successfully completed at some later time, see dmesg tail below $ umount /mnt/user umount: /mnt/user: no mount point specified. $ rmdir /mnt/user rmdir: failed to remove '/mnt/user': No such file or directory $ zpool list no pools available $ date Thu Jul 10 13:25:23 EDT 2025 $ dmesg -T | tail [Thu Jul 10 13:04:02 2025] veth8f0ab61: renamed from eth0 [Thu Jul 10 13:04:02 2025] br-e769d98838f0: port 1(veth4e25d88) entered disabled state [Thu Jul 10 13:04:02 2025] device veth4e25d88 left promiscuous mode [Thu Jul 10 13:04:02 2025] br-e769d98838f0: port 1(veth4e25d88) entered disabled state [Thu Jul 10 13:04:27 2025] script (5153): drop_caches: 3 [Thu Jul 10 13:04:40 2025] BTRFS info (device dm-0): last unmount of filesystem 3b2ddebb-0f7c-470e-b80b-746480624f45 [Thu Jul 10 13:04:40 2025] BTRFS info (device dm-1): last unmount of filesystem 781ec874-cba1-4e0b-9de6-6724ea81035c [Thu Jul 10 13:04:41 2025] BTRFS info (device dm-2): last unmount of filesystem ab612068-00dd-4727-9c67-44b97ca3b95a [Thu Jul 10 13:04:41 2025] BTRFS info (device dm-3): last unmount of filesystem 71712954-1765-4aa6-84dc-1ee9efeb66e8 [Thu Jul 10 13:04:42 2025] XFS (dm-4): Unmounting Filesystem $ lsof /mnt/user* /mnt/disk* /mnt/cache* lsof: status error on /mnt/user*: No such file or directory $ lsof /mnt/disk* /mnt/cache* $ find /mnt/ /mnt/ /mnt/cache_zfs /mnt/rootshare /mnt/addons /mnt/remotes /mnt/disks Thought it might be open luks devices, but after this it is still on that message $ dmsetup ls --target crypt md1p1 (254, 0) md2p1 (254, 1) md3p1 (254, 2) md4p1 (254, 3) nvme0n1p1 (254, 4) sdb1 (254, 5) sdc1 (254, 6) sdd1 (254, 7) $ for d in /dev/mapper/*; do [[ "$(basename "$d")" != "control" ]] && cryptsetup luksClose "$(basename "$d")"; done $ dmsetup ls --target crypt No devices foundt1001-diagnostics-20250710-1320.zip

Any reason why we just don't keep referring to the obfuscated chattr while running the script? Might as well remove the small chance that ransomware tries to do chattr -i while the script is running? Or what scenario am I missing?

I think you just encountered the issue I did with temperature pause/resume. From the comments, it looks like the logic also applies to pause/resume from mover action. I made a pull request here. Making these changes in /usr/local/bin/parity.check seem to be working for my case, though I did not check the second case yet. TESTING logs before the change for context

So it turns out the issue I stated earlier (RAM that went bad) was actually something else: a bad SATA cable or PSU after some more googling. Since replacing RAM and verifying on memtest, some writes were submitted that were ignored and the sata link has to be reset and thus actually fits this scenario more precisely than just bad RAM. However, there is indeed a problem by not having the checksumming and parity integrated together, as apandey's intuition suggested, at least when set up for copy-on-write filesystem (which is the default for BTRFS). There is still a solution I used to recover without a full rebuild, but it's advanced. Some more details: While I was able to recover the particular file that had this corruption (which manifested as "Input/Output Error", parity is still out of sync. Why? Not from the recovery process, it was already out of sync. The emulated drive did not have those BTRFS checksum errors when accessing the file, and the array's stats showed 0 in the write column for all drives when recovering (and a bunch of similar reads even though only the emulated device was mounted, which shows emulation was working). But the physical drive did have those BTRFS checksum errors. The problem: updating/deleting the file on a copy-on-write filesystem does not modify the corrupted sectors. As I mentioned earlier, while parity had the correct bits and correctly wrote the file, parity was out of sync because the physical drive had incorrect bits. When updating or deleting the file, BTRFS by default will place the new data in a new sector if it was not created with the chattr attribute to disable COW. That means the physical sectors that were used to hold the data, while now unreferenced by the BTRFS filesystem, still hold data that is out of sync with the parity. So if one needed to restore from those sectors on one of the other hard drives, it will be incorrect. Eventually BTRF will reclaim that space, which could be sped up via balance or defrag, but that's not really acceptable. The solutions is to tell unRAID to parity sync specific sectors. You can do this with dd or other advanced low-level tools after looking up the physical locations associated with the file by inspecting the BTRFS filesystem metadata. But I'll just strongly recommend against it unless you know what you are doing. I was able to get it working with dd, btrfs-progs, and help from #btrfs on irc. Some snippets of the errors. When trying to access a particular file, "Input/Output Error" is received. I did the steps that I outlined and After re-mounting with the disk, a parity check has shown no sync errors so far. Steps (WARNING, you will lose data unless you know what you are doing and how to navigate low level filesystem metadata) stop array (optional) backup /boot/config select "no device" for disk with error start array in maintenance mode mount device with `-o ro,loop,norecovery` Copy file in question to somewhere not on the disk/array. I put it on my `/tmp` ram but /boot or an unassigned device can also work stop array Tools -> New Config, preserve all configurations (some settings like shutdown timeout are reset, can also restore /boot/config) start array with "Parity is valid" write 0's for the sectors with sync errors For btrfs, you have to look up the logical offsets for each extent associated with the file, and zero the those extents for the size of those extents (optional) start a parity check, there should be no corrections, but you can uncheck write corrections to see if you messed something up. There is some irony in performing this step, as it will cause essentially the same wear/workload as rebuilding, at least for HDDs where reading/writing are likely similar in wear

To expand on this, pointing directly to /mnt/cache instead of /mnt/user, it will skip the unraid shfs layer which incurs some CPU cost. It depends on the access pattern, but my theory is it is a fixed cost per request, e.g. many small requests, perhaps certain database queries, incur greater cost. One advanced way you can try to get a handle on this is checking for a lot of cpu usage of the shfs from the dockers accessing cache. You can check the cpu usage using `htop` or the glances docker app and look for shfs. All normal array access is through here though, so lots of access to the array (instead of cache) can result in high shfs usage. To see what is being requested through the shfs layer, you can run `lsof -c shfs` and see if the usage is more cache or array.

Well certainly sounds like a bug as this is not what occurs for multiple small files selected (as opposed to a folder) I've attached a log file of this scenario, let me know if you need any more information for this scenario or if I am doing something incorrectly Thanks! unbalance.log

At this point it's clear you aren't reading what I write. I told you exactly how you will know, checksums, which are built into btrfs. Let me change my language then, if the btrfs filesystem says the emulated drive from parity is good, then it's good, not "highly likely"— the only reason I used that terminology is to be mathematically precise. File integrity plugin will also get you to essentially the same place in a less real-time fashion with more steps. If you read all those same bits and the checksum comes out the same, then for all effective purposes (an essentially impossible random hash collision), it is good. There is no guessing here, please read up on cryptographic functions. As for "to an extent" that was more of a commentary on knowing "how off" your physical filesystem is. You can't really know how much of the block (or file in the plugin's case) but well, it's not really important how damaged your physical filesystem is if your emulated parity drive with checksums is good. Yes, integrating in those pieces together like zfs has advantages, particularly in terms of usability, automation, and ease, but all the information is still there. We're not going in circles, I've been explaining the same scenario since my first comment. I'm more than willing to continue to educate you on this as long as you want to respond. I've already laid out a practical implementation that works, given the other comments here stating when data is and is not written from the unraid side. You don't have to use these utilities to safeguard your data or streamline the restoration process if you don't want to.

Indeed, we still have a communication breakdown. I'll try to clarify again. No, you are covering a different case than the scenario I laid out. In your scenario, the parity is synchronized with the corrupted filesystem. It is numerically valid, but it does not satisfy my condition 2: that the parity has the correct bits, as in uncorrupted. I explicitly did not say whether or not the parity is invalid in this scenario, because it depends on the point of reference. In this scenario, it is synchronized with the uncorrupted filesystem, and out of sync with the physical corrupted filesystem. In my original comment's scenario, restoring from backup incurs a cost that is not desirable: remaining offsite backup that didn't fail is annoying to access: on a tape drive, in a different geographic location, on the cloud with expensive restore costs, etc. There are various scenarios where a write can be successful on one disk and not on another. The one I had thought up was that the connection with an externally connected SAS drive could be disrupted while the connection with the parity drive is intact. And then my actual scenario with RAM corruption, any particular computation could be corrupted, and the parity computation is separate from writing the block metadata or data itself. You could also have sata bus or CRC errors due to a bad cable. I did state these conditions up front for my scenario, if this was your big issue you could've brought this up in the beginning. I made no claims about the likelihood of this scenario, in fact in my first comment I stated it's probably rare. It doesn't mean rare cases should be ignored. The filesystem I mentioned already, btrfs, checksums both the data and metadata of every write. So yes, you can with a very high likelihood know if your parity is "correct" by emulating it, and also to know, to an extent, how "off" your physical filesystem is. And you'll get similar results with the file integrity plugin and other filesystems: once you notice your checksums are off with your physical disk, you can mount the emulated disk and see if the checksum is correct. This, of course, requires that you haven't done a parity check that writes corrections to parity, something that is explicitly recommended avoiding by those that use that plugin. There are plenty of advantages of unraid over zfs for my and other's use cases, why would I move to zfs if I can get some protection by adding additional layers on top of unraid's parity?

Ah, I see where the communication breakdown happened. I'll modify the first requirement to be: know a particular drive had a failure in particular files Yes, it's not by default in unraid that you get this, but there are tools that exist that give you this and would certainly to use them. In my case recently, it was btrfs (and it's fsck/associated tools) that alerted me to the specific files and folders with errors. It's also likely the file integrity plugin that I use would also have done so. I grabbed those files from backup. If I had known about the option I put forth in this thread, I would've at least investigated to see requirement 2 (know parity has the right bits) was satisfied by just mounting the emulated drive as true read-only and grabbing a checksum of the file. In my case, the issue ended up being bad RAM, so I would guess it's equally likely that the error was propagated both the parity and the drive vs just one of the two. I also did know that requirement 3 (the rest of the disk was good) was satisfied after extensive testing of the disk in another system and discovering the root cause.

Just thought I'd mention some unintuitive behavior for scatter: if I select multiple individual files, it attempts to place each one on a separate drive, even if there is enough space on a single drive. If there are more individual files than target drives selected, it claims there is not enough space in the target disks. E.g. I'm moving a several small log files (10 MB each), and my disks have several TB free. It will place the first log file on the first target disk, second on the next target disk, and then if there are more individual files than disks available, it will say "PLANNING: The following items will not be transferred, because there's not enough space in the target disks" I selected scatter because I want to move them from a specific full drive to other drives.

So then a single file restore with the steps I mentioned is actually currently possible then without any modifications. The required steps are to mount the disks with explicit options to do no writes: "ro,norecovery" with btrfs. xfs, reiserfs, and ext* also have similar options, and though there is some debate whether they actually work, creating a "ro,loop" device guarantees it with any block device. There is an additional layer if encrypted, so one would have to verify luks opening doesn't write any data either just in case. Thanks, this is definitely good to know and can potentially save days of rebuilding then under very specific scenarios.

Ok, then can you answer my original question? What data, if any, is written when just starting the array maintenance mode by itself? This is before any filesystems are interacted with. I did find an answer to the second half dealing with the filesystems, specifically btrfs: If it is mounted with the options "ro,norecovery" then indeed absolutely no data is written to the disk, and this step would not create a divergence between the physical disk and parity/emulated disk. So the remaining question is the preceeding steps.