fry_the_solid

To be clear, 'poweroff' will not perform a clean shutdown

I understand which threshold you mean now, though I don't understand why it needs to be 10%. I'd like it to move any files on my cache to my array, even if it only takes up 1% of the cache so that they will be protected by parity. However, I did set the moving threshold to 10% and ran it again (with the cache >10% full) with the same files as in my above post and got the same results. I don't believe I have an override dir in my plugin config. I've attached my config file but looking at it I don't see anything like that. Wouldn't the fact that all my files are moving as expected except those with special characters filtered by the 'validate input filenames' setting imply that the issue is with those special characters? ca.mover.tuning.cfg

I'm sorry I don't understand what you mean by this. What overrides? Also is it the 'Fill up/prime up to this level of used Primary (cache) space' setting you are referring to? I wouldn't think that would matter as the share I am testing with is 'cache -> array'. I tested it again changing that setting to 10% as you can see in the screenshot below, but otherwise using the same settings as above. I used 5MB files this time. I also reinstalled the plugin for good measure. The commands are included in a screenshot below, and I list the files before and after running the mover. I also attached the logs from this most recent test. Mover_action_2025-04-23T023816.list Mover_tuning_2025-04-23T023816.log Debug_Mover_tuning_2025-04-23T023816.log Filtered_files_2025-04-23T023816.list

correct Attached: - Screenshot of my settings - Screenshot of me creating the test files and listing the files before and after the mover is run - Requested log files Filtered_files_2025-04-22T112656.list Mover_tuning_2025-04-22T112656.log Debug_Mover_tuning_2025-04-22T112656.log

What setting are you referring to? Is it 'Free down/prime up to this level of used Primary (cache) space'? If so that is set to 0%. As far as I can tell I have no filters enabled, all files should be moved each run.

I'm having an issue transferring files with the characters filtered by the 'Validate input filenames to prevent attacks' setting, such as '$', despite the setting being off/set to 'no'. For example, I have a file, 'a$test.mkv' and again 'Validate input filenames to prevent attacks' is set to 'no'. However, when I run the mover it does not move and it still shows up on the filtered_files list in the log. Any idea why this is happening?

I just have to say thank you to @metabubble for identifying the cause of this issue. I've had a problem with this for as long as I've used Unraid and this is certainly it. With 64GB of RAM and a dirty_ratio of 20% Plex and other processes were only able to access the disk being written to briefly every 12.8 GB, if I'm understanding correctly. I tested his solution, reducing the dirty_ratio to 1, and sure enough it fixed it—the mover, rsync, and any other continuous write to a disk no longer bottlenecked my system. Although it probably wouldn't affect my system much, I didn't want to keep it at 1 all the time, so I wrote a 2 line script that lowered the dirty_ratio to 1 and another that raised it back to 20, and used the CA Mover Tuning plugin to run the scripts before/after the mover. Thanks again.

Thank you again for the advice Jorge, this solution is so much better that what I was using before. Both my primary and backup server are using disks of the same size, and (single-disk) btrfs, so I am able to backup the disks 1:1. Fortunately for me it was a perfect solution. In case anyone is in the same situation and wants to look into it, start with btrbk. I spent about 3 hours writing my own script before I stumbled upon it. It's easy to configure and does just about everything you could want. However, this alternative may not work for everyone. If the source filesystem was not btrfs, this wouldn't work. Similarly, if the disks in the destination are not equal in size to the source, it will be a challenge/not possible to backup using this solution either. It's true that if the disks are in the same btrfs pool this wouldn't be an issue, but then you need to use either raid0 with no redundancy or raid1cX where at most you're able to utilize 50% of your raw storage capacity. In the context of Unraid, I'd guess that this would be an issue for many users. edit: I believe I understand the issue now, it is a problem with BTRFS. Whenever you overwrite existing data in BTRFS it is not actually overwritten but rather a new block is written to, but for whatever reason with these inplace transfers (and similar cases like writing to VM disk images) it does not delete the old block leading to unusable/wasted space. You can use a tool called btdu to view how your space is being used; the space that has been overwritten but not deleted is listed as "unreachable". The only solution I found were to run a btrfs defrag (though I didn't test this myself), or transfer the file off of the volume then back on, however neither of these are something you'd want to do often. All in all an unfortunate limitation (bug?) of BTRFS.

@JorgeB I did a little reading on it and if I'm understanding correctly, this won't be possible in Unraid if the destination disks are not of equal or greater size to the source disks. For example I couldn't backup a 16TB disk to 2 x 8TB disks.

I didn't know about incremental send/receive, I'll use that instead. Thanks!

What's going on: I have a 1 TB file that I routinely backup with rsync --inplace from one unraid server to another unraid server. Both the source and destination are using BTRFS for their filesystems (single-disk). The file is encrypted and has been backed up before so the full 1 TB for the file is fully allocated on the destination I recently changed about 500 GB of the file and backed it up again. During the transfer I noticed that my free space on the disk was going down and by the end of the transfer I had lost about 500 GB of free space. I would expect it to stay the same, as it should be overwriting already allocated space. I unmounted, restarted, remounted but it stayed the same size du -s showed about what I expected, but df was off by about 500 GB, which makes me think the issue has something to do with the filesystem I deleted the file and sure enough it freed up over 1500 GB I'd appreciate any insight into this issue. I'm guessing that not transferring with inplace would 'fix' it, but I don't want to do a 1 TB write every backup when usually I only need to transfer a gig or two. Not to mention all my VM disks. Also while in this case it was happening with one large file making it obvious, I assume it happens with files of all sizes. Also here is the rsync command I used: rsync -aHXxrv --numeric-ids --inplace --no-whole-file --delete "$source" "$destination"