Silver226

On a slight tangent, have you figured out how to make changes to age_mover stick after reboots? I made my own code changes to fix ignore files but my file gets replaced on every reboot. So I manually have to replace it at the moment (I can prob just automate via script if I get round to it). I've also tried replacing the age_mover file in the package file under /boot/config/plugins/ca.mover.tuning but that gets overwritten as well on reboot. So just wondered if there is a trick to it I'm not aware of.

No problem @foux, at least you know what the cause is now. It'd been driving me crazy for months before I figured out what was going on. Fingers crossed we'll see a proper fix soon!

Just to add my findings as ignore list was working super inconsistently for me. Basically I would see some stuff would be ignored while others would not and there was no rhyme or reason why, as it could be any folder or file. Plus it would vary from 1 day to the next, where a file was not ignored 1 day, would be ignored the next day. The TLDR is the current latest version is bugged for ignore lists if you have a lot of entries in your list. Apart from lowering the amount of entries to ignore, I found no fix bar rewriting the script. The longer version, digging through each Filtered_files_....list and Mover_action_....list, I noticed double entries for ignored files and folders. Essentially 1 saying it will be skipped and another that it will be moved. Going through the age_mover script and adding debugging commands, I got the final find command, it's loooong! The script will add a switch " -not -path \"${skipped_path}\"" or " -not -name \"${skipped_path}\"" for every entry on the ignore list. I have thousands of entries in my list, so no surprise the resultant find command is super chunky. My use case for having a large ignore list is, ctime is super inconsistent on my system, so I wrote a python script that will determine age to move and writes files and folders to the ignore list and then let mover tuning do it's thing. Copying and pasting the full find command into my unraid console displayed the same behaviour of some results being filtered while others not. Turns out find, bash or something else, has, I guess a character limit or maybe just limits amount of switches, that it will handle. When it hits the limit, it truncates and ignores the rest of the find command, so for entries that should be ignored, fall into the truncated section, they are returned in results for what to move. If I limited the number of entries on my ignore list to less than a hundred entries it seems to work fine. I didn't bother trying to work out the exact cut off point though. The code change happened in the 2024-08-15 version and seems to affect all versions after. The previous releases didn't work for me either as they seemed to ignore folders from my ignore list (plus suffered from other bugs). My only fix, was to go ahead and rewrite a large part of the code to use a different method to apply filters for ignored entries. The method I found to work was running a broad find search with all other switches minus the "-not path" and "-not name" elements, write results to a tmp file, read the tmp file and remove entries that match from the ignore list and feed the results to $FILTERFILELIST. Bar some buggy reporting (size left is always wrong), mover now works properly for me and will always ignore files from my list. Note I don't now if this is the best approach but every other thing I tried had some sort of issue. For instance, I only went with writing to tmp files to allow me to debug, when it was working, I tried just writing results to variables in memory, but that ended up being super buggy and inconsistent and at that point I couldn't be motivated to troubleshoot it further so left my version using tmp files. This might also explain why some people have issues with smart caching and etc, as they may be truncated / malformed, if used in combination with a large ignore list. Hopefully this might help someone if a newer version is developed.

Cool, looks like you have it in hand and your approach sounds sensible. Don't worry about adding to mover_tuning if it's going to be a hassle, I'm sure you have better things to work on. It was only a nice to have but I can still see directories affected from syslog so it's super low priority.

Just on, deleting empty dirs. I can confirm it no longer removes folders when doing a dry run and I can now see what folders would be affected on syslog so thank you for the change! A few observations, the script is still picking up empty folders not involved in any moves. Using a placefolder file does work for that but was thinking, would a possible approach be to do a check to see if the empty folder exists on user0 first, then move or create it on user0 if not or delete from cache if it does already exist on user0? Also would it be possible to write what folders will be removed to the Mover_tuning log file as well? Just makes it easier than going through the syslog for entries.

Hi, just tried this version and still getting similar path errors to what I had yesterday. I get loads of can't access directory error because du is splitting on spaces in path. Using an ignore file list, I now see the 3 errors raised below, the last one would cause mover to hang indefinitely. I had to use stop in the end to recover, softstop didn't do anything. I had a look at your code and think they're pretty easy to fix. For the first 2 errors wrapping $skipped_path in double quotes resolves it for me. I didn't have an issue with $SHAREPATH as none of my shares have spaces or special characters but changed that anyway in case. SKIPPED_PATH_RESERVED_SPACE=$(du -sh "$skipped_path" | awk '{print $1}' | human_to_bytes) SKIPPED_PATH_RESERVED_SPACE=$(du -sh "${skipped_path}" | awk '{print $1}') CACHESHARERESERVEDSPACE=$(du -sh "$SHAREPATH" | awk '{print $1}' | human_to_bytes) Then, when I ran it again, du command ran fine but I get the below message. Mover would again hang, needing stop to recover. Just seems like a variable mismatch error. I changed the code to the below converting $SKIPPED_PATH_RESERVED_SPACE to a numeric value and it now seems to work fine. I don't know if this also fixes the "invalid arithmetic operator" error as not been able to reproduce .0G value. I assume line 640 would need a similar adjustment but not seeing any errors with the original code. if [ ! -z "$SKIPPED_PATH_RESERVED_SPACE" ]; then SKIPPED_PATH_RESERVED_SPACE_BYTES=$(human_to_bytes "$SKIPPED_PATH_RESERVED_SPACE") echo "$PRIMARYSTORAGENAME|$PRIMARYSTORAGENAME|$SHARENAME|skipped|9999999999|$PRIMARYSIZETHRESH|$SKIPPED_PATH_RESERVED_SPACE|0|0|0|${skipped_path}" >> $FILTERED_FILELIST TOTAL_SKIPPED_PATH_RESERVED_SPACE=$((TOTAL_SKIPPED_PATH_RESERVED_SPACE + SKIPPED_PATH_RESERVED_SPACE_BYTES)) else mvlogger "Error calculating size for $skipped_path" fi Now when I run the script it completes with no errors. I've only tried a dry run so far and still need to go through the logs properly but not seeing anything obviously wrong. I'll make another post on my observations relating to empty folders as that is still picking up unusual folders

It's all good, luckily not a massive problem to recover. New changes sound positive. For the unexpected token error, it just seems the command isn't escaping out some characters when running the stat -c%W command, all examples I've seen have brackets somewhere in the path. I also just noticed I get loads of the below appear in the logs as well. With that one it seems to truncate and line cartridge where there are spaces in the path so something like "/mnt/user/Download/Dir1/dir2/dir 5/Dir6 1234 5678/File 1234 5678.txt" Comes out like below so I'm guessing a split command or something is doing something screwy. move: stat: cannot statx '/mnt/user/Download/Dir1/dir2/dir': No such file or directory move: stat: cannot statx '5/Dir6': No such file or directory move: stat: cannot statx '1234': No such file or directory move: stat: cannot statx '5678': No such file or directory move: stat: cannot statx 'File': No such file or directory move: stat: cannot statx '1234': No such file or directory move: stat: cannot statx '5678.txt': No such file or directory The files that don't error, don't look particularly different as they have spaces, brackets, same file paths etc, so not sure why it fails for some and not others. I'll try it with mtime set and see if it does the same and try spot any patterns and report back. Yeah, I've never found ctime particularly reliable so don't envy you trying to figure out a solution! At the mo I workaround it with a python script that figures out the logic for time based and write files paths to an ignore list. I just use ctime in the mover as a safety net, so nothing too new accidentally slips in a move when it shouldn't.

Hi, I just installed the latest version. I set my configuration and ran a dry run. It went mostly fine but it's removed all empty folders. I know it was mentioned above and the workaround is to use a placeholder file but I just didn't anticipate I'd need to do that as part of a dry run. It looks like it's removed all empty folders, even though it was not involved in the Move process. It's also removed hidden / system folders like .Recycle.Bin. I had some of the empty folders listed in my ignorelist.txt file but I'm guessing the function does not reference that. Lastly, the script does not write what folders were removed to Mover_tuning_***.log or to syslog, so I'm having to try figure out what was removed manually which is a minor pain. Lastly, I know it's been already reported but I also get the below error related to CTIME move: sh: -c: line 1: syntax error near unexpected token `(' Appreciate the rework is still a WIP and your efforts so this is just one of those things but thought I'd feedback