Silver226

Cool, looks like you have it in hand and your approach sounds sensible. Don't worry about adding to mover_tuning if it's going to be a hassle, I'm sure you have better things to work on. It was only a nice to have but I can still see directories affected from syslog so it's super low priority.

Just on, deleting empty dirs. I can confirm it no longer removes folders when doing a dry run and I can now see what folders would be affected on syslog so thank you for the change! A few observations, the script is still picking up empty folders not involved in any moves. Using a placefolder file does work for that but was thinking, would a possible approach be to do a check to see if the empty folder exists on user0 first, then move or create it on user0 if not or delete from cache if it does already exist on user0? Also would it be possible to write what folders will be removed to the Mover_tuning log file as well? Just makes it easier than going through the syslog for entries.

Hi, just tried this version and still getting similar path errors to what I had yesterday. I get loads of can't access directory error because du is splitting on spaces in path. Using an ignore file list, I now see the 3 errors raised below, the last one would cause mover to hang indefinitely. I had to use stop in the end to recover, softstop didn't do anything. I had a look at your code and think they're pretty easy to fix. For the first 2 errors wrapping $skipped_path in double quotes resolves it for me. I didn't have an issue with $SHAREPATH as none of my shares have spaces or special characters but changed that anyway in case. SKIPPED_PATH_RESERVED_SPACE=$(du -sh "$skipped_path" | awk '{print $1}' | human_to_bytes) SKIPPED_PATH_RESERVED_SPACE=$(du -sh "${skipped_path}" | awk '{print $1}') CACHESHARERESERVEDSPACE=$(du -sh "$SHAREPATH" | awk '{print $1}' | human_to_bytes) Then, when I ran it again, du command ran fine but I get the below message. Mover would again hang, needing stop to recover. Just seems like a variable mismatch error. I changed the code to the below converting $SKIPPED_PATH_RESERVED_SPACE to a numeric value and it now seems to work fine. I don't know if this also fixes the "invalid arithmetic operator" error as not been able to reproduce .0G value. I assume line 640 would need a similar adjustment but not seeing any errors with the original code. if [ ! -z "$SKIPPED_PATH_RESERVED_SPACE" ]; then SKIPPED_PATH_RESERVED_SPACE_BYTES=$(human_to_bytes "$SKIPPED_PATH_RESERVED_SPACE") echo "$PRIMARYSTORAGENAME|$PRIMARYSTORAGENAME|$SHARENAME|skipped|9999999999|$PRIMARYSIZETHRESH|$SKIPPED_PATH_RESERVED_SPACE|0|0|0|${skipped_path}" >> $FILTERED_FILELIST TOTAL_SKIPPED_PATH_RESERVED_SPACE=$((TOTAL_SKIPPED_PATH_RESERVED_SPACE + SKIPPED_PATH_RESERVED_SPACE_BYTES)) else mvlogger "Error calculating size for $skipped_path" fi Now when I run the script it completes with no errors. I've only tried a dry run so far and still need to go through the logs properly but not seeing anything obviously wrong. I'll make another post on my observations relating to empty folders as that is still picking up unusual folders

It's all good, luckily not a massive problem to recover. New changes sound positive. For the unexpected token error, it just seems the command isn't escaping out some characters when running the stat -c%W command, all examples I've seen have brackets somewhere in the path. I also just noticed I get loads of the below appear in the logs as well. With that one it seems to truncate and line cartridge where there are spaces in the path so something like "/mnt/user/Download/Dir1/dir2/dir 5/Dir6 1234 5678/File 1234 5678.txt" Comes out like below so I'm guessing a split command or something is doing something screwy. move: stat: cannot statx '/mnt/user/Download/Dir1/dir2/dir': No such file or directory move: stat: cannot statx '5/Dir6': No such file or directory move: stat: cannot statx '1234': No such file or directory move: stat: cannot statx '5678': No such file or directory move: stat: cannot statx 'File': No such file or directory move: stat: cannot statx '1234': No such file or directory move: stat: cannot statx '5678.txt': No such file or directory The files that don't error, don't look particularly different as they have spaces, brackets, same file paths etc, so not sure why it fails for some and not others. I'll try it with mtime set and see if it does the same and try spot any patterns and report back. Yeah, I've never found ctime particularly reliable so don't envy you trying to figure out a solution! At the mo I workaround it with a python script that figures out the logic for time based and write files paths to an ignore list. I just use ctime in the mover as a safety net, so nothing too new accidentally slips in a move when it shouldn't.

Hi, I just installed the latest version. I set my configuration and ran a dry run. It went mostly fine but it's removed all empty folders. I know it was mentioned above and the workaround is to use a placeholder file but I just didn't anticipate I'd need to do that as part of a dry run. It looks like it's removed all empty folders, even though it was not involved in the Move process. It's also removed hidden / system folders like .Recycle.Bin. I had some of the empty folders listed in my ignorelist.txt file but I'm guessing the function does not reference that. Lastly, the script does not write what folders were removed to Mover_tuning_***.log or to syslog, so I'm having to try figure out what was removed manually which is a minor pain. Lastly, I know it's been already reported but I also get the below error related to CTIME move: sh: -c: line 1: syntax error near unexpected token `(' Appreciate the rework is still a WIP and your efforts so this is just one of those things but thought I'd feedback