[Plugin] Mover Tuning

Faceman · August 12

2 hours ago, Reynald said:

Hi @Faceman

/./ in rsync is for relative path. From rsync man:

It's also how unbalanced plugin works.

That's a bug, it's corrected (see below)

Ahh that makes sense. Neat

AngelAbaddon · August 13

Just moved from the deprecated version and running into a few issues - firstly I've looked back and it looks like it was meant to be fixed, but the CTIME option seems to still be having issues with special characters:

sh: -c: line 1: syntax error near unexpected token `('

Leaving it as MTIME works as expected.

My other issue is the deleting of empty directories is causing an issue as my main share holds both my media and usenet folders - since the usenet folders should be structured but empty unless it's processing downloads, when the mover is deleting the folders it's causing sabnzbd to freak out since they no longer exist.

Reynald · August 13

Hi,

Thank you for your report

For ctime, I'm going to check that again.

For deletion empty directory, you can create a .placeholder or any other empty file in those you do not want to be deleted. In my tests I did not encounter that as I always have some dusts in usenet folders. I will try something cleaner.

AngelAbaddon · August 13

No worries, absolutely love the new features, has some major ones I've been after for a long while now. Nice idea on the placeholder, that should hold things together for now.

ZGief · August 13

Question about the combination of hardlinks and moving files from cache to array.

Following trashguides i have two folders each contains the same file with different names, but they are hardlinked.

I want to keep newer files in cache and move the older ones to the array, but I want the hardlinked files to move at the same time to the array and be hardlinked when they are on the array (don't want dupes that take up 2x space).

What is the best config to achieve this?

My thinking is:

- Move about 70% full - no needs to move anything until cache is getting full

- When move does run, either: only move files > 7days old.... or... move files until cache is 30% full (not sure which one)

The two things I want to achieve are:

Hardlinks are always maintained on both the cache & array - don't want dupes, don't want one file on cache and the other on array (not sure how this is possible, maybe with % settings)
Newer files stay on cache as long as possible

Any thoughts/guidance would be appreciated!

Reynald · August 13

Hardlinks are preserved. It was the case in 2023 versions also.
A file won't be moved on cache without all files hardlinked to it
Every hardlinked files will be recreated on array.

You can check Mover_action file in /tmp/ca.mover.tuning/: the NBLINKS column represent the number of hardlinks to a file.
Then you have the inode, which represent the unique file on disk. The filesize of hardlinks is set to zero (to not be counted x times when calculating sizes)

The mover sends all the inode to the same batch to rsync with the --hard-links option, so it recreates hardlinks on destination

Your settings seems great for what you are trying to achieve.

Silver226 · August 14

Hi, I just installed the latest version. I set my configuration and ran a dry run. It went mostly fine but it's removed all empty folders. I know it was mentioned above and the workaround is to use a placeholder file but I just didn't anticipate I'd need to do that as part of a dry run.

It looks like it's removed all empty folders, even though it was not involved in the Move process. It's also removed hidden / system folders like .Recycle.Bin. I had some of the empty folders listed in my ignorelist.txt file but I'm guessing the function does not reference that. Lastly, the script does not write what folders were removed to Mover_tuning_***.log or to syslog, so I'm having to try figure out what was removed manually which is a minor pain.

Lastly, I know it's been already reported but I also get the below error related to CTIME

move: sh: -c: line 1: syntax error near unexpected token `('

Appreciate the rework is still a WIP and your efforts so this is just one of those things but thought I'd feedback

Reynald · August 14

Hi,

I think I have to rewrite the function, it's too efficient at cleaning it seems.
I went the easy, quick, and efficient way by looking for empty dirs since root of a share where files were moved.
I've just added some verification and limits: added a dry-run in testmode, and do not go below a min depth of 2.

For ctime, I have to debug it yes. It's quiet inefficient (long process) to look at ctime when filtering files and not all filesystem handle ctime correctly (especially with Copy On Write), but there is a real use case for that.

Oh, and thank you for the report

Edited August 14 by Reynald

Silver226 · August 14

It's all good, luckily not a massive problem to recover. New changes sound positive.

For the unexpected token error, it just seems the command isn't escaping out some characters when running the stat -c%W command, all examples I've seen have brackets somewhere in the path. I also just noticed I get loads of the below appear in the logs as well. With that one it seems to truncate and line cartridge where there are spaces in the path so something like

"/mnt/user/Download/Dir1/dir2/dir 5/Dir6 1234 5678/File 1234 5678.txt"

Comes out like below so I'm guessing a split command or something is doing something screwy.

move: stat: cannot statx '/mnt/user/Download/Dir1/dir2/dir': No such file or directory
move: stat: cannot statx '5/Dir6': No such file or directory
move: stat: cannot statx '1234': No such file or directory
move: stat: cannot statx '5678': No such file or directory
move: stat: cannot statx 'File': No such file or directory
move: stat: cannot statx '1234': No such file or directory
move: stat: cannot statx '5678.txt': No such file or directory

The files that don't error, don't look particularly different as they have spaces, brackets, same file paths etc, so not sure why it fails for some and not others. I'll try it with mtime set and see if it does the same and try spot any patterns and report back.

Yeah, I've never found ctime particularly reliable so don't envy you trying to figure out a solution! At the mo I workaround it with a python script that figures out the logic for time based and write files paths to an ignore list. I just use ctime in the mover as a safety net, so nothing too new accidentally slips in a move when it shouldn't.

Reynald · August 14

Got it

Doing some testing and pushing an update later

Reynald · August 14

2024.08.15

Even Better cache priming (hopefully) (R3yn4ld)
- Rewriten Ignore filelist from file and filetypes filtering functions (major => test mode)
- Improve calculating size of filtered files and filetypes
- Update calculation from basic/bc to numfmt. Removed bc option
- Added verification for not breaking hardlinks when an hardlinked file is filtered
Added testmode to cleaning empty folder function, and a min depth of 2.
Fix ctime bug (R3yn4ld)

Faceman · August 14

should the empty folder cleanup really be trying to clean up folders in shares that are set to only reside on the cache and have no secondary location? or do I need to manually add them to the exceptions?

I'm not sure how and why but I just let it run out of test mode and it removed a huge number of what seemed to be empty directories in my docker share, they seem to have all been empty but it still caused some weirdness with losing my docker folder icons and such. I think they were leftover from when I was running a docker folder rather than currently back to using an image, but it shouldn't been removing things in shares that the mover should ignore.

It might have been unrelated since it fixed itself after some time, but still I think it should ignore cache only shares by default

Edited August 15 by Faceman

Reynald · August 15

7 hours ago, Faceman said:

should the empty folder cleanup really be trying to clean up folders in shares that are set to only reside on the cache and have no secondary location? or do I need to manually add them to the exceptions?

Here is how folder cleaner works:

- It looks for shares where files have been previously moved

- It find empty folder (litteraly, empty, not any file inside)

- It remove the folder
Since 2024.08.15, it also:

- Apply filter from exclusion

- Have a min depth of 2: /mnt/*/share/dir1/dir2 and won't delete below dir3 then

7 hours ago, Faceman said:

I'm not sure how and why but I just let it run out of test mode and it removed a huge number of what seemed to be empty directories in my docker share, they seem to have all been empty but it still caused some weirdness with losing my docker folder icons and such. I think they were leftover from when I was running a docker folder rather than currently back to using an image, but it shouldn't been removing things in shares that the mover should ignore.

It might have been unrelated since it fixed itself after some time, but still I think it should ignore cache only shares by default

I don't think it's related, because it won't delete any file (it would throw error if there are files in a folder it attempt to delete)

Jensus11 · August 15

Hello,

if I start the mover manually, I get this message in the log.

But the mover never stops.

What is that, or what can I do?

Quote

ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start

Reynald · August 15

4 minutes ago, Jensus11 said:

Hello,

if I start the mover manually, I get this message in the log.

But the mover never stops.

What is that, or what can I do?

Hi,

This message is very normal, it is the mechanism to start the mover taking these options into account:
image.png.cf577a0f6d480f646bebb9a4b56c22e4.png

Do you have some verbosity from the mover?
It should if this option is enabled:
image.png.550c7c10a621eaaaf75ee18bbabe9ed0.png
This option will also create a log file: /tmp/ca.mover.tuning/Mover_tuning_(date).log

Three other files are also created in /tmp/ca.mover.tuning, the most usefull are Filtered_filelist and Mover_action.

Jensus11 · August 15

The Log option is active, but where I find the /tmp/ folder?

Reynald · August 15

The folder is /tmp/ca.mover.tuning/
You can ls to it in the terminal, or use a File manager.

There is nothing else in the syslog? No error, nothing?

Jensus11 · August 15

In the File manager, I can't find the tmp folder.

I deactivated the "Mover - Protokollierung", because "Fix Common Problems" said:

With activated "Mover - Protokollierung" ran great and said WE ARE DONE.

Is that right that the Protokollierung must be activated?

Reynald · August 15

To check for mover tuning files, you can:

open a terminal and then:

cd /tmp/ca.mover.tuning/
ls

Regarding your option "Mover - logging" should not made difference for Mover Tuning (or maybe prevent mover tuning logs to appear, I have to check/test)

I will check about verbosity in different logging option cases.

Silver226 · August 15

14 hours ago, Reynald said:

2024.08.15

Even Better cache priming (hopefully) (R3yn4ld)

Rewriten Ignore filelist from file and filetypes filtering functions (major => test mode)

Improve calculating size of filtered files and filetypes

Update calculation from basic/bc to numfmt. Removed bc option

Added verification for not breaking hardlinks when an hardlinked file is filtered

Added testmode to cleaning empty folder function, and a min depth of 2.

Fix ctime bug (R3yn4ld)

Hi, just tried this version and still getting similar path errors to what I had yesterday. I get loads of can't access directory error because du is splitting on spaces in path.

Using an ignore file list, I now see the 3 errors raised below, the last one would cause mover to hang indefinitely. I had to use stop in the end to recover, softstop didn't do anything.

Quote

move: du: cannot access 'dir': No such file or directory
move: Error calculating size for /mnt/cache/dir...
move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 623: 7.0G: syntax error: invalid arithmetic operator (error token is ".0G")

I had a look at your code and think they're pretty easy to fix. For the first 2 errors wrapping $skipped_path in double quotes resolves it for me. I didn't have an issue with $SHAREPATH as none of my shares have spaces or special characters but changed that anyway in case.

SKIPPED_PATH_RESERVED_SPACE=$(du -sh "$skipped_path" | awk '{print $1}' | human_to_bytes)
SKIPPED_PATH_RESERVED_SPACE=$(du -sh "${skipped_path}"  | awk '{print $1}')
CACHESHARERESERVEDSPACE=$(du -sh "$SHAREPATH" | awk '{print $1}' | human_to_bytes)

Then, when I ran it again, du command ran fine but I get the below message. Mover would again hang, needing stop to recover.

Quote

move: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 623: 20G: value too great for base (error token is "20G")

Just seems like a variable mismatch error. I changed the code to the below converting $SKIPPED_PATH_RESERVED_SPACE to a numeric value and it now seems to work fine. I don't know if this also fixes the "invalid arithmetic operator" error as not been able to reproduce .0G value. I assume line 640 would need a similar adjustment but not seeing any errors with the original code.

if [ ! -z "$SKIPPED_PATH_RESERVED_SPACE" ]; then
        SKIPPED_PATH_RESERVED_SPACE_BYTES=$(human_to_bytes "$SKIPPED_PATH_RESERVED_SPACE")
        echo "$PRIMARYSTORAGENAME|$PRIMARYSTORAGENAME|$SHARENAME|skipped|9999999999|$PRIMARYSIZETHRESH|$SKIPPED_PATH_RESERVED_SPACE|0|0|0|${skipped_path}" >> $FILTERED_FILELIST
        TOTAL_SKIPPED_PATH_RESERVED_SPACE=$((TOTAL_SKIPPED_PATH_RESERVED_SPACE + SKIPPED_PATH_RESERVED_SPACE_BYTES))
    else
        mvlogger "Error calculating size for $skipped_path"
fi

Now when I run the script it completes with no errors. I've only tried a dry run so far and still need to go through the logs properly but not seeing anything obviously wrong. I'll make another post on my observations relating to empty folders as that is still picking up unusual folders

Silver226 · August 15

Just on, deleting empty dirs. I can confirm it no longer removes folders when doing a dry run and I can now see what folders would be affected on syslog so thank you for the change!

A few observations, the script is still picking up empty folders not involved in any moves. Using a placefolder file does work for that but was thinking, would a possible approach be to do a check to see if the empty folder exists on user0 first, then move or create it on user0 if not or delete from cache if it does already exist on user0? Also would it be possible to write what folders will be removed to the Mover_tuning log file as well? Just makes it easier than going through the syslog for entries.

Jensus11 · August 15

oh sorry, I looked in the /tmp/CA_logs folder. My mistake.

AngelAbaddon · August 15

Just done the update, looks like CTIME is fixed. I think the folder thing is likely going to take some time to find a graceful solution to if one is possible, in my case it's still deleting some of the empty usenet folders (/mnt/cache_data/data/usenet/incomplete for example), but I'm happy to stick with the placeholder file for now, but thanks for working on it.

Bigger issue I've discovered is Share overrides aren't reverting to global settings:

Aug 15 15:00:40 Cerebro move: ------------------------------------------------------------- Processing backup share -----------------------------------------------------------
Aug 15 15:00:40 Cerebro move: Settings override:
Aug 15 15:00:40 Cerebro move: ------------------
Aug 15 15:00:40 Cerebro move: Using share moving threshold: 0 %
Aug 15 15:00:40 Cerebro move: Using share freeing threshold: 0 %
Aug 15 15:00:40 Cerebro move: Age: no = 0 ; daysold: 
Aug 15 15:00:40 Cerebro move: ------------------------------------------------------------------------  -----------------------------------------------------------------------
Aug 15 15:00:40 Cerebro move: Primary storage: cache_data - size: 932GiB - used:  70 % (649GiB)
Aug 15 15:00:40 Cerebro move: Secondary storage: user0
Aug 15 15:00:40 Cerebro move: /mnt/cache_data/backup does not exist. Is the share still used? Consider removing /boot/config/shares/backup.cfg if not.
Aug 15 15:00:40 Cerebro move: => Skipping
Aug 15 15:00:40 Cerebro move: ------------------------------------------------------------- Processing cloud share ------------------------------------------------------------
Aug 15 15:00:40 Cerebro move: Primary storage: cache_data - size: 932GiB - used:  70 % (649GiB)
Aug 15 15:00:40 Cerebro move: Secondary storage: none
Aug 15 15:00:40 Cerebro move: Share Information: Name: cloud - Path: /mnt/cache_data/cloud
Aug 15 15:00:40 Cerebro move: Mover action: no action, only cache_data used (cache:only).
Aug 15 15:00:40 Cerebro move: => Nothing will be moved. Share usage is taken into account in the calculation of the threshold for other shares.
Aug 15 15:00:40 Cerebro move: Calculating share usage... (can take a moment)
Aug 15 15:00:40 Cerebro move: cache_data/cloud used: 380MiB
Aug 15 15:00:40 Cerebro move: -------------------------------------------------------------- Processing data share ------------------------------------------------------------
Aug 15 15:00:40 Cerebro move: Primary storage: cache_data - size: 932GiB - used:  70 % (649GiB)
Aug 15 15:00:40 Cerebro move: Secondary storage: user0
Aug 15 15:00:40 Cerebro move: Share Information: Name: data - Path: /mnt/cache_data/data
Aug 15 15:00:40 Cerebro move: Moving threshold: 0% (0B) ; Freeing threshold: 0% (0B)
Aug 15 15:00:40 Cerebro move: Mover action: cache_data->user0 (cache:yes). Pool is above moving threshold percentage:   70% >= 0%.
Aug 15 15:00:40 Cerebro move: => Will smart move old files from cache_data to user0. Nothing will be moved from user0 to cache_data

My global threshold settings are 70%, but the backup share overrides that to 0% (I want this moved every run regardless). In the prior version, once the backup share was run, it would revert the settings to the global ones so data is set back to 70% (can actually see this in the log from the live run that occured earlier this morning), but it looks like thats being missed as data is treating the threshold as 0% still.

Edit: Just noticed the backup folder does not exist - guessing it's to do with the folder deletion as I didn't have a placeholder in there. I'll get this set back up and drop one in and see what happens.

Edit 2: Looks like the share folder getting deleted was the cause, now that I've locked it with a placeholder, the global settings are getting reapplied.

It also helped me notice the placeholders were getting treated in the same way as normal files, "Ignore all hidden files and folders" didn't work, but adding the placeholder name (minus the .) to the "ignore file types" does look to have sorted it.

Edited August 15 by AngelAbaddon

Reynald · August 15

1 hour ago, Silver226 said:
Hi, just tried this version and still getting similar path errors to what I had yesterday. I get loads of can't access directory error because du is splitting on spaces in path.

Using an ignore file list, I now see the 3 errors raised below, the last one would cause mover to hang indefinitely. I had to use stop in the end to recover, softstop didn't do anything.

I had a look at your code and think they're pretty easy to fix. For the first 2 errors wrapping $skipped_path in double quotes resolves it for me. I didn't have an issue with $SHAREPATH as none of my shares have spaces or special characters but changed that anyway in case.
SKIPPED_PATH_RESERVED_SPACE=$(du -sh "$skipped_path" | awk '{print $1}' | human_to_bytes)
SKIPPED_PATH_RESERVED_SPACE=$(du -sh "${skipped_path}"  | awk '{print $1}')
CACHESHARERESERVEDSPACE=$(du -sh "$SHAREPATH" | awk '{print $1}' | human_to_bytes)
Then, when I ran it again, du command ran fine but I get the below message. Mover would again hang, needing stop to recover.

Just seems like a variable mismatch error. I changed the code to the below converting $SKIPPED_PATH_RESERVED_SPACE to a numeric value and it now seems to work fine. I don't know if this also fixes the "invalid arithmetic operator" error as not been able to reproduce .0G value. I assume line 640 would need a similar adjustment but not seeing any errors with the original code.
if [ ! -z "$SKIPPED_PATH_RESERVED_SPACE" ]; then
        SKIPPED_PATH_RESERVED_SPACE_BYTES=$(human_to_bytes "$SKIPPED_PATH_RESERVED_SPACE")
        echo "$PRIMARYSTORAGENAME|$PRIMARYSTORAGENAME|$SHARENAME|skipped|9999999999|$PRIMARYSIZETHRESH|$SKIPPED_PATH_RESERVED_SPACE|0|0|0|${skipped_path}" >> $FILTERED_FILELIST
        TOTAL_SKIPPED_PATH_RESERVED_SPACE=$((TOTAL_SKIPPED_PATH_RESERVED_SPACE + SKIPPED_PATH_RESERVED_SPACE_BYTES))
    else
        mvlogger "Error calculating size for $skipped_path"
fi
Now when I run the script it completes with no errors. I've only tried a dry run so far and still need to go through the logs properly but not seeing anything obviously wrong. I'll make another post on my observations relating to empty folders as that is still picking up unusual folders

Great job, thank you for your review and patches ! I'm going to implement this

Reynald · August 15

58 minutes ago, Silver226 said:

Just on, deleting empty dirs. I can confirm it no longer removes folders when doing a dry run and I can now see what folders would be affected on syslog so thank you for the change!

A few observations, the script is still picking up empty folders not involved in any moves. Using a placefolder file does work for that but was thinking, would a possible approach be to do a check to see if the empty folder exists on user0 first, then move or create it on user0 if not or delete from cache if it does already exist on user0? Also would it be possible to write what folders will be removed to the Mover_tuning log file as well? Just makes it easier than going through the syslog for entries.

The code block was misplaced and that's why it picks to many dirs.
I added options to tune or disable empty folder cleaner.

It will be difficult to log un Mover_tuning log as well, but I'll give it a try

[Plugin] Mover Tuning

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid

hugenbdd

hugenbdd

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation