[Plugin] Mover Tuning

[methanoid]

Can someone tell me my stupid mistake, please?? It's still moving stuff I want to keep on my cache drive

 

cat /mnt/cache/appdata/ignore.files 

 

/mnt/cache/data/torrents
/mnt/cache/data/usenet
/mnt/cache/data/nzb
/mnt/cache/data/post
/mnt/cache/data/backups (no CR at end of line - do I need??)

 

[Reynald]

4 minutes ago, methanoid said:

Can someone tell me my stupid mistake, please?? It's still moving stuff I want to keep on my cache drive

It might also be a stupid bug   (I won't tell, because you are using an old version and due to respect I have to other devs of this plugins, I won't tell the previous dev is stupid, he's not, probably, certainly,...  )

Can you please post the log ? 

We will focus on a line stating:

skipfilelist string: find "/mnt/ ....

 

[alturismo]

39 minutes ago, Reynald said:

Can you please PM me to give me some details about the player, the Alturismo's way to make the excluded list... ?

i will  thanks ahead, but really only when you find the spare time and the motivation therefore

[Reynald]

I need this TVShow function for my own use case also (I've teased this function already, based on an old script of mine), and I need a tester because it's a major change and I cannot test (think about) all scenarios before releasing. Actually, my code is just reviewed and tested by Gemini and Copilot AI. So if you agree, I need your help and if you have also some spare time, we could experiment together!

I'm going to create a beta version for those willing to try.

[Notwist]

11 hours ago, Reynald said:

OK I will investigate.

Was this the case with older versions (prior to 2024.06) ?

It worked about a week and a half ago, and then I noticed it no longer works. In fact, it even ignores my "ignore files" list as well. It will run pre-scripts though. How can I install the 7-10-24 build in the meantime?

 

EDIT: I got 7-10-24 build installed and it doesn't work at all. When I run it, this is the log output:

Aug  2 13:59:55 Calcifer emhttpd: shcmd (121): /usr/local/sbin/mover |& logger -t move &
Aug  2 13:59:55 Calcifer move: Log Level: 1
Aug  2 13:59:55 Calcifer move: mover: started
Aug  2 13:59:55 Calcifer move: mover: finished

 

 

I have tried this with ignore hidden files and folders checked and unchecked, it just doesn't want to move anything period. I created a test case with 1 file that should absolutely be moved but it's not being touched. Any assistance would be greatly appreciated.

 

UPDATE: I've reinstalled the latest version of the plugin, and with the same settings (ignore hidden files and folders) doing a test move, the expanded log details don't even mention the filter being used in the output, and proceeds to move the items in the hidden folders anyway. At this point, there isn't a version of Mover Tuning that works for me. Not sure what to do.

Edited August 2 by helpful-tune3401

[Reynald]

1 hour ago, helpful-tune3401 said:

It worked about a week and a half ago, and then I noticed it no longer works. In fact, it even ignores my "ignore files" list as well. It will run pre-scripts though. How can I install the 7-10-24 build in the meantime?

 

EDIT: I got 7-10-24 build installed and it doesn't work at all. When I run it, this is the log output:

Aug  2 13:59:55 Calcifer emhttpd: shcmd (121): /usr/local/sbin/mover |& logger -t move &
Aug  2 13:59:55 Calcifer move: Log Level: 1
Aug  2 13:59:55 Calcifer move: mover: started
Aug  2 13:59:55 Calcifer move: mover: finished

 

 

I have tried this with ignore hidden files and folders checked and unchecked, it just doesn't want to move anything period. I created a test case with 1 file that should absolutely be moved but it's not being touched. Any assistance would be greatly appreciated.

 

UPDATE: I've reinstalled the latest version of the plugin, and with the same settings (ignore hidden files and folders) doing a test move, the expanded log details don't even mention the filter being used in the output, and proceeds to move the items in the hidden folders anyway. At this point, there isn't a version of Mover Tuning that works for me. Not sure what to do.

 

Can you please pack /tmp/ça.mover.tuning/ and share the zip here or via PM?

[Notwist]

3 hours ago, Reynald said:

 

Can you please pack /tmp/ça.mover.tuning/ and share the zip here or via PM?

I'll get that together in a bit, but think I noticed what could be the issue: it successfully lists the files to be filtered in the filter list, but the mover log shows it actually moves a redundant reference.

 

For example, it shows it's filtering everything in

 

/mnt/optanecache/media/.files

 

but the mover log shows it moves those same files using the directory path

 

/mnt/optanecache/./media/.files

 

This would also explain why my item filter was being ignored as well.

 

UPDATE: uninstalled, rebooted, and used standard mover, and it doesn't user redundant references. So this is unique to Mover Tuning

ca.mover.tuning.tar.gz

Edited August 2 by helpful-tune3401
Uploaded requested files

[Peuuuur Noel]

Hi,

I've setup a share with primary on cache, secondary on array and cache mode on "Prefer/Array->Cache" because I prefer to keep these data on cache (SSD) and overflow on array (HDD) if not enough space.

On previous version (last 2023.12.19), mover move data from array to cache if enough space, but with this new version (current 2024-08-01), mover smart move data based on age to array.

Is there a way to prevent mover to move data from cache to array like before?

[Reynald]

14 hours ago, helpful-tune3401 said:

I'll get that together in a bit, but think I noticed what could be the issue: it successfully lists the files to be filtered in the filter list, but the mover log shows it actually moves a redundant reference.

 

For example, it shows it's filtering everything in

 

/mnt/optanecache/media/.files

 

but the mover log shows it moves those same files using the directory path

 

/mnt/optanecache/./media/.files

 

This would also explain why my item filter was being ignored as well.

 

UPDATE: uninstalled, rebooted, and used standard mover, and it doesn't user redundant references. So this is unique to Mover Tuning

ca.mover.tuning.tar.gz 1.58 kB · 1 download

Thanks for the log.

Filtered_files are to be understood as Files found according to filters, not excluded files.
Mover_actions list what going to happen (or what happend if test mode = no) to listed files.

The fact that media/.files/Screenshot 2024-07-29 at 10.45.53 AM.png is moved is a bug, I'm going to investigate while the filter list it and publish a new plugin version

[Reynald]

8 hours ago, Peuuuur Noel said:

Hi,

I've setup a share with primary on cache, secondary on array and cache mode on "Prefer/Array->Cache" because I prefer to keep these data on cache (SSD) and overflow on array (HDD) if not enough space.

On previous version (last 2023.12.19), mover move data from array to cache if enough space, but with this new version (current 2024-08-01), mover smart move data based on age to array.

Is there a way to prevent mover to move data from cache to array like before?

I'm not sure to understand what you try to achieve.

In 2024 versions, mover will move/keep most recent files to cache (not moving from array if cache mode is Yes/Cache->Array), and move older files to cache.

What do you try to achieve? What is the expected behavior?
 

[Peuuuur Noel]

1 hour ago, Reynald said:

I'm not sure to understand what you try to achieve.

In 2024 versions, mover will move/keep most recent files to cache (not moving from array if cache mode is Yes/Cache->Array), and move older files to cache.

What do you try to achieve? What is the expected behavior?
 

I'm trying to keep all files on cache for shares with cache preferred and write on array if there is no more space on cache (if full before mover runs). In 2023 version, older files was not moved to array (only array to cache if enough space).

[Reynald]

I'm sorry some notions are missing from your answer.

You want to keep older files on cache and overflow to array? No need for mover then.

When do you want files to move from cache to array? At a threshold? At age? Never? (so you basically want Cache: Only plus an overflow for new files?)

To have a similar behavior that 2023 version, I'm going to not sort files by date when something different from age=auto is selected.

[Peuuuur Noel]

I've got multiple shares with primary on cache and secondary on array.

Share A is cache preferred, when mover runs (daily, threshold), if files are on array, they are moved to cache (if enough space).

Share B will write to cache then move to array when mover runs (daily, threshold).

 

I started with cache only on share A but ended up with an problem when cache was full before mover ran and share A was not able to write data. I set to cache preferred to avoid this issue.

 

If cache is full, shares A and B write to array, then when mover runs (cron+threshold) files on B are moved from cache to array and files on A moved from array to cache.

 

It works like this on 2023 version, but on 2024 version, for shares with cache set to preferred, files are moved to array based on age.

I want them to stay on cache if set to preferred.

 

Maybe i'm wrong in my usage of this version of this plugin.

 

[Reynald]

Thank you for explanation !

You can set Share B to Cache:Yes (Cache->Array) and a share settings for mover tuning like this:

Then ALL files from share B will be moved to Array
Files of share A that are on Array will be moved back to share A if any, and if there are not enough space on cache during moving, then the overflow from cache to array will occur for older files.

You may as well use these settings instead:

(not much tested for 2024 version, please report)

If share B is set to Cache:Prefer (Array->Cache), then the most recent files from A and B stays on cache. I believe that this is the most efficient way to use a cache: keep most recent files.

 

The difference (without setting a custom config for a share) from 2023 to 2024 version here is that Cache=Yes (cache->array) files can be kept if they are recent enough wereas 2023 move all files with no question.

Edited August 3 by Reynald

[Reynald]

Update: 2024-08-04

• Fix default Settings handling causing a "Unary operator" bug (R3yn4ld) (thanks to @alturismo)
• Add freeing threshold option (R3yn4ld)

Coming next:

• Fixing not ignoring hidden files bug
• Better repair of shares when files out of there intended place (i.e. cache: only on array, cache: no on cache, cach2 files on cache1...)

[Peuuuur Noel]

  

18 hours ago, Reynald said:

Thank you for explanation !

You can set Share B to Cache:Yes (Cache->Array) and a share settings for mover tuning like this:

Then ALL files from share B will be moved to Array
Files of share A that are on Array will be moved back to share A if any, and if there are not enough space on cache during moving, then the overflow from cache to array will occur for older files.

You may as well use these settings instead:

(not much tested for 2024 version, please report)

If share B is set to Cache:Prefer (Array->Cache), then the most recent files from A and B stays on cache. I believe that this is the most efficient way to use a cache: keep most recent files.

 

The difference (without setting a custom config for a share) from 2023 to 2024 version here is that Cache=Yes (cache->array) files can be kept if they are recent enough wereas 2023 move all files with no question.

Thanks for your message.

 

I tested but it didn't do what expected.

I've no issue with Cache:Yes (as in 2023 version), all files are moved to array when mover runs. That's OK.

 

For Cache:Prefer, files are moved to array based on age, even if I set "Move files off cache based on age" to "No" in Mover Tuning settings (should prevent it, right?).

My appdata, domains and system shares are set to Cache:Prefer and when mover runs, VM images, docker and old files are now moved to array which is annoying to have them on HDDs (array) and no longer on SSDs (cache).

I want them to always stay on cache, not to move to array. It changes from the original purpose of mover (Mover action: Array->Cache) and from the 2023 version (all files remain on cache, old files are not moved to array).

 

Also, in settings, shouldn't Cache:Yes/No/Only/Prefer be updated to new names to avoid mistakes and for new unraid users (from 6.12) started after these changes?

Yes = Primary: [Pool name], Secondary: Array, Mover action: Primary->Secondary

No = Primary: Array, Secondary: none, Mover action: not used

Only = Primary: [Pool name], Secondary: none, Mover action: not used

Prefer = Primary: [Pool name], Secondary: Array, Mover action: Secondary->Primary

[alturismo]

13 hours ago, Reynald said:

• Add freeing threshold option (R3yn4ld)

 

may as note, treehold is resetted to 0% after update (at least here) ... so may moves are occuring after update and triggered mover

[Reynald]

1 hour ago, Peuuuur Noel said:

  

Thanks for your message.

 

I tested but it didn't do what expected.

I've no issue with Cache:Yes (as in 2023 version), all files are moved to array when mover runs. That's OK.

 

For Cache:Prefer, files are moved to array based on age, even if I set "Move files off cache based on age" to "No" in Mover Tuning settings (should prevent it, right?).

No, it won't prevent that because the file list is presented by the OS via the find command. Usually it's by inode number, which is an increasing number: https://serverfault.com/a/181815

 

Quote

 

My appdata, domains and system shares are set to Cache:Prefer and when mover runs, VM images, docker and old files are now moved to array which is annoying to have them on HDDs (array) and no longer on SSDs (cache).

I want them to always stay on cache, not to move to array. It changes from the original purpose of mover (Mover action: Array->Cache) and from the 2023 version (all files remain on cache, old files are not moved to array).

 

 

You'd better set these Primary: cache; Secondary None, with a very low minimum free space. This is my setting:

 

If you set correctly the Cache:Yes and Cache:Prefer shares (sorry for using old naming, it's quicker) with larger Minimum free space (like xxGB or even xxxGB) you will never have out of space froblem for Cache:Only shares.

 

Quote

 

Also, in settings, shouldn't Cache:Yes/No/Only/Prefer be updated to new names to avoid mistakes and for new unraid users (from 6.12) started after these changes?

Yes = Primary: [Pool name], Secondary: Array, Mover action: Primary->Secondary

No = Primary: Array, Secondary: none, Mover action: not used

Only = Primary: [Pool name], Secondary: none, Mover action: not used

Prefer = Primary: [Pool name], Secondary: Array, Mover action: Secondary->Primary

 

 Yes, good point (It's already in the next version, will redo a pass to check on the UI)

[Reynald]

20 minutes ago, alturismo said:

may as note, treehold is resetted to 0% after update (at least here) ... so may moves are occuring after update and triggered mover

Arghhh, sorry for that

It's a side effect of "Alturismo likes tennis" new freeing threshold option 😇

[Peuuuur Noel]

6 minutes ago, Reynald said:

No, it won't prevent that because the file list is presented by the OS via the find command. Usually it's by inode number, which is an increasing number: https://serverfault.com/a/181815

But if set to "No" it should use mover.old and not age_mover like in 2023 version?

 

 

6 minutes ago, Reynald said:

You'd better set these Primary: cache; Secondary None, with a very low minimum free space. This is my setting:

 

If you set correctly the Cache:Yes and Cache:Prefer shares (sorry for using old naming, it's quicker) with larger Minimum free space (like xxGB or even xxxGB) you will never have out of space froblem for Cache:Only shares.

That's what I've done before and ended up to issues after cache was full before mover run. Shares with Cache:Yes were writing on array once minimum free space was reached, then my appdata share (Cache:Only) kept writing on cache on existing files until there was no more space and apps crashed with some corrupted files.

 

Keep in mind that, following unraid manual, Cache:Prefer is the default behavior for appdata, domains and system shares. So if people are not aware of this, their files will be spread across cache and array.

 

Quote

Prefer: Write new files to the cache if the free space on the cache is above the Minimum free space setting for the share, and if the free space falls below that value then write the files to the main array instead.

When Mover runs it will attempt to move any files for this share that are on the main array back to the cache as long as the free space on the cache is above the Minimum free space setting for the cache

It is the default setting for the appdata and system shares that support the Docker and VM subsystems. Generally, you want the files/folders belonging to these shares to reside on the cache as it provides better performance for Docker containers and VMs if their files are not on the main array. This is due to the performance overhead of maintaining parity on the main array which slows down write operations.

 

[Reynald]

Just now, Peuuuur Noel said:

But if set to "No" it should use mover.old and not age_mover like in 2023 version?

No, mover.old is not used by mover tuning anymore. It caused issue with hardlinks, plus the "Synchronize cache files to array" cannot work with old mover.

 

Quote

That's what I've done before and ended up to issues after cache was full before mover run. Shares with Cache:Yes were writing on array once minimum free space was reached, then my appdata share (Cache:Only) kept writing on cache on existing files until there was no more space and apps crashed with some corrupted files.

Then limits are not set correctly (I myself had this situation quite a few times, it can happen on any system)

 

Quote

Keep in mind that, following unraid manual, Cache:Prefer is the default behavior for appdata, domains and system shares. So if people are not aware of this, their files will be spread across cache and array.

 

That's the normal behavior of Cache:Prefer with an out of space situation.

 

If you want Mover Tuning to ignore appdata, domains, and system, you can use this option:

 

And all in all, if the tradeoff brang by 2024.07+ versions are too much for you, you may stick with 2023 version. I proposed several options to reach your desired behaviour, I'm pretty sure you'll manage something that suits you

[PPH]

Hello @Reynald,

When attempting to run the latest version of the Mover Tuning plugin, I am receiving the following error message:

 

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 421: * 75 / 100: syntax error: operand expected (error token is "* 75 / 100")

 

unRAID then shows that "mover is running" but I have to then manually terminate by running the 'mover stop' command as it does not complete or time out.

 

Also, I changed Mover Tuning to use CTIME (instead of MTIME) but some recently downloaded items were also being picked up to be moved when I would have expected them to be left as there were files that were downloaded many weeks earlier that I would have expected to be moved first.

 

Diagnostics and "Mover Tuning" settings attached.

 

Thanks.

littlebox-diagnostics-20240804-1447.zip

[Reynald]

1 hour ago, Reynald said:

Arghhh, sorry for that

It's a side effect of "Alturismo likes tennis" new freeing threshold option 😇

Will be corrected in the future version, threshold will be converted to movingThreshold and freeingThreshold in configfile

[Reynald]

4 minutes ago, PPH said:

Hello @Reynald,

When attempting to run the latest version of the Mover Tuning plugin, I am receiving the following error message:

 

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 421: * 75 / 100: syntax error: operand expected (error token is "* 75 / 100")

 

 

Hello @PPH

Seems there is a problem with Freeing Threshold which is blank:

Aug  4 09:45:34 LittleBox move: Using global moving threshold: 75 %
Aug  4 09:45:34 LittleBox move: Using global freeing threshold:  %

Can you please change it in settings and retry?
(shall be handled by plugin installation script in next version)
 

Quote

Also, I changed Mover Tuning to use CTIME (instead of MTIME) but some recently downloaded items were also being picked up to be moved when I would have expected them to be left as there were files that were downloaded many weeks earlier that I would have expected to be moved first.

I will investigate this one. Can you post /tmp/ca.mover.tuning/Filtered_filelist and Moving_action please? In PM if filenames are sensitives

[PPH]

14 minutes ago, Reynald said:

 

Hello @PPH

Seems there is a problem with Freeing Threshold which is blank:

Aug  4 09:45:34 LittleBox move: Using global moving threshold: 75 %
Aug  4 09:45:34 LittleBox move: Using global freeing threshold:  %

Can you please change it in settings and retry?
(shall be handled by plugin installation script in next version)
 

I will investigate this one. Can you post /tmp/ca.mover.tuning/Filtered_filelist and Moving_action please? In PM if filenames are sensitives

Hi @Reynald,

Could you advise where I set the value for the "global freeing threshold" please?

Thanks.