[Plugin] Mover Tuning

[malaki86]

On 6/3/2024 at 2:00 PM, StylishQuoter said:

I may have run into a bug.

 

Running mover 2023.12.19 on unraid 6.12.10

 

```

Jun  3 03:00:02 root: mover: started
Jun  3 03:00:02 root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 372: ((: TOTALARRAYSIZE += : syntax error: operand expected (error token is "+= ")
Jun  3 03:00:02 root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 372: ((: TOTALARRAYSIZE += : syntax error: operand expected (error token is "+= ")
Jun  3 03:00:02 root: skip: /mnt/disk2/appdata/tautulli/tautulli.db-wal
Jun  3 03:00:02 root: skip: /mnt/disk2/appdata/tautulli/tautulli.db-shm
Jun  3 03:00:02 root: mover: finished

```

 

Thanks!

I'm getting that same error. Posted my logs a few minutes ago.

[hugenbdd]

16 hours ago, malaki86 said:

I'm getting that same error. Posted my logs a few minutes ago.

Are you on ZFS or just a traditional cache pool?

[Swarles]

On 6/16/2024 at 2:56 AM, hugenbdd said:

Are you on ZFS or just a traditional cache pool?

I think the "TOTALARRAYSIZE += : syntax error" is the same as the one identified back in Jan here. And the problem was identified here.

 

This commit highlights the change but I don't think I ever shared it with you. Hopefully that solves it. 

[hugenbdd]

Can both update this file and let me know the results?  I have updated it locally, but am not having the issue.

File Location: /usr/local/emhttp/plugins/ca.mover.tuning/

 

Permissions should be 755   (chmod 755 age_mover)  if they are not.  (rwxr-xr-x)

age_mover

[Mihle]

So I have my Nextcloud share on the cache. I want plenty of the files to be on the cache, but I dont want everything to be as its become 300 GB now.

Is there any way to specify what folders the mover have on cache, and what is on the Array, inside an share? I rather want, "Everything on cache, except for X" rather than "Everything on Array, except for X", but may be able to work with both.

 

If the share is set to Array --> Cache, how do the settings in this plugin work then?

I am guessing "Move files based on minimum size?" counts for the same direction an share is set to, so in this case move from Array to cache and not the other way?

 

Do "Move files off cache based on age?" ignore that Array --> Cache setting and move from cache to array no matter what?

Maybe I have to flip the Array --> Cache setting to get something closer to what I want.

[Swarles]

1 hour ago, Mihle said:

I rather want, "Everything on cache, except for X" rather than "Everything on Array, except for X", but may be able to work with both.

You can specify folders that you want to keep on the cache at all times. 

 

1 hour ago, Mihle said:

 the share is set to Array --> Cache, how do the settings in this plugin work then?

It does nothing and behaves like the normal mover. Tuning array -> cache is probably a use case for less than 1% of people. But I do acknowledge some may want it.

 

1 hour ago, Mihle said:

I am guessing "Move files based on minimum size?" counts for the same direction an share is set to, so in this case move from Array to cache and not the other way?

See above. Will only count on cache -> array moves.

1 hour ago, Mihle said:

Do "Move files off cache based on age?" ignore that Array --> Cache setting and move from cache to array no matter what?

It won't ever ignore the "Array -> cache" or "cache -> array" ahre settings. Think of mover tuning as a filter that uses the built in mover. It will apply it's filter on "cache -> array" moves but it won't on the other. 

 

1 hour ago, Mihle said:

Maybe I have to flip the Array --> Cache setting to get something closer to what I want.

If you want everything on the cache until it fills up you can set it to "Array -> cache" and it will move stuff to the cache. If you add more files and they can't fit, they will go onto the Array. If you delete files off the Cache and space becomes available, they will move from the array to the cache. I think thats a bad setup though. 

 

Otherwise set it to "Cache -> array" think of some filters you want to use to keep only certain files or folders on the cache. Stuff that you think you will access constantly you can make exclude that folder in the mover tuning settings. The rest for example can just be moved. Maybe you only want the mover to move when you reach 50% capacity in the cache.  Maybe you only want files that have been on the cache for 1 month already to be moved. Or both. 

[Mihle]

4 hours ago, Swarles said:

You can specify folders that you want to keep on the cache at all times. 

 

It does nothing and behaves like the normal mover. Tuning array -> cache is probably a use case for less than 1% of people. But I do acknowledge some may want it.

 

See above. Will only count on cache -> array moves.

It won't ever ignore the "Array -> cache" or "cache -> array" ahre settings. Think of mover tuning as a filter that uses the built in mover. It will apply it's filter on "cache -> array" moves but it won't on the other. 

 

If you want everything on the cache until it fills up you can set it to "Array -> cache" and it will move stuff to the cache. If you add more files and they can't fit, they will go onto the Array. If you delete files off the Cache and space becomes available, they will move from the array to the cache. I think thats a bad setup though. 

 

Otherwise set it to "Cache -> array" think of some filters you want to use to keep only certain files or folders on the cache. Stuff that you think you will access constantly you can make exclude that folder in the mover tuning settings. The rest for example can just be moved. Maybe you only want the mover to move when you reach 50% capacity in the cache.  Maybe you only want files that have been on the cache for 1 month already to be moved. Or both. 

Hm, ok, In my case it would have been easier in my next-cloud share to choose what ones should be moved, rather than what one shouldnt.

 

The only move files that is older than certain amount setting, I am guessing that one only counts from what time the file was written, not when its read from last time? Because if its only from when its written, its worthless in my scenario, as files may be read from without changed and still stay on the cache.

 

Is it possible to have the move stuff over X size for example from cache different on different shares? Because of one share I may want it to be 0, but another I may want it to be 200 mb?

 

If you enable larger than 200mb, and older than 60 days, is it AND or OR between them?

Edited June 19 by Mihle

[Swarles]

12 hours ago, Mihle said:

Hm, ok, In my case it would have been easier in my next-cloud share to choose what ones should be moved, rather than what one shouldnt.

You could write a script that runs before the mover to do this. Puts all the paths except the one you want to move into the skiplist file.

12 hours ago, Mihle said:

The only move files that is older than certain amount setting, I am guessing that one only counts from what time the file was written, not when its read from last time? Because if its only from when its written, its worthless in my scenario, as files may be read from without changed and still stay on the cache.

Unraid doesn't update access time so unless your next cloud does, then no.

12 hours ago, Mihle said:

Is it possible to have the move stuff over X size for example from cache different on different shares? Because of one share I may want it to be 0, but another I may want it to be 200 mb?

Yes you can set most settings on a per share basis.

12 hours ago, Mihle said:

If you enable larger than 200mb, and older than 60 days, is it AND or OR between them?

It's AND.

[Mihle]

15 hours ago, Swarles said:

You could write a script that runs before the mover to do this. Puts all the paths except the one you want to move into the skiplist file.

Hm....

15 hours ago, Swarles said:

Unraid doesn't update access time so unless your next cloud does, then no.

Then I would consider it not that useful, personally.

15 hours ago, Swarles said:

Yes you can set most settings on a per share basis.

Hm.. I didnt find it at first, but I think I found it now.

15 hours ago, Swarles said:

It's AND.

Hm.. OK
Thanks

[malaki86]

On 6/15/2024 at 12:56 PM, hugenbdd said:

Are you on ZFS or just a traditional cache pool?

ZFS Raid-0 4 drive pool

[Reynald]

Hi,

First, thanks and congrats to dev and contributors for this plugin !

I made a fork, and will request a pull if people are interested and if contributors review/comment/agree.

My commit add a feature I missed:

 

What it does under the hood is to sort by date and reduce to the necessary files only the custom file list sent to the mover

 

Repo is there:  R3yn4ld/ca.mover.tuning at automatic-age-threshold (github.com)

Commit is here: https://github.com/R3yn4ld/ca.mover.tuning/commit/560cf0791459eb1206d3cb091017849aa3e040b6

@hugenbdd I would be happy to contribute if you like. I'm skilled in bash but... do you still take pull requests?

Edited June 23 by Reynald
links and greetings

[Reynald]

On 6/17/2024 at 12:27 AM, Swarles said:

I think the "TOTALARRAYSIZE += : syntax error" is the same as the one identified back in Jan here. And the problem was identified here.

 

This commit highlights the change but I don't think I ever shared it with you. Hopefully that solves it. 

I haven't check the file proposed by hugenbdd, but I can see your commit is already in the code:

(please don't mind the changes, I've just checked in mine that was still opened in my browser)

[Reynald]

On 6/19/2024 at 4:49 PM, Mihle said:

The only move files that is older than certain amount setting, I am guessing that one only counts from what time the file was written, not when its read from last time? Because if its only from when its written, its worthless in my scenario, as files may be read from without changed and still stay on the cache.

In the mover tuning settings, ctime is creation time. Else modification time is took into account.

Also, until my fork/commit, custom file list which is sent to mover is not sorted by ctime or atime (which is in the released plugin only a filter) but as explained here on StackExchange:
 

Quote

find will be traversing the directory tree in the order items are stored within the directory entries. This will (mostly) be consistent from run to run, on the same machine and will essentially be "file/directory creation order" if there have been no deletes.

However, some file systems will re-order directory entries as part of compaction operations or when the size of the entry needs to be expanded, so there's always a small chance the "raw" order will change over time. If you want a consistent order, feed the output through an extra sorting stage.

this re-order is very likely to happen on ZFS or BTRFS.

Access time is worthless (but could be tested) as nextcloud scans your file library from time to time, so all files are accessed periodically. Same is true for media (plex) libraries

[Swarles]

3 hours ago, Reynald said:

I made a fork, and will request a pull if people are interested and if contributors review/comment/agree.

My commit add a featured I missed:

 

What it does under the hood is to sort by date and reduce to the necessary files only the custom file list sent to the mover

 

Repo is there:  https://github.com/R3yn4ld/ca.mover.tuning

Commit is here: https://github.com/R3yn4ld/ca.mover.tuning/commit/560cf0791459eb1206d3cb091017849aa3e040b6

Awesome! I haven't taken a look through the code but I'm sure this is a feature that many would find useful. Have you done much testing for it? I probably won't be able to take a look for another week or so.

 

3 hours ago, Reynald said:

I haven't check the file proposed by hugenbdd, but I can see your commit is already in the code:

(please don't mind the changes, I've just checked in mine that was still opened in my browser)

I'm not sure which version of age_mover you have taken this snippet from but the error is on line 370 showing in the latest release of the plugin.

 

3 hours ago, Reynald said:

Also, until my fork/commit, custom file list which is sent to mover is not sorted by ctime or atime (which is in the released plugin only a filter) but as explained here on 

Yes this is correct. We don't sort the files at all currently. When the user selects the ctime or mtime option this is purely for the sake of determining the "Age" of the file.

[Reynald]

1 hour ago, Swarles said:

Awesome! I haven't taken a look through the code but I'm sure this is a feature that many would find useful. Have you done much testing for it? I probably won't be able to take a look for another week or so.

 

It's running once an hour since a couple of hour, moving old files as torrents downloads...

 

However, I've only one cache pool (with several shares). Seems solid at the moment.

 

As all it does is deleting lines from $CUSTOM_MOVER_FILELIST in totalsizeFilelist() while looping for $TOTALCACHESIZE, the worse thing that could happen is that it does not move enough files because $CUSTOM_MOVER_FILELIST wouldn't be populated enough. (and lead to an Out Of Space situation for shares on Cache Only)

 

Before deleting lines, two fields are added in place of $FILECOUNT in customFilelist(), it's the ctime/mtime date and the usage quota of cache. Then $CUSTOM_MOVER_FILELIST is sorted according to date. At the end, date and quotas are replaced by line number of the file, in replacement of $FILECOUNT (which is finally the same)

 

Quote

I'm not sure which version of age_mover you have taken this snippet from but the error is on line 370 showing in the latest release of the plugin.

 

I've forked master branch from hugenbdd repo.

 

Quote

Yes this is correct. We don't sort the files at all currently. When the user selects the ctime or mtime option this is purely for the sake of determining the "Age" of the file.

Yes, mine does when "Auto" age is selected. It can be extended to other filters easily anyway. I tried to keep original behaviour as much as possible 

Edited June 23 by Reynald

[Swarles]

15 hours ago, Reynald said:

I've forked master branch from hugenbdd repo.

Yeah it's on line 370

[Reynald]

I don't see the same😲:

 

You see that:

 

But here I see that:

which is correct.

 

OK, got it...

 

The packages seem not to be made from the github but rather from hugenbdd system. 

 

The totalsizeArraylist has the bug in ca.mover.tuning-2023.12.19-x86_64-1 (latest plugin version) but not in ca.mover.tuning-2023.08.22-x86_64-1 (previous plugin version) 

 

@hugenbdd if you like, I can save you some time and take over the repo maintenance

 

I may as well create a "beta" repo and plugin so hugenbdd keep the official stable ones.

 

(P.S.: that is all kidness, not rebellion 😊😇)

Edited June 29 by Reynald

[rodolfosb]

Hi All, hoping someone could help me with this issue I am having.

 

I've just got a new SSD to add to my cache pool, going from 1 drive to 2 drives in btrfs raid 1.

My single cache drive was using xfs, so I had to move the files out of the cache in order to format it and use as a cache pool.

I adjusted all my shares as Primary Storage: Cache, Secondary Storage: Array, and mover action Cache -> Array.

I invoked the mover and managed to empty the drive, create a cache pool and format to btrfs raid 1.

Now that this is all done, I'm having issues moving the shares from the array back to the cache.

For example (see image below), my appdata share is configured as Primary Storage: Cache, Secondary Storage: Array, and mover action Array -> Cache. However, this selection disables the "mover tuning - share settings" section. Invoking the mover on the main page does not move anything at all.

I'm sure this is possible to do, but I can't quite figure it out. Any thoughts on how to resolve this?

 

Edit 1: Both Docker and VM services are disabled. Invoking the Mover from the Main page, only returns a single line in the log. See below.

 

 

 

Edited June 26 by rodolfosb
Additional information

[Swarles]

6 hours ago, rodolfosb said:

Now that this is all done, I'm having issues moving the shares from the array back to the cache.

For example (see image below), my appdata share is configured as Primary Storage: Cache, Secondary Storage: Array, and mover action Array -> Cache. However, this selection disables the "mover tuning - share settings" section. Invoking the mover on the main page does not move anything at all.

I'm sure this is possible to do, but I can't quite figure it out. Any thoughts on how to resolve this?

Mover tuning settings are "disabled" for any Array->Cache moves. Effectively it just uses the default mover without filter and should move everything. It's disabled on the share level for this same reason, customising the settings wouldn't do anything.

 

Regardless, it should still be moving all files associated with that share back on to the cache. The only thing I can suggest trying is disabling or removing mover tuning in the settings to see if it's the thing causing problems. Alternatively you could set the option "Move now button uses mover tuning settings:" to no and then invoke the mover on the main page.

 

Otherwise I'll need some logs to help see what's happening. See FAQ item #3.

[hugenbdd]

On 6/24/2024 at 3:42 AM, Reynald said:

I don't see the same😲:

 

You see that:

 

But here I see that:

which is correct.

 

OK, got it...

 

The packages seem not to be made from the github but rather from hugenbdd system. 

 

The totalsizeArraylist has the bug in ca.mover.tuning-2023.08.22-x86_64-1 (latest plugin version) but not in ca.mover.tuning-2023.08.22-x86_64-1 (previous plugin version) 

 

@hugenbdd if you like, I can save you some time and take over the repo maintenance

 

I may as well create a "beta" repo and plugin so hugenbdd keep the official stable ones.

 

(P.S.: that is all kidness, not rebellion 😊😇)

@Swarles and yourself are welcome to take over this plug-in. 

 

I have not been able to spend the time it needs on it recently.

[rodolfosb]

1 hour ago, Swarles said:

Mover tuning settings are "disabled" for any Array->Cache moves. Effectively it just uses the default mover without filter and should move everything. It's disabled on the share level for this same reason, customising the settings wouldn't do anything.

 

Regardless, it should still be moving all files associated with that share back on to the cache. The only thing I can suggest trying is disabling or removing mover tuning in the settings to see if it's the thing causing problems. Alternatively you could set the option "Move now button uses mover tuning settings:" to no and then invoke the mover on the main page.

 

Otherwise I'll need some logs to help see what's happening. See FAQ item #3.

Thanks for the suggestion! I removed the CA Mover Tuning plugin and the move worked like a charm (being invoked from the main screen). Go figure. I'll re-install once my shares are adjusted.

[aldumil]

Hello! I am having a problem with Mover Tuning plugin, I'm not sure if anyone can point me in the right direction because of how little information I have to share, but can't hurt right?

 

Short story, it doesn't seem to work. I enabled logs and set the Test Mode to yes, but Mover seems to have just worked as usual as my cache is empty now, and the /tmp/Mover folder doesn't have anything new in it, just these files, but they all look to be empty, though I'm not very experienced using vi so maybe I'm doing it wrong. Or maybe there is a better way to access the logs in that folder?

Cache_Tuning_2024-06-23T010001.list         Custom_Mover_Tuning_2024-06-23T010001.list
Custom_Cache_Tuning_2024-06-23T010001.list  Mover_Tuning_2024-06-23T010001.list

 Would appreciate any help! As it is now the cache will just fill up completely if I don't manually trigger it.

 

I'm including this in case it is helpful. Also, I had to add the filename because if I copy/paste from the FAQ post, it doesn't work. Maybe that is a problem also? I wanted to put this in a spoiler thing but could not figure that out.

sh-5.2# cat "/boot/config/plugins/ca.mover.tuning/ca.mover.tuning.cfg"
moverDisabled="no"
moverNice="0"
moverIO="-c 2 -n 0"
threshold="70"
age="no"
sizef="no"
sparsnessf="no"
filelistf="no"
filetypesf="no"
parity="yes"
enableTurbo="yes"
logging="yes"
force="no"
ignoreHidden="no"
beforeScript=""
afterScript=""
omovercfg="no"
movenow="no"
testmode="yes"

 

Edited June 27 by aldumil
added settings

[wgstarks]

8 minutes ago, aldumil said:

Would appreciate any help! As it is now the cache will just fill up completely if I don't manually trigger it.

You should probably include screenshots of your current settings along with a description of what you are trying to achieve in your next post.

[aldumil]

Just now, wgstarks said:

You should probably include screenshots of your current settings along with a description of what you are trying to achieve in your next post.

Sorry! I edited my post to add that while you were typing! I couldn't figure out the spoiler thing to do it fast enough.

[wgstarks]

Just now, aldumil said:

Sorry! I edited my post to add that while you were typing! I couldn't figure out the spoiler thing to do it fast enough.

I wasn’t clear, I meant the mover settings not mover tuning.