Individual Mover settings for different cache pools

[SpaceInvaderOne]

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

[JonathanM]

13 minutes ago, SpaceInvaderOne said:

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

And allow moves between cache pools.

[JustOverride]

18 hours ago, jonathanm said:

And allow moves between cache pools.

And for a better notifier when the mover is working.

also.. cough 😁

[JonathanM]

Maybe this would be a good time to reintroduce my idea for the cache and mover settings.

Instead of cache yes no prefer only, how about mover jobs can specify source and destination, and shares specify initial placement.

 

As an example, on the share settings, you would specify new files destination pool, and select which mover jobs to enable.

The mover jobs would have time to start running, source pool, destination pool, file age restrictions, file size restrictions.

 

So, a share could be configured to start on cache pool 1, and have a mover job that selects all files older than 5 days and smaller than 100MB move to pool 2 every week, and a mover job that moves all files in that share on pool 1 older than 30 days and larger than 100MB to the array every 3 weeks.

 

Mover status could show which jobs are currently active, with cancel buttons.

 

Yes, you could easily set up a loop where files get moved round robin and back again by accident (or on purpose). It would add a level of complexity that may be too much for some people, but with general settings that mimic the current yes no prefer only, I don't think it would be that bad, and would be WAY more intuitive than the current situation.

[JustOverride]

1 hour ago, jonathanm said:

Maybe this would be a good time to reintroduce my idea for the cache and mover settings.

Instead of cache yes no prefer only, how about mover jobs can specify source and destination, and shares specify initial placement.

 

As an example, on the share settings, you would specify new files destination pool, and select which mover jobs to enable.

The mover jobs would have time to start running, source pool, destination pool, file age restrictions, file size restrictions.

 

So, a share could be configured to start on cache pool 1, and have a mover job that selects all files older than 5 days and smaller than 100MB move to pool 2 every week, and a mover job that moves all files in that share on pool 1 older than 30 days and larger than 100MB to the array every 3 weeks.

 

Mover status could show which jobs are currently active, with cancel buttons.

 

Yes, you could easily set up a loop where files get moved round robin and back again by accident (or on purpose). It would add a level of complexity that may be too much for some people, but with general settings that mimic the current yes no prefer only, I don't think it would be that bad, and would be WAY more intuitive than the current situation.

Took 3 reads but I got it. I like your idea because I could then use this idea to implement something I've been wanting to do as linked above. Question, how is the age restriction suggested specified? Is it age since it was copied to unraid, last moved, or...? I would also like to add the option to "copy/overwrite" (with last modified) instead of move just in case the files being moved need to be duplicated for security purposes. "Pool 1 is raid 0, pool 2 is raid 1 / or disk array". 

You're correct, this can be rather confusing to some users as you mentioned. Maybe the current normal mover should stay how it is, and this could added as "advance mover settings".

PS: We're totally not hijacking your thread Ed 😆

Edited June 27, 2020 by XiuzSu
typo, more info

[JonathanM]

16 hours ago, XiuzSu said:

Question, how is the age restriction suggested specified? Is it age since it was copied to unraid, last moved, or...? I would also like to add the option to "copy/overwrite" (with last modified) instead of move just in case the files being moved need to be duplicated for security purposes.

Age would typically be defined by one of the available dates in the file's meta data. https://www.howtogeek.com/517098/linux-file-timestamps-explained-atime-mtime-and-ctime/ Which one to use, well that's a good question.

 

As far as copying instead of moving, that's not a function that should be in mover, at least in my opinion. Here's why. A user share is the combination of all the root folders of that name in all the various pools and the array. If you have identically named files in the same path on different pools, you end up with an issue when you present that file to be viewed or modified in the user share. How do you decide which one to show, and what do you do with the duplicates?

 

In my opinion, file backup should be a totally separate function, not directly linked to user shares or mover, for the reason above.

[JustOverride]

Yea you're right. I guess I'll have to look for another way to have a "recently read/access cache" some other way. Either way, I think the mover should be updated with some of the functionalities mentioned above. This would open many doors for UNRAID.

[Netjet1980]

On 6/26/2020 at 11:37 PM, SpaceInvaderOne said:

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

I second this! We are running regular large backups, which I would like to run via a cache to make full use of 10gbe), but these would need to be moved off that cache quickly due to size restrictions. But I don't want to run the mover for the main cache pool too often during peak times. Individual schedules for different cache pools would be ideal.

There are countless use cases for this. I hope this can be introduced before 6.9 stable goes live.

[Tjlejeune]

I always thought a mover option based off file date would be nice. Like, move files that haven't change in a few days to the array

[Swarles]

+1 

Did anyone figure out a workaround for this? Having a different Mover for a specific share?

[dopeytree]

This would be the simplest way to upgrade mover!

 

That way you could just set the data download share to move every month or 90 days. and other other data to move every day or once a week.