Jump to content
SpaceInvaderOne

Individual Mover settings for different cache pools

9 posts in this topic Last Reply

Recommended Posts

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

Share this post


Link to post
13 minutes ago, SpaceInvaderOne said:

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

And allow moves between cache pools.

Share this post


Link to post
18 hours ago, jonathanm said:

And allow moves between cache pools.

And for a better notifier when the mover is working.

also.. cough 😁

Share this post


Link to post

Maybe this would be a good time to reintroduce my idea for the cache and mover settings.

Instead of cache yes no prefer only, how about mover jobs can specify source and destination, and shares specify initial placement.

 

As an example, on the share settings, you would specify new files destination pool, and select which mover jobs to enable.

The mover jobs would have time to start running, source pool, destination pool, file age restrictions, file size restrictions.

 

So, a share could be configured to start on cache pool 1, and have a mover job that selects all files older than 5 days and smaller than 100MB move to pool 2 every week, and a mover job that moves all files in that share on pool 1 older than 30 days and larger than 100MB to the array every 3 weeks.

 

Mover status could show which jobs are currently active, with cancel buttons.

 

Yes, you could easily set up a loop where files get moved round robin and back again by accident (or on purpose). It would add a level of complexity that may be too much for some people, but with general settings that mimic the current yes no prefer only, I don't think it would be that bad, and would be WAY more intuitive than the current situation.

Share this post


Link to post
Posted (edited)
1 hour ago, jonathanm said:

Maybe this would be a good time to reintroduce my idea for the cache and mover settings.

Instead of cache yes no prefer only, how about mover jobs can specify source and destination, and shares specify initial placement.

 

As an example, on the share settings, you would specify new files destination pool, and select which mover jobs to enable.

The mover jobs would have time to start running, source pool, destination pool, file age restrictions, file size restrictions.

 

So, a share could be configured to start on cache pool 1, and have a mover job that selects all files older than 5 days and smaller than 100MB move to pool 2 every week, and a mover job that moves all files in that share on pool 1 older than 30 days and larger than 100MB to the array every 3 weeks.

 

Mover status could show which jobs are currently active, with cancel buttons.

 

Yes, you could easily set up a loop where files get moved round robin and back again by accident (or on purpose). It would add a level of complexity that may be too much for some people, but with general settings that mimic the current yes no prefer only, I don't think it would be that bad, and would be WAY more intuitive than the current situation.


Took 3 reads but I got it. I like your idea because I could then use this idea to implement something I've been wanting to do as linked above. Question, how is the age restriction suggested specified? Is it age since it was copied to unraid, last moved, or...? I would also like to add the option to "copy/overwrite" (with last modified) instead of move just in case the files being moved need to be duplicated for security purposes. "Pool 1 is raid 0, pool 2 is raid 1 / or disk array". 

You're correct, this can be rather confusing to some users as you mentioned. Maybe the current normal mover should stay how it is, and this could added as "advance mover settings".

PS: We're totally not hijacking your thread Ed 😆

Edited by XiuzSu
typo, more info

Share this post


Link to post
16 hours ago, XiuzSu said:

Question, how is the age restriction suggested specified? Is it age since it was copied to unraid, last moved, or...? I would also like to add the option to "copy/overwrite" (with last modified) instead of move just in case the files being moved need to be duplicated for security purposes.

Age would typically be defined by one of the available dates in the file's meta data. https://www.howtogeek.com/517098/linux-file-timestamps-explained-atime-mtime-and-ctime/ Which one to use, well that's a good question.

 

As far as copying instead of moving, that's not a function that should be in mover, at least in my opinion. Here's why. A user share is the combination of all the root folders of that name in all the various pools and the array. If you have identically named files in the same path on different pools, you end up with an issue when you present that file to be viewed or modified in the user share. How do you decide which one to show, and what do you do with the duplicates?

 

In my opinion, file backup should be a totally separate function, not directly linked to user shares or mover, for the reason above.

Share this post


Link to post

Yea you're right. I guess I'll have to look for another way to have a "recently read/access cache" some other way. Either way, I think the mover should be updated with some of the functionalities mentioned above. This would open many doors for UNRAID.

Share this post


Link to post
On 6/26/2020 at 11:37 PM, SpaceInvaderOne said:

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

I second this! We are running regular large backups, which I would like to run via a cache to make full use of 10gbe), but these would need to be moved off that cache quickly due to size restrictions. But I don't want to run the mover for the main cache pool too often during peak times. Individual schedules for different cache pools would be ideal.

There are countless use cases for this. I hope this can be introduced before 6.9 stable goes live.

Share this post


Link to post

I always thought a mover option based off file date would be nice. Like, move files that haven't change in a few days to the array

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.