Individual Mover settings for different cache pools


Recommended Posts

Maybe this would be a good time to reintroduce my idea for the cache and mover settings.

Instead of cache yes no prefer only, how about mover jobs can specify source and destination, and shares specify initial placement.

 

As an example, on the share settings, you would specify new files destination pool, and select which mover jobs to enable.

The mover jobs would have time to start running, source pool, destination pool, file age restrictions, file size restrictions.

 

So, a share could be configured to start on cache pool 1, and have a mover job that selects all files older than 5 days and smaller than 100MB move to pool 2 every week, and a mover job that moves all files in that share on pool 1 older than 30 days and larger than 100MB to the array every 3 weeks.

 

Mover status could show which jobs are currently active, with cancel buttons.

 

Yes, you could easily set up a loop where files get moved round robin and back again by accident (or on purpose). It would add a level of complexity that may be too much for some people, but with general settings that mimic the current yes no prefer only, I don't think it would be that bad, and would be WAY more intuitive than the current situation.

  • Like 1
Link to comment
1 hour ago, jonathanm said:

Maybe this would be a good time to reintroduce my idea for the cache and mover settings.

Instead of cache yes no prefer only, how about mover jobs can specify source and destination, and shares specify initial placement.

 

As an example, on the share settings, you would specify new files destination pool, and select which mover jobs to enable.

The mover jobs would have time to start running, source pool, destination pool, file age restrictions, file size restrictions.

 

So, a share could be configured to start on cache pool 1, and have a mover job that selects all files older than 5 days and smaller than 100MB move to pool 2 every week, and a mover job that moves all files in that share on pool 1 older than 30 days and larger than 100MB to the array every 3 weeks.

 

Mover status could show which jobs are currently active, with cancel buttons.

 

Yes, you could easily set up a loop where files get moved round robin and back again by accident (or on purpose). It would add a level of complexity that may be too much for some people, but with general settings that mimic the current yes no prefer only, I don't think it would be that bad, and would be WAY more intuitive than the current situation.


Took 3 reads but I got it. I like your idea because I could then use this idea to implement something I've been wanting to do as linked above. Question, how is the age restriction suggested specified? Is it age since it was copied to unraid, last moved, or...? I would also like to add the option to "copy/overwrite" (with last modified) instead of move just in case the files being moved need to be duplicated for security purposes. "Pool 1 is raid 0, pool 2 is raid 1 / or disk array". 

You're correct, this can be rather confusing to some users as you mentioned. Maybe the current normal mover should stay how it is, and this could added as "advance mover settings".

PS: We're totally not hijacking your thread Ed 😆

Edited by XiuzSu
typo, more info
Link to comment
16 hours ago, XiuzSu said:

Question, how is the age restriction suggested specified? Is it age since it was copied to unraid, last moved, or...? I would also like to add the option to "copy/overwrite" (with last modified) instead of move just in case the files being moved need to be duplicated for security purposes.

Age would typically be defined by one of the available dates in the file's meta data. https://www.howtogeek.com/517098/linux-file-timestamps-explained-atime-mtime-and-ctime/ Which one to use, well that's a good question.

 

As far as copying instead of moving, that's not a function that should be in mover, at least in my opinion. Here's why. A user share is the combination of all the root folders of that name in all the various pools and the array. If you have identically named files in the same path on different pools, you end up with an issue when you present that file to be viewed or modified in the user share. How do you decide which one to show, and what do you do with the duplicates?

 

In my opinion, file backup should be a totally separate function, not directly linked to user shares or mover, for the reason above.

Link to comment
On 6/26/2020 at 11:37 PM, SpaceInvaderOne said:

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

I second this! We are running regular large backups, which I would like to run via a cache to make full use of 10gbe), but these would need to be moved off that cache quickly due to size restrictions. But I don't want to run the mover for the main cache pool too often during peak times. Individual schedules for different cache pools would be ideal.

There are countless use cases for this. I hope this can be introduced before 6.9 stable goes live.

Link to comment
  • 2 years later...
  • 4 months later...
3 hours ago, Sickness said:

Is this something that is being worked on? 

How have other people solved mover for 2 separate cache pools?

As far as I know this works fine.   Each User Share can specify which pool is used for caching purposes.

 

what is not supported at the moment is moving files directly between pools or one pool acting as a cache for another pool.    I think both of these are on the roadmap although no idea of the ETA.

  • Like 1
Link to comment
  • 1 month later...

Can someone explain to me what happens when you have more than two cache assigned drives (one SSD named cache and one nvme  drive named cache_nvme) and then run mover? Will it move both 'cache' drives contenst to the array sequentially? Or, has the nvme drive contents have to be manually moved? Apologies for a simplistic question, no doubt yet again, but i'm way behind with Unraid these days. having just added a nvme cache drive i now need to move 'stuff', just how exactly? Thanks!

Edited by superloopy1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.