Individual Mover settings for different cache pools

[SpaceInvaderOne]

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

[JonathanM]

13 minutes ago, SpaceInvaderOne said:

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

And allow moves between cache pools.

[JustOverride]

18 hours ago, jonathanm said:

And allow moves between cache pools.

And for a better notifier when the mover is working.

also.. cough 😁

[JonathanM]

Maybe this would be a good time to reintroduce my idea for the cache and mover settings.

Instead of cache yes no prefer only, how about mover jobs can specify source and destination, and shares specify initial placement.

 

As an example, on the share settings, you would specify new files destination pool, and select which mover jobs to enable.

The mover jobs would have time to start running, source pool, destination pool, file age restrictions, file size restrictions.

 

So, a share could be configured to start on cache pool 1, and have a mover job that selects all files older than 5 days and smaller than 100MB move to pool 2 every week, and a mover job that moves all files in that share on pool 1 older than 30 days and larger than 100MB to the array every 3 weeks.

 

Mover status could show which jobs are currently active, with cancel buttons.

 

Yes, you could easily set up a loop where files get moved round robin and back again by accident (or on purpose). It would add a level of complexity that may be too much for some people, but with general settings that mimic the current yes no prefer only, I don't think it would be that bad, and would be WAY more intuitive than the current situation.

[JustOverride]

1 hour ago, jonathanm said:

Maybe this would be a good time to reintroduce my idea for the cache and mover settings.

Instead of cache yes no prefer only, how about mover jobs can specify source and destination, and shares specify initial placement.

 

As an example, on the share settings, you would specify new files destination pool, and select which mover jobs to enable.

The mover jobs would have time to start running, source pool, destination pool, file age restrictions, file size restrictions.

 

So, a share could be configured to start on cache pool 1, and have a mover job that selects all files older than 5 days and smaller than 100MB move to pool 2 every week, and a mover job that moves all files in that share on pool 1 older than 30 days and larger than 100MB to the array every 3 weeks.

 

Mover status could show which jobs are currently active, with cancel buttons.

 

Yes, you could easily set up a loop where files get moved round robin and back again by accident (or on purpose). It would add a level of complexity that may be too much for some people, but with general settings that mimic the current yes no prefer only, I don't think it would be that bad, and would be WAY more intuitive than the current situation.

Took 3 reads but I got it. I like your idea because I could then use this idea to implement something I've been wanting to do as linked above. Question, how is the age restriction suggested specified? Is it age since it was copied to unraid, last moved, or...? I would also like to add the option to "copy/overwrite" (with last modified) instead of move just in case the files being moved need to be duplicated for security purposes. "Pool 1 is raid 0, pool 2 is raid 1 / or disk array". 

You're correct, this can be rather confusing to some users as you mentioned. Maybe the current normal mover should stay how it is, and this could added as "advance mover settings".

PS: We're totally not hijacking your thread Ed 😆

Edited June 27, 2020 by XiuzSu
typo, more info

[JonathanM]

16 hours ago, XiuzSu said:

Question, how is the age restriction suggested specified? Is it age since it was copied to unraid, last moved, or...? I would also like to add the option to "copy/overwrite" (with last modified) instead of move just in case the files being moved need to be duplicated for security purposes.

Age would typically be defined by one of the available dates in the file's meta data. https://www.howtogeek.com/517098/linux-file-timestamps-explained-atime-mtime-and-ctime/ Which one to use, well that's a good question.

 

As far as copying instead of moving, that's not a function that should be in mover, at least in my opinion. Here's why. A user share is the combination of all the root folders of that name in all the various pools and the array. If you have identically named files in the same path on different pools, you end up with an issue when you present that file to be viewed or modified in the user share. How do you decide which one to show, and what do you do with the duplicates?

 

In my opinion, file backup should be a totally separate function, not directly linked to user shares or mover, for the reason above.

[JustOverride]

Yea you're right. I guess I'll have to look for another way to have a "recently read/access cache" some other way. Either way, I think the mover should be updated with some of the functionalities mentioned above. This would open many doors for UNRAID.

[Netjet1980]

On 6/26/2020 at 11:37 PM, SpaceInvaderOne said:

Hi,

I think it would be really great to be able to set individual mover schedules that are specific to each cache pool.

As i would find it useful to have some pools move files more often than others.

 

I second this! We are running regular large backups, which I would like to run via a cache to make full use of 10gbe), but these would need to be moved off that cache quickly due to size restrictions. But I don't want to run the mover for the main cache pool too often during peak times. Individual schedules for different cache pools would be ideal.

There are countless use cases for this. I hope this can be introduced before 6.9 stable goes live.

[Tjlejeune]

I always thought a mover option based off file date would be nice. Like, move files that haven't change in a few days to the array

[Swarles]

+1 

Did anyone figure out a workaround for this? Having a different Mover for a specific share?

[dopeytree]

This would be the simplest way to upgrade mover!

 

That way you could just set the data download share to move every month or 90 days. and other other data to move every day or once a week.

[Sickness]

Is this something that is being worked on? 

How have other people solved mover for 2 separate cache pools?

[itimpi]

3 hours ago, Sickness said:

Is this something that is being worked on? 

How have other people solved mover for 2 separate cache pools?

As far as I know this works fine.   Each User Share can specify which pool is used for caching purposes.

 

what is not supported at the moment is moving files directly between pools or one pool acting as a cache for another pool.    I think both of these are on the roadmap although no idea of the ETA.

[superloopy1]

Can someone explain to me what happens when you have more than two cache assigned drives (one SSD named cache and one nvme  drive named cache_nvme) and then run mover? Will it move both 'cache' drives contenst to the array sequentially? Or, has the nvme drive contents have to be manually moved? Apologies for a simplistic question, no doubt yet again, but i'm way behind with Unraid these days. having just added a nvme cache drive i now need to move 'stuff', just how exactly? Thanks!

Edited December 10, 2023 by superloopy1

[itimpi]

It will move both.   It will work through the shares, checking which pool (if any) is associated for caching purpose for each User Dhare and if needed move the files to/from the main array according to the mover direction set.

 

[superloopy1]

2 hours ago, itimpi said:

It will move both.   It will work through the shares, checking which pool (if any) is associated for caching purpose for each User Dhare and if needed move the files to/from the main array according to the mover direction set.

 

 

Thankyou for a speedy reply!