Improved Mover for better Cache function

[dazzathewiz]

It would be nice to see the mover script be more configurable from the UI. I'd also like to see the mover settings configurable per share also - basically have a 'default' mover configuration and for shares (which use cache) can overwrite the defaults.

 

These features would only apply to "protected" cache volumes (more than 1 cache disk) since the idea of the mover on a single disk cache volume is to get data off ASAP (although an over-ride for advanced users is probably nice!). These options would change when the data moves off the cache volume:

- Only move off cache files of a certain age (file created date). I think the mover uses rsync (?) so this could be implemented via rsync options.

- Similar to above, if possible to only move files which have not been modified/added to within a defined timeframe. Not sure if possible with timestamps/rsync without further investigation.

- Only move files when cache volume usage is at a defined level (% or GB free I guess)

 

On my own server I would modify the mover script to accommodate some of these functions, but it would be much better not to maintain through upgrades and be done properly from source.

 

Use cases

 

I guess a lot of my thinking is more around tiering of data...

 

PLEX

If you have share where users consume content within a typical period after the file is first put on the share (and are able to provide large enough cache disk volume) then files could sit on the cache for when the majority of users will access it without the need to spinning up/keeping spinning array drives. 

 

Office Shares

Again where content is created and worked on frequently for a few weeks and then once finished is not viewed with any regularity unless required, it would be good to keep on the cache

 

 

[SimonAG]

I would use a read cache by having new content stay on the cache until new content replaces it, by setting say a 70% max disk usage limit. Of course still have the nightly copy but only repoint the files to the non cache once they are removed from the cache. That would be easier to do than have existing files on the array move to the cache when they modified. As that would only work for document stores and smaller files. While if I watch a 50gb film I could finish watching it before its even finished copying to the cache, after which I might not need it on the cache.

Sent from my SHIELD Tablet using Tapatalk

[kizer]

You can easily write up some User.Scripts and put them on a Cron for nearly everything you are asking for. 

[dazzathewiz]

23 hours ago, Simontv said:

I would use a read cache by having new content stay on the cache until new content replaces it, by setting say a 70% max disk usage limit. Of course still have the nightly copy but only repoint the files to the non cache once they are removed from the cache. That would be easier to do than have existing files on the array move to the cache when they modified. As that would only work for document stores and smaller files. While if I watch a 50gb film I could finish watching it before its even finished copying to the cache, after which I might not need it on the cache.

Sent from my SHIELD Tablet using Tapatalk
 

So my thoughts didn't involve any moving of data from array to cache - I think that's a different thing altogether and not a feature currently existing in Unraid. Really what I'm talking about is a way of targeting files to delay their move off the cache to the array. IE: currently the mover moves all files on all shares that use cache indiscriminately - it would be cool to filter what gets moved at the time the mover is scheduled to run so that some stuff stays on the cache if it is new, but old stuff gets moved.

 

23 hours ago, kizer said:

You can easily write up some User.Scripts and put them on a Cron for nearly everything you are asking for. 

100%! I can write these scripts for my own use. I guess in my searching around forums I've seen a few ideas thrown around about what people think are good improvements to cache moves, but a lot of them are more complex under the surface. I was just trying to think of more simple changes to existing functionality that (it seems) might be useful to others. And generally I think it would be a better solution if implemented by the Unraid team than my own thinking off the top of my head, not to mention it could just be put in as options in the web GUI.

[kizer]

Yep. A few have asked for my complex or simpler manners using the Cache. @Squid wrote a plug-in recently so you can literally stop the Mover from working or suspend it during Parity checks. I’m on my phone but it’s new to unRAID. You could use something like that and create your own script with some RSYNC magic. 

 

 

[dazzathewiz]

Oh cool! Hadn't seen that yet!

[micdoesit]

This would be amazing! I am 100% incapable of creating a script for this though. lol For plex it would be nice to keep newer media in cache for transcode performance! If you did/will create a script for this I would greatly appreciate a share.