This will work same as cache:yes to move data from cache to array

@alturismo

If you can check our discussion will be cool. 

https://github.com/masterwishx/ca.mover.tuning/issues/25

So after discussing in github it seems cache:prefer should only move to cache from array. And not move back at all? 

15 hours ago, Masterwishx said:

If you can find this instruction will be cool becose I'm trying to understand how the previous maintainer coded this to work.. 

 

9 hours ago, Masterwishx said:

So after discussing in github it seems cache:prefer should only move to cache from array. And not move back at all? 

i readed also the git discussion

 

but yes, if i would implement this, then only array > cache operations, but also i would "sort" the order

 

1/ cache > array Shares, run the mover with filters (should free up N %)

2/ cache < array Shares, move files from array to cache if there are any ... and fill N % back

 

while i personally still see no usecase therefore  operation 1/ should always keep enough space on cache so 2/ should never be of interest, if my setup is such a edge case, i would look that i run 1/ more often so 2/ wont be happening ... or extend my cache a little before i play ping pong with those files ... but that is my personal opinion.

41 minutes ago, alturismo said:

but yes, if i would implement this, then only array > cache operations, but also i would "sort" the order

Yep. This by unraid mover design only move to cache from array. Then unraid write to array and when cache is free again mover should move to cache

 

So I think the idea of author of plugin was automatically free cache by moving old files to array but seems it was not working good at least by autoage. Also not sure all users will want to move old files back to array? 

 

Also this move to array option seems like cache:yes and confusing but can be easily implemented as it's already in code. 

1 hour ago, alturismo said:

and fill N % back

So move back to array? 

 

 

1 hour ago, alturismo said:

while i personally still see no usecase

Yep I also use cache only for appdata etc.. 

But seems users use cache prefer for some media... 

8 minutes ago, Masterwishx said:

So move back to array? 

 

no, only cache < array operations, no cache > array operation (i use direction ...< or >)

 

7 minutes ago, Masterwishx said:

But seems users use cache prefer for some media... 

then they shouldnt need this plugin if they want to keep media on cache ... via cache prefer ...  always personal opinion

may we shoul dconsider what the intentions are for cache < array Shares

 

usually where meant for Shares like

 

/appdata

/domains

/system

 

which most likely resides on cache drives as

 

- these are "system" Shares and most likely nobody wants them on HDD's ... noise, temps, spinups, ...

 

now, the feature helps now in "edge emergency" cases where cache gets filled up and cache only Shares may collaps as filled up while Media are downloaded or whatever and filling up the cache drive ...

 

this is likely the case when you dont want to run the mover to often during the day as you dont want spinups, keep moving partly download files and so on ... but, with mover tuning adn the treeholds you shouldnt run into this issue anymore as you can run it hourly, look for your cache > array Shares and only IF you reach N % treehold, you move files away only IF they are already N days old ... so this makes the upper described usecase usually obsolete ... nevermind now.

 

so may to nail it down or better explained what i understand from the comments now, when we talk about Media Shares in a Layout like cache < array and move files back from array to cache ... next demand probably will be

 

age sorted ... overall ... also on these Shares, cause otherwise its complete nonsense cause some day you will run full with Media on cache and there is no space left ...

 

so when we talk about this feature together with some tuning option, it would be then more like

 

1/ cache > array -- move files like we do all the time filtered by rules

2/ cache < array -- look for aged files and move them also to array WHEN

3/ cache < array -- IF there are newer files on the array which should come into the cache now

 

following SOME rules HOWEVER ... which is also NOT the standard mover behaviour for those Shares ... it will just keep files on cache as much as possible, and now i come back to the start, thats why it only shoul dbe intended for "system" Shares and hell not for Media stuff and so on  personal opinion ... as you probably never will satisfy this construction ...

 

now again back to my personal opinion, if Media is the challenge ... standard setup is the way to go

 

cache > array move direction

treehold, only move when reached like 80 %

treehold, move down to like 30 %

age based, only move files older like 30 days

 

sample now, a 2 tb cache drive, we have now like 1 tb (80 - 30 = 50% from 2 tb) playground where cached Media will stay on cache anyhow ... and only files older then 30 days will get moved away once we reach the 80 % ... so may mover will only take 600 GB off as all other files are newer, still we shoul dbe good to go now until we again reach the 80 % ...

 

and the down under 30 % (~ 600 GB) on the cache are still fine for /appdata /system /domains ... 

 

so when you ask me or if i would maintain this plugin, i wouldnt support cache < array in combo with mover tuning  as the follow up will be like described ... somehow keep newer files on cache < array (old wording cache prefer) Shares ... and then you have fun to sort that stuff out in another run through all files ... while you can have the same result already with standard cache > array setups with the already functional filter rules (treehold up, down and age based).

I noticed it was mentioned in this post that the feature "Synchronize Primary files to Secondary" should be handled with care. I'd like to use this feature, could someone provide more details on what this mean.

2 minutes ago, Gio_nul said:

could someone provide more details on what this mean.

if you have dublettes on your Shares ... you should be aware using /mnt/user/ operations ... copy, move, delete, ...

 

so you should know what you do when you (or apps or plugins or scripts) handle files now ... as this can destroy the files easily ...

 

so, you can have dublette files, but then you always need to use as sample

 

/mnt/cache/Media/whatever/file.mkv

/mnt/user0/Media/whatever/file.mkv

 

or even better instead user0 to be 100% safe

 

/mnt/cache/Media/whatever/file.mkv

/mnt/disk2/Media/whatever/file.mkv (depending where the file is really located on the array)

 

so a cp /mnt/user/Media/whatever/file.mkv ... or a mv or a rm or ... can be dangerous

20 minutes ago, alturismo said:

so when you ask me or if i would maintain this plugin

Thanks a lot again. Well it's really headache, in short I will leave only cache<array for cache prefer. 

 

I just buyed a bigger size ssd for appdata etc and made cache only. 

 

So next question if we need move threshold option for like move from array to cache until 95% or maybe fixed 95% or maybe:

fixed cache size - minimum space=move threshold? 

1 minute ago, Masterwishx said:

fixed cache size - minimum space=move threshold? 

that would be a way to go if you want to implement it, even knowing we play ping pong ...

 

free cache > array

fill cache < array

 

so mover will run more often, but its a user choice using cache < array (cache prefer) Shares, so yes ... perfect approach

4 minutes ago, alturismo said:

that would be a way to go if you want to implement it, even knowing we play ping pong ...

 

free cache > array

fill cache < array

 

so mover will run more often, but its a user choice using cache < array (cache prefer) Shares, so yes ... perfect approach

I meaned only for fill cache<array

Not moving back to array as we only want move to cache for cache prefer 

and i forgot to mention as this has been discussed too in git, new files ... why using cache prefer and filled up caches, makes no difference ... the cache < array (cache prefer) is ONLY for mover operations and has nothing todo with new files ...

 

in any case, cache < array OR cache > array ... if cache is full, new files come to array anyway

 

 

cache < array is ONLY to move them backwards to cache IF there is free space when mover runs ...

Just now, Masterwishx said:

I meaned only for fill cache<array

Not moving back to array as we only want move to cache for cache prefer 

me too  may uncleared written

 

1st we free space on cache as mover runs regular (filtered) for cache > array Shares

2nd we fill up on cache again as mover runs "special" for cache < array Shares

 

so next run we are closer again on the treeholds ... and closer and closer ...  but users decision.

2 minutes ago, alturismo said:

so next run we are closer again on the treeholds ... and closer and closer ...  but users decision.

Ohh you mean on same cache drive. 

So you think we need to have threshold in plugin for cache prefer for how much % to fill cache or unraid will take care and we don't need it? 

12 minutes ago, Masterwishx said:

So you think we need to have threshold in plugin for cache prefer for how much % to fill cache or unraid will take care and we don't need it? 

i would let Unraid take care of it

3 hours ago, alturismo said:

would let Unraid take care of it

I agree 100%, but seems by plugin code design and becouse rsync, would be easy to implement fixed thresholds move threshold=0 fill up theshold=95

I also very much hope to add more options for moving data from array to cache, because I want to accelerate my video editing through cache. The materials needed for my video projects are very likely to have been shot a long time ago and have been stored in the array, but my recent video projects need to use those materials. I hope that when mover finds that there are earlier files in the array being accessed and the cache has free space, it can move those files from the array to the cache (or if the cache has no free space, it can move the files that have not been accessed for the longest time in the cache back to the array and make room for the recently called files). This plugin is already very good, but the existing functions seem unable to meet my needs.I'm not very good at English. The above sentence was written by me using a translation software. Please forgive me.

Edited March 9, 20251 yr by FelixLu

59 minutes ago, FelixLu said:

I hope that when mover finds that there are earlier files in the array being accessed and the cache has free space, it can move those files from the array to the cache (or if the cache has no free space, it can move the files that have not been accessed for the longest time in the cache back to the array and make room for the recently called files). This plugin is already very good, but the existing functions seem unable to meet my needs.I'm not very good at English. The above sentence was written by me using a translation software. Please forgive me.

what you looking for is a read cache, and (sadly) no, that is and wont be the intention of the mover, also does Unraid not trigger the access time by default ... just as note.

 

there is a sep plugin for Media caching (cache mover), but has nothing todo with editing photos ...

you can try it and use general mode only if your (not mentioned app) is perm accessing the file, it ll move it to cache then, yes, if not, then not ... there is a special option for Mediaservers, but not for other apps

3 hours ago, FelixLu said:

that have not been accessed for the longest time

 

i added atime option for plugin but seems unraid dont change atime for accessed files , also as posted above.

 

2 hours ago, alturismo said:

it can move the files that have not been accessed for the longest time in the cache back to the array

 

as we discussed before array->cache should only move files to cache from array.  at least for now maybe will change it later...

 

20 minutes ago, Masterwishx said:

 

i added atime option for plugin but seems unraid dont change atime for accessed files , also as posted above.

 

 

as we discussed before array->cache should only move files to cache from array.  at least for now maybe will change it later...

 

thanks and looking forward to the arrival of change.😂

I follow this thread and it is BY FAR the most active thread that bombs my email.

(I probably need to change the email settings to not send my multiple mail for this thread if possible)

...thing is I have to post in this thread too.

 

I am not sure my mover works properly. I see weird things happening and I suspect here is where to find solution as I AM using Mover Tuning. Attached I have all my settings.

 

Please read on to see my various issues...

 

1) I recently added a second pool (named "core"), where I moved my appdata/domains/system folders. I set those three folders, to be primary "core", secondary "cache", and move from cache to core. I started Mover. (note that also in my cache I have other data that need to move to my data disks) It indeed started moving appdata... My appdata is about 10Gbytes. It took HOURS and never fully finished. We are talking about M.2 PCIe4 to M.2 PCIe4. Then just moved appdata which also left residue (and shouldn't as docker service was stopped) and never moved the other two folders.
I since Mover apparently stopped (?) - also never moved any of the data files to my array - I just manually moved the little appdata remaining (like 2Mbytes?) and the other two folders by using UNRAID's own file manager. It took 3 minutes (and my VMs are around 200GB). So no idea why mover took ALL this time for a single folder AND stopped.

2) Then I set the three folders to be "core" only (no mover), but then decided to make appdata primary "core", secondary "cache" and move direction to be "cache to core", because 2-3 containers had to be reset to not use /mnt/cache/appdata and instead use /mnt/appdata. So I put it like that as a safeguard. Well tough luck.
I started Mover again, honestly to start moving my data from cache to my data array.
AND I REALIZED it started moving files from /core/appdata back to /cache/appdata!!! WHY!?

At least I was "lucky" that again it took its time (waaaaaaaaaaay slower than UNRAID's own file manager). So I issued a "mover stop" and manually put little appdata it moved back to "core", then set appdata to use only primary storage "core". So HOPEFULLY Mover won't touch it.

3) But the Odyssey continues. I started Mover a third time. It seemed to run a for a while. Only, no data (or no noticeable amount) every moved. They are all there... 450Gbytes of data that need to move to my array. Still there.

 

I am now at a loss. I will manually move everything to my array, ok, but I don't want to remove Mover Tuning (if it's the problem here)...

(please tag me if you reply, the thread is very long)

 

Edited March 9, 20251 yr by NLS

19 minutes ago, NLS said:

to be primary "core", secondary "cache", and move from cache to core

If you mean array->cache please read above posts, in lasted version was temporary fix, in next version will be move to cache from array only. 

 

You can check logs and debug for errors. 

 

If you have some speed issues, disable file validation check its checking filenames against attack so if you have long filenames and many files maybe can discrease speed.