2 hours ago, elmlund said:

ve them wipe after a few days.

They already auto removed 5days for logs 10 for list.

But in next version you can change the age for them. 

 

Note :  When enable/disable logging . Then ONLY syslog and /Mover_tuning_$NOW.log" will be disbled , also will be fixed in new version

Can't disable list files

Hey Everyone, I'm hoping you can help me out.

 

My RAM keeps filling up and I've identified that it's the mover log and run files causing the issue. They are not being cleaned up or removed and therefore keep filling up the RAM, particularly the .list files. I've tried rebooting and reinstalling the plugin, but no luck. I've manually deleted older files to stop the RAM completely filling up and crashing the system, which has happened once already.

 

 

For additional context if it helps, I recently had some corruption issues on my cache pool due to one of the two ssd drives failing. I've removed the failing drive from the pool and left it as a single drive for the time being. I don't think this would have anything to do with this issue but wanted to mention it.

 

I have also recently upgraded to v7 and 7.0.1.

vault-diagnostics-20250329-1636.zip

Edited March 29, 20251 yr by SubbyDew

1 hour ago, SubbyDew said:

My RAM keeps filling up and I've identified that it's

Please read the above posts 

3 minutes ago, Masterwishx said:

Please read the above posts 

Apologies if I didn't fully understand the previous posts.

 

18 hours ago, Masterwishx said:

They already auto removed 5days for logs 10 for list.

But in next version you can change the age for them. 

I see you mentioned the above. I'm a little confused as I've never had this issue before, is this a bug or did something change with how the plugin functions?

49 minutes ago, SubbyDew said:

see you mentioned the above. I'm a little confused as I've never had this issue before, is this a bug or did something change with how the plugin functions

No it's like always was befor, not sure why you have so big lists. 

 

But in next version I will add options to change thouse setting 

I will try to push new version today to fix this issue

15 minutes ago, Masterwishx said:

not sure why you have so big lists.

I've had a look into the files and they seem to have every single file from my shares that use the cache (both yes or prefer) listed in them, primarily being from the 'system' and 'appdata' shares.

 

There is just over 490,000 files listed in those .list files, with the majority being Plex metadata files (very large media collection).

 

00:56:02.986 Deciding the action (move/sync/keep) for each file. There are 490165 files, it can take a while...
01:06:00.142 No new files will be moved/synced from primary to secondary
01:06:00.143 No new files will be moved/synced from secondary to primary
01:06:00.146 Cleaning lock and stop files

 

Is this the expected behavior where it will list all the files from shares that have move rules setup, even if those files are already on the correct drive?

 

I'm just curious as it only seems to be a very recent issue for me, even though I've had a large Plex collection for some time.

21 minutes ago, SubbyDew said:

Is this the expected behavior where it will list all the files from shares that have move rules setup, even if those files are already on the correct drive?

Most of them have keep on primary? 

22 minutes ago, SubbyDew said:

I'm just curious as it only seems to be a very recent issue for me, even though I've had a large Plex collection for some time.

Do you have cache:prefer? 

24 minutes ago, SubbyDew said:

Is this the expected behavior where it will list all the files from shares that have move rules setup, even if those files are already on the correct drive?

If you think this is issue you can open issue in github

38 minutes ago, Masterwishx said:

Most of them have keep on primary? 

Yes, only my data share moves from primary (cache) to secondary (array).

All other shares are keep on primary.

 

30 minutes ago, Masterwishx said:

Do you have cache:prefer? 

Yes, the shares filling up the .list files are cache:prefer. 

See appdata section of log below. Same is for system share as well.

20:28:56.643 Primary storage: cache - size: 932GiB - used:  16 % (141GiB)
20:28:56.644 Secondary storage: user0
20:28:56.651 Share Information: Name: appdata - Path: /mnt/*/appdata
20:28:56.656 Moving threshold: % (0B) ; Filling up threshold: 70% (653GiB)
20:28:56.657 Mover action: user0->cache (cache:prefer). Pool is below priming threshold percentage:  16% < 70%.
20:28:56.658 => Will smart move newest files from user0 to cache until threshold.
20:31:02.177 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2025-03-29T202856.list for appdata

------------

20:31:03.388 Deciding the action (move/sync/keep) for each file. There are 465466 files, it can take a while...
20:40:39.006 No new files will be moved/synced from primary to secondary
20:40:39.007 No new files will be moved/synced from secondary to primary
20:40:39.011 Cleaning lock and stop files

 

1 hour ago, SubbyDew said:

Yes, only my data share moves from primary (cache) to secondary (array).

All other shares are keep on primary.

 

Yes, the shares filling up the .list files are cache:prefer. 

See appdata section of log below. Same is for system share as well.

20:28:56.643 Primary storage: cache - size: 932GiB - used:  16 % (141GiB)
20:28:56.644 Secondary storage: user0
20:28:56.651 Share Information: Name: appdata - Path: /mnt/*/appdata
20:28:56.656 Moving threshold: % (0B) ; Filling up threshold: 70% (653GiB)
20:28:56.657 Mover action: user0->cache (cache:prefer). Pool is below priming threshold percentage:  16% < 70%.
20:28:56.658 => Will smart move newest files from user0 to cache until threshold.
20:31:02.177 Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2025-03-29T202856.list for appdata

------------

20:31:03.388 Deciding the action (move/sync/keep) for each file. There are 465466 files, it can take a while...
20:40:39.006 No new files will be moved/synced from primary to secondary
20:40:39.007 No new files will be moved/synced from secondary to primary
20:40:39.011 Cleaning lock and stop files

 

Please check if you using last version also delete and make again if you have override for cache prefer

1 hour ago, Masterwishx said:

Please check if you using last version also delete and make again if you have override for cache prefer

I've already reinstalled the plugin once and I don't believe I've ever used any of the override options.

 

For now, I've just changed my appdata and system shares to cache:yes so that mover will ignore them.

 

Thanks for the help @Masterwishx. I'll also keep an eye on the github issue #41 that seems to be the same thing.

Long-time Unraid user here, but only started to use "mover" and this plugin.

 

It seems to work well, but some empty folders remain on the cache disk. Not sure what's the pattern. I have "delete parent folder" enabled. Maybe it is the parent of the parent that still stays? Shouldn't all folders be deleted / moved?

As of the last few updates mover has become extremely slow moving folders with thousands of files, with logging enabled it spams the following after every single file:

Mar 29 19:40:11 Tower move: Moving "/mnt/cache/./<share>/<filepath>"  to  /mnt/user0/ 
Mar 29 19:40:11 Tower move: Not deleting dir containing 16148 files: /mnt/cache/<share>/<directorypath>
Mar 29 19:40:11 Tower move: 121183 files remaining from caches to array  276GiB

and very clearly takes about a second for each file despite most of them being very small.

 

I suspect querying the file count in the directory and remaining size among other many redundant checks is slowing the entire process down, there's no point querying how many files remain in the directory after every operation, I'm guessing that's done to transfer "most recent" files first but you could instead build a queue with a 100 files or so to speed things up, and if instead the query is done to delete the folder after it's done transferring all the files then either query the file count once and store that as a counter, decrement every time you transfer files within it and finally query again when the counter reaches 0 (so you wouldn't remove a folder that ends up with more files inside it during the mover operation) or alternatively don't perform folder removing operations until mover is done, in which case you'd scan all folders for whether they're empty and remove the most nested folders first (so then the parents become empty if they contain no files and no other subfolders).

27 minutes ago, Dor said:

As of the last few updates mover has become extremely slow moving folders with thousands of files

The delete of folder was not changed if you check update history you will find new option = validate files, try disable it

2 hours ago, steve1977 said:

Shouldn't all folders be deleted / moved?

Is they folders or folders on zfs instead datasets? 

1 hour ago, Dor said:

but you could instead build a queue with a 100 files or so to speed things up, and if instead

Any prs are welcome 😉

3 hours ago, Dor said:

but you could instead build a queue with a 100 files or so to speed things up,

Please open issue for it

1 hour ago, Masterwishx said:

Please open issue for it

Done, #44.

5 hours ago, Masterwishx said:

Is they folders or folders on zfs instead datasets? 

I don't use ZFS. What are datasets?

thanks for picking up this project, mover tuning like this should really be baked into unraid imo.

 

anyway, random thought. Would it be beneficial to have an emergency auto-move trigger? You could set a cache threshold at, say, 95% that automatically triggers mover despite the set schedule.

4 hours ago, acosmichippo said:

Would it be beneficial to have an emergency auto-move trigger? You could set a cache threshold at, say, 95% that automatically triggers mover despite the set schedule.

there is no "live watchdog" keeping track, so the scheduler is the trigger todo something ...

 

so may just run mover often enough considering your worst case scene ...

 

sample, 2tb cache, run hourly, max treehold 90 %, so you have a 200 GB buffer for 1 hour ... means ~ 55 MB/s perm file transfer to cache .... if you are loading more and more perm, lets say 100 MB/s .... lower the max treehold

11 hours ago, steve1977 said:

don't use ZFS. What are datasets?

 zfs-related, yes folders should be deleted will check it