9 minutes ago, alturismo said:

actually that wasnt what i mentioned, but ... starting mover from terminal is simple as typing

 

mover

 thats it

oh ok lol thought that would just launch the native mover but thanks for letting me know

Also getting the empty folder issue, running Beta 3 and 4 and same issue. Files move under auto based on age, but the folder structure still stays there. 

Is mover tuning safe to run on 7.x yet?

Hello there,

Hope I won't disturb Your conversation here, but I have a problem with mover.

I did not start mover process but it seems to automatically start every hour for few seconds and absolutely nothing to move.

 

Can I ask You for help, what can be possibly wrong?
Thank You.

 

  

9 hours ago, brwoodford said:

Also getting the empty folder issue, running Beta 3 and 4 and same issue. Files move under auto based on age, but the folder structure still stays there. 

Have the same problem as You can see below. Don't know why.

 

  

Edited October 28, 20241 yr by rogales

6 hours ago, rogales said:

I did not start mover process but it seems to automatically start every hour for few seconds and absolutely nothing to move.

 

well, what do you expect when i may ask ? you telling mover to run hourly ...

 

 

and like YOU setted it it runs hourly and like you expected it there was nothing to move ... sample ...

 

 

the empty Folders seems to be still open, but overall just a "cosmetic" issue i would say for now, pretty sure @Reynald will look into it when he has some spare time for his plugin.

8 hours ago, garntom said:

Is mover tuning safe to run on 7.x yet?

may install, may set "dry run", may test it ... ? looking good here on v7 beta

5 hours ago, alturismo said:

well, what do you expect when i may ask ? you telling mover to run hourly ...

 

 

and like YOU setted it it runs hourly and like you expected it there was nothing to move ... sample ...

 

Yes I do understand that, but I was sure that it does not matter if i chose to start mover when disk utilization is 95% or more?

Edited October 29, 20241 yr by rogales

1 hour ago, rogales said:

Yes I do understand that, but I was sure that it does not matter if i chose to start mover when disk utilization is 95% or more?

effectiv it doesnt ... but mover needs to run to know the utilisation ... so it ll check every hour, if <95 ... do nothing, else, move

Ohh, now I do understand it completely.

Thank You for Your patience and clarification.

hello, I just saw in the community apps there's a new version but when I click update, the log says already last version. But my version is 2023.12.19. I'm on 6.2.11 unraid.

 

I also see 2 Mover Tuning in CA that point here, one is 2023.12.19 and the other is newer but even if I click update on the newer, it seems to only install the old one

 

edit: well when I open the plugin, it shows 2024.08.16 so I guess it's the reporting in CA and in the plugin page that is not right

Edited November 1, 20241 yr by Nodiaque

When I download this plugin, even when disabled, my mover stops working as expected and logs interesting messages that make sense but I can't seem to figure out why they're happening. if I delete the plugin and run mover everything works as expected. Any thoughts @Reynald? Thank you

Nov  1 16:41:57 G2 move: ----------------------------------------------------------- Processing paperless share ----------------------------------------------------------
Nov  1 16:41:57 G2 move: /mnt/cache does not exist. Is the pool still used? Consider removing /boot/config/shares/paperless.cfg if not.
Nov  1 16:41:57 G2 move: => Skipping
Nov  1 16:41:57 G2 move: ------------------------------------------------------------- Processing photos share -----------------------------------------------------------
Nov  1 16:41:58 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:58 G2 move: Secondary storage: none
Nov  1 16:41:58 G2 move: Share Information: Name: photos - Path: /mnt/user0/photos
Nov  1 16:41:58 G2 move: Mover action: no action, only user0 used (cache:no).
Nov  1 16:41:58 G2 move: => Skipping
Nov  1 16:41:58 G2 move: ------------------------------------------------------------ Processing postgis share -----------------------------------------------------------
Nov  1 16:41:58 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:58 G2 move: Secondary storage: none
Nov  1 16:41:58 G2 move: Share Information: Name: postgis - Path: /mnt/user0/postgis
Nov  1 16:41:58 G2 move: Mover action: no action, only user0 used (cache:no).
Nov  1 16:41:58 G2 move: => Skipping
Nov  1 16:41:58 G2 move: ------------------------------------------------------------ Processing scripts share -----------------------------------------------------------
Nov  1 16:41:58 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:58 G2 move: Secondary storage: none
Nov  1 16:41:58 G2 move: Share Information: Name: scripts - Path: /mnt/user0/scripts
Nov  1 16:41:58 G2 move: Mover action: no action, only user0 used (cache:no).
Nov  1 16:41:58 G2 move: => Skipping
Nov  1 16:41:58 G2 move: ------------------------------------------------------------ Processing scrypted share ----------------------------------------------------------
Nov  1 16:41:58 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:58 G2 move: Secondary storage: none
Nov  1 16:41:58 G2 move: Share Information: Name: scrypted - Path: /mnt/user0/scrypted
Nov  1 16:41:58 G2 move: Mover action: no action, only user0 used (cache:no).
Nov  1 16:41:58 G2 move: => Skipping
Nov  1 16:41:58 G2 move: ----------------------------------------------------------- Processing smokeping share ----------------------------------------------------------
Nov  1 16:41:58 G2 move: /mnt/cache does not exist. Is the pool still used? Consider removing /boot/config/shares/smokeping.cfg if not.
Nov  1 16:41:58 G2 move: => Skipping
Nov  1 16:41:58 G2 move: ------------------------------------------------------------- Processing system share -----------------------------------------------------------
Nov  1 16:41:58 G2 move: Primary storage: cache_nvme - size: 923GiB - used: 49 % (469GiB)
Nov  1 16:41:58 G2 move: Secondary storage: none
Nov  1 16:41:58 G2 move: Share Information: Name: system - Path: /mnt/cache_nvme/system
Nov  1 16:41:58 G2 move: Mover action: no action, only cache_nvme used (cache:only).
Nov  1 16:41:58 G2 move: => Nothing will be moved. Share usage is taken into account in the calculation of the threshold for other shares.
Nov  1 16:41:58 G2 move: cache_nvme/system used: 146GiB
Nov  1 16:41:58 G2 move: ------------------------------------------------------------ Processing tandoor share -----------------------------------------------------------
Nov  1 16:41:58 G2 move: /mnt/cache does not exist. Is the pool still used? Consider removing /boot/config/shares/tandoor.cfg if not.
Nov  1 16:41:58 G2 move: => Skipping
Nov  1 16:41:58 G2 move: -------------------------------------------------------------- Processing temp share ------------------------------------------------------------
Nov  1 16:41:58 G2 move: /mnt/cache does not exist. Is the pool still used? Consider removing /boot/config/shares/temp.cfg if not.
Nov  1 16:41:58 G2 move: => Skipping
Nov  1 16:41:58 G2 move: ------------------------------------------------------------- Processing tiles share ------------------------------------------------------------
Nov  1 16:41:59 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:59 G2 move: Secondary storage: user0
Nov  1 16:41:59 G2 move: /mnt/user0/tiles does not exist. Is the share still used? Consider removing /boot/config/shares/tiles.cfg if not.
Nov  1 16:41:59 G2 move: => Skipping
Nov  1 16:41:59 G2 move: ---------------------------------------------------------- Processing timemachine share ---------------------------------------------------------
Nov  1 16:41:59 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:59 G2 move: Secondary storage: none
Nov  1 16:41:59 G2 move: Share Information: Name: timemachine - Path: /mnt/user0/timemachine
Nov  1 16:41:59 G2 move: Mover action: no action, only user0 used (cache:no).
Nov  1 16:41:59 G2 move: => Skipping
Nov  1 16:41:59 G2 move: ---------------------------------------------------------- Processing timemachine2 share --------------------------------------------------------
Nov  1 16:41:59 G2 move: /mnt/cache does not exist. Is the pool still used? Consider removing /boot/config/shares/timemachine2.cfg if not.
Nov  1 16:41:59 G2 move: => Skipping
Nov  1 16:41:59 G2 move: ----------------------------------------------------------- Processing transcode share ----------------------------------------------------------
Nov  1 16:41:59 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:59 G2 move: Secondary storage: none
Nov  1 16:41:59 G2 move: /mnt/user0/transcode does not exist. Is the share still used? Consider removing /boot/config/shares/transcode.cfg if not.
Nov  1 16:41:59 G2 move: => Skipping
Nov  1 16:41:59 G2 move: ---------------------------------------------------------- Processing windows_disk share --------------------------------------------------------
Nov  1 16:41:59 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:59 G2 move: Secondary storage: none
Nov  1 16:41:59 G2 move: Share Information: Name: windows_disk - Path: /mnt/user0/windows_disk
Nov  1 16:41:59 G2 move: Mover action: no action, only user0 used (cache:no).
Nov  1 16:41:59 G2 move: => Skipping
Nov  1 16:41:59 G2 move: --------------------------------------------------------- Processing windows_disk2 share --------------------------------------------------------
Nov  1 16:41:59 G2 move: /mnt/cache does not exist. Is the pool still used? Consider removing /boot/config/shares/windows_disk2.cfg if not.
Nov  1 16:41:59 G2 move: => Skipping
Nov  1 16:41:59 G2 move: ---------------------------------------------------------- Processing zfs_backups share ---------------------------------------------------------
Nov  1 16:41:59 G2 move: Primary storage: user0 - size: 26TiB - used:  75 % (19TiB)
Nov  1 16:41:59 G2 move: Secondary storage: none
Nov  1 16:41:59 G2 move: Share Information: Name: zfs_backups - Path: /mnt/user0/zfs_backups
Nov  1 16:41:59 G2 move: Mover action: no action, only user0 used (cache:no).
Nov  1 16:41:59 G2 move: => Skipping
Nov  1 16:41:59 G2 move: ************************************************************ ANALYSING MOVING ACTIONS ***********************************************************
Nov  1 16:41:59 G2 move: Deciding the action (move/sync/keep) for each file. There are 4 files, it can take a while...
Nov  1 16:41:59 G2 move: No new files will be moved/synced from primary to secondary
Nov  1 16:41:59 G2 move: No new files will be moved/synced from secondary to primary
Nov  1 16:41:59 G2 move: Cleaning lock and stop files
Nov  1 16:42:00 G2 move: ****************************************************************** WE ARE DONE ! ****************************************************************
Nov  1 16:42:41 G2 root: plugin: running: anonymous
Nov  1 16:42:41 G2 root: plugin: ca.mover.tuning.plg removed
Nov  1 16:43:11 G2 emhttpd: shcmd (524): /usr/local/sbin/mover |& logger -t move &
Nov  1 16:43:11 G2 move: mover: started
Nov  1 16:43:11 G2 move: file: /mnt/cache_nvme/data/removed rest for privacy

 

9 hours ago, cambriancatalyst said:

Any thoughts @Reynald? Thank you

from this output, nope, may a screen from your Shares page and what the expected behaviour should be

 

 

 

I just yesterday noticed that there was a duplicate entry for this plugin in CA and my previous install was not updating (was stuck on 2023.12.19). I got the new version, and was very happy to see some of the new options (especially free down/prime up)!

 

However, I've noticed some issues:

1. It seems like the ignore list option is not working (files listed inside the ignore file are being moved even with this option enabled). It did work for me until I migrated to the new version of the plugin.
2. With the ignore list enabled, more data is moved than when this option is disabled. The difference seems to be roughly equal to the total size of the files in the ignore list.
3. The percent used calculation seems to differ from what is shown in the Unraid GUI. The GUI shows 88.7% full (1.71 TB/1.9TB), while the plugin shows 85% used (the logs say 1.6 TiB / 1.8 TiB). This is a mirrored ZFS cache pool of 2 devices. Not a huge deal if the difference is consistent, I just don't want to risk running out of space if the difference between the two free space calculations were to grow.

 

You can see my options in the screenshot below. Pretty simple, I just want to trigger the mover at 85% and move files until the Cache is down to 80% full, while ignoring certain files.

 

 

Edited November 6, 20241 yr by crafty35a

I'm just needing this plugin to exclude a few folders from the mover. Is there a way to set things so the mover will essentially always clear my cache except for a few folders that I always want to live on cache. I have my text file with exclusions set up but I can't seem to find a way to disable "Only move if above this threshold of used Primary (cache) space:". I'd like to set it at 0% and with the latest plugin there is no way to do this.

 

Edit: the solution was to configure everything per share individually. So I'm all good now.

Edited November 10, 20241 yr by mackid1993

Just to get some clarification, my goal is to leave the mover basically default but exclude a few folders so they always stay on cache. I set everything per share individually. My regular shares that start on cache and nightly move to the array have Only move if above this threshold of used Primary (cache) space: set to 0 percent. Then the ones like appdata, system, domains that should only write to the array if cache is full is set to 95 percent on each share. The only other thing I did was added my exclusion text file. I'm just looking to make sure my logic is sound and in this configuration the mover will essentially function like normal but while excluding those directories. Thanks!

 

Edit: based on my testing it seems for the appdata, system, and domains shares where I only want them to move to the array in the off chance cache is nearly full I want to also set: Free down to this level of used Primary (cache) space to 95%.

 

If I understand the logic the mover will move data from cache to the array if cache fills beyond 95% and when it drops below 95% the data will move back to the cache. Obviously my other shares that have TV Shows and Movies, Music etc should stay on the array so both should be set to 0% so no matter what they move to the array and stay there. I just want to make sure I fully understand how this works.

Edited November 11, 20241 yr by mackid1993

Hi,

 

It appears that this plugin has problems moving files containing "$" symbol.

 

Nov 11 03:00:42 abyss root: Moving "/mnt/cache/./Backup/Surface/MyDocs/~$K 7031A.pdf"  to  /mnt/user0/ 
Nov 11 03:00:42 abyss root: rsync: [sender] link_stat "/mnt/cache/Backup/Surface/MyDocs/~ 7031A.pdf" failed: No such file or directory (2)
Nov 11 03:00:42 abyss root: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]

 

Note the Moving filename, the rsync filename differ - I suspect its not escaping the '$' character in the rsync command.

17 minutes ago, Flubster said:

Note the Moving filename, the rsync filename differ - I suspect its not escaping the '$' character in the rsync command.

actually this is a rsync issue, nothing really to-do ...

17 hours ago, Flubster said:

Hi,

 

It appears that this plugin has problems moving files containing "$" symbol.

 

Nov 11 03:00:42 abyss root: Moving "/mnt/cache/./Backup/Surface/MyDocs/~$K 7031A.pdf"  to  /mnt/user0/ 
Nov 11 03:00:42 abyss root: rsync: [sender] link_stat "/mnt/cache/Backup/Surface/MyDocs/~ 7031A.pdf" failed: No such file or directory (2)
Nov 11 03:00:42 abyss root: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1336) [sender=3.2.7]

 

Note the Moving filename, the rsync filename differ - I suspect its not escaping the '$' character in the rsync command.

I had that happen with a few music files, your best plan of attack there is to avoid symbols in filenames entirely which is just good practice anyway. This isn't an issue specific to this plug-in it's common across a number of applications on Linux.

 

For windows PC backups where the file and folder names arent easily changed I switched to using duplicati as it can do backups as chunks of zip files of a controllable size, instead of the raw files, makes using the mover much faster and easier.

 

5 hours ago, Faceman said:

I had that happen with a few music files, your best plan of attack there is to avoid symbols in filenames entirely which is just good practice anyway. This isn't an issue specific to this plug-in it's common across a number of applications on Linux.

 

Yeah these are office temp files, which are being sync'd across to the backup location on the cache, which is in turn being moved to the array. I've excluded the files from the desktop sync process now so shouldn't occur in future. Does seem strange that in a modern OS, issues are still arising with non escaped symbols in filenames, especially as technically $ is not a restricted character in either NTFS/FAT or any *NIX spec filesystem.

 

I had initially piped a find command to a file, and pointed the tuner to this file for exclusion, but the files would remain on the cache which wasn't ideal.

 

Anyway issue is solved. I hadn't ever seen any reference to rsync and issues with some filenames, so its a useful post for reference.

Edited November 12, 20241 yr by Flubster
I ain't getting on no plane, fool

I noticed the ignore list isn't working on my machine either. Running the newest build for Mover tuner and 6.11.5 if that matters at all. 

 

When I run the original mover turner it does follow the ignore list. 

I have a number of items which will not move due to hardlinks, any ideas?

 

unraid move: Expected 2 files, got 1. Not moving "/mnt/cache/./data/media/audiobooks/REDACTED/REDACTED/REDACTED.m4b"  to prevent breaking hardlinks

didnt have this problem on the old fork. what am I missing now? Does mover not recreate the hardlinks anymore?

Edited November 14, 20241 yr by nebb00

Running Unraid version 7 beta 4, the plugins seems to be broken and won't move anything. 

 

Cache pool ended up being full and no way to push it to start for more than a few seconds. 

 

As soon as I disable the plugin the mover runs without issue. 

 

Being beta this may be expected so hopefully it will be resolved OR Unraid can implement the schedule settings natively as they are much more useful than just running once a day etc. @Spencer

Hey, Ibracorp, the man the myth the legend.

 

Are you certain you have version 2024.09.05.0222 of the plugin? the old version is still in the app store but has been broken on Unraid 7 for a very long time.

 

And yea, we all wish Limetech would build in at least some of these features too.

Edited November 16, 20241 yr by Faceman

9 hours ago, Sycotix said:

OR Unraid can implement the schedule settings natively as they are much more useful than just running once a day etc.

running more often is the "nativ" feature from mover ...

 

all the extra settings are the features here ... so how often mover is triggered is done by the default mover settings, just as note.

I have the mover schedule set to run hourly, most of the time it doesn't actually move anything because of the mover tuning rules. With this plugin the native schedule options become how often you want the system to check the rules.