On 9/4/2024 at 5:06 PM, Reynald said:

I see but... I don't see the point of having the same file with 2 different names/paths on 2 different storages. Moving just the primary file to secondary storage won't save any space on primary storage, so better to keep the hardlinked files on the same place.

However, if I understand your use case correctly you can keep a file on both the cache and the array by using this option:

 

I'd like feedback on this feature from those who use it, regarding deletion in particular.

Will that option respect the option to ignore hidden files and folders?

Almost: per share options will be honoured

 

After testing and checking: no it won't

Edited September 7, 20242 yr by Reynald

On 9/4/2024 at 4:06 PM, Reynald said:

I see but... I don't see the point of having the same file with 2 different names/paths on 2 different storages. Moving just the primary file to secondary storage won't save any space on primary storage, so better to keep the hardlinked files on the same place.
 

I have the same setup as @Notwist it seems, where my cache pool has a /data/torrents folder that I have set to ignore so it can continue seeding until a certain time or ratio is met, and then /data/media folder that I'd like to move daily to the array so it is parity protected. The files in the /media folder are renamed so they are organized properly, whereas the /torrents folder are named as released, which is only temporary because it's based on the time / ratio / tracker.

I tried using the Synchronize feature but still get the same error, not moved in order to "prevent breaking hardlinks".

 

Is there another way around this or more efficient way of doing things? I'm using trash-guides for the hardlinks and everything is working great besides now the mover, as I am trying to keep the seeding torrents on the cache only and move only the renamed library files.

Edited September 7, 20242 yr by tehnards

Oh, I understand now!

The point is that synchronization does not completly work has it does not want to break hardlinks (which is not really as it shall be a copy).

 

I'm going to work on that

On 9/7/2024 at 8:06 AM, tehnards said:

I have the same setup as @Notwist it seems, where my cache pool has a /data/torrents folder that I have set to ignore so it can continue seeding until a certain time or ratio is met, and then /data/media folder that I'd like to move daily to the array so it is parity protected. The files in the /media folder are renamed so they are organized properly, whereas the /torrents folder are named as released, which is only temporary because it's based on the time / ratio / tracker.

I tried using the Synchronize feature but still get the same error, not moved in order to "prevent breaking hardlinks".

 

Is there another way around this or more efficient way of doing things? I'm using trash-guides for the hardlinks and everything is working great besides now the mover, as I am trying to keep the seeding torrents on the cache only and move only the renamed library files.

 

I'm still working on Synchronization to allow breaking hardkinks and also better cleanup.

 

In the meantime you can sync everything (remove filter), there is no extra used space as you have the counterparts of each links on both array and cache in your use case (plus, in case of SSD failure, you can still finish downloads and seedings)

1 hour ago, Reynald said:

 

I'm still working on Synchronization to allow breaking hardkinks and also better cleanup.

 

 

 

@Reynald Rather then fix synchronize first, if you can fix

Ignore files and folders listed inside of a text file: & File list path: you will fix the biggest obstacle to use using the plugin. That would give you more time to fix synchronize. Also, I suspect the ignore files and folders will be an easier fix.

 

8 hours ago, unraid userr said:

 

 

@Reynald Rather then fix synchronize first, if you can fix

Ignore files and folders listed inside of a text file: & File list path: you will fix the biggest obstacle to use using the plugin. That would give you more time to fix synchronize. Also, I suspect the ignore files and folders will be an easier fix.

 

You're certainly right but... What is not working with ignore files & folder list?

16 hours ago, Reynald said:

 

I'm still working on Synchronization to allow breaking hardkinks and also better cleanup.

 

In the meantime you can sync everything (remove filter), there is no extra used space as you have the counterparts of each links on both array and cache in your use case (plus, in case of SSD failure, you can still finish downloads and seedings)

Okay, thank you. When you say "remove filter", you mean remove the ignore folder? I have unsorted / unimported files in there too I don't want to move till they are renamed correctly.

On 9/5/2024 at 8:25 PM, Reynald said:

Hi,

 

It is not natively supported by the plugin, but you could create a user script (with user script plugin), like this one (untested but it's similar to how the share "move now" button works):

#!/bin/bash

sharename=yourshare
/usr/local/emhttp/plugins/ca.mover.tuning/share_mover {$shareName} >> /var/log/syslog &

 

and add the schedule.

If you need several to move several shares, duplicate the script but set different custom schedule so execution don't overlap, else you'll get "mover already running" message and second instance won't move anything.

 

Maybe others have better ideas

Thanks for the advice. I think the script method will work fine for me. Testing this script I'm not sure if it works for me, should the mover show as running in the "Main" tab, as it doesn't when I attempt to run. My share name is media btw, and I corrected the variable to have a lower case in the shareName to sharename just in case that makes a difference.

I also googled there is a command just called mover, so I tried that but it just runs the mover tuning settings instead and there-fore skips unless at 70%, which kinda makes sense haha.

Any other ideas please?

10 hours ago, 32wqesdvb5 said:

Testing this script I'm not sure if it works for me, should the mover show as running in the "Main" tab, as it doesn't when I attempt to run. My share name is media btw, and I corrected the variable to have a lower case in the shareName to sharename just in case that makes a difference.

Any other ideas please?

 

Script configuration in your case is like this:

 

#!/bin/bash

 

shareName=media

/usr/local/emhttp/plugins/ca.mover.tuning/share_mover {$shareName} >> /var/log/syslog &

 

(There was a typo in my previous proposal, you've nicely seen it 😊)

 

You can run it from terminal and check for any error.

 

You can check syslog (from UnRaid webui) to see if mover is running and get feeback

I'm not sure how feasible this would be... but is it possible to have an "Opportunistic Mover" mode?

 

Basically instead of scheduling things based on time or age or free space on cache, why not move things after array disks have spun down for a certain period of time? You could maybe even get creative and move certain shares based on the disks they have in scope. For example if my Movies share occupies disks 1,2,3 and all 3 of those have been spundown for at least 30 minutes... then kick on mover and move stuff to those disks. If lets say disk 2 is active and spun up, then don't move anything for shares that have disk2 in scope.

 

On 8/19/2024 at 1:14 PM, Reynald said:

Once in appdata, you have to 'chmod +x /mnt/appdata/bothscripts.sh'

Ok, I know it's been a while, but now I could sit and try to test it.

and.... it's strange, I do not understand that.

Having scripts in appdata folder seem not to work (trying to run scripts even with "bash" command in terminal, it gives me an information about that docker is not known), 

BUT pasting scripts to /boot/config/plugins/ca.mover.tuning and run them with terminal bash /boot/config/plugins/ca.mover.tuning/dockeroff.sh command seem to work.

 

any tips?

8 hours ago, rogales said:

Having scripts in appdata folder seem not to work (trying to run scripts even with "bash" command in terminal, it gives me an information about that docker is not known), 

may post your results here fully ... should work just fine

Regarding the ignore file list, if I want to ignore a couple folders, what should I specify for the latest version of the mover turning plugin?  I tried to search in this forum but I found contradict comments.

 

Some suggest to use

/mnt/cache/media/downloads/*

/mnt/cache/media/incomplete/*

 

Some said

/mnt/cache/media/downloads

/mnt/cache/media/incomplete

 

 

I wonder which one is the correct format to put into the ignore file list text file? 

4 hours ago, SDUGoten said:

Regarding the ignore file list, if I want to ignore a couple folders, what should I specify for the latest version of the mover turning plugin?  I tried to search in this forum but I found contradict comments.

 

Some suggest to use

/mnt/cache/media/downloads/*

/mnt/cache/media/incomplete/*

 

Some said

/mnt/cache/media/downloads

/mnt/cache/media/incomplete

 

 

I wonder which one is the correct format to put into the ignore file list text file? 

/mnt/cache/media/downloads

/mnt/cache/media/incomplete

On 9/17/2024 at 5:38 AM, alturismo said:

may post your results here fully ... should work just fine

which results should I post?

7 hours ago, rogales said:

which results should I post?

when you run them manually in terminal

I'm using my flash drive on a new config, so the plugin is throwing a bunch of errors in the logs referencing the names of previous cache pools that don't exist anymore. Would a reinstall of the plugin solve this? Or perhaps is there a config file somewhere I can delete and have the plugin recreate?

12 hours ago, JohnnyGrey said:

Would a reinstall of the plugin solve this? Or perhaps is there a config file somewhere I can delete and have the plugin recreate?

may take a look at /flash/config/shares and check which are still valid

14 hours ago, alturismo said:

may take a look at /flash/config/shares and check which are still valid

@JohnnyGrey I confirm Alturismo advise. If "error" are thrown by the plugin, it is because the cfg file exists in /boot/config/shares and the associated folder does not exist. Please create the (empty) folder on cache or delete cfg file.
You can also ignore this warning, this one is harmless.

The mover does not appear to be cleaning up empty directories anymore using the age mover.  the "Clean Empty Folders" setting is set to true. 

I'm seeing the mover doing something that I do not understand. It is moving the files from my cache to my array, but the share is set Array -> Cache-docker. So that does not make sense? Any idea why it is doing this? Is this a setting I wrongfully set, or is it a bug?

 

edit: and the next run it moves everything back… and then again.  I disabled it for now. 

Quote

Oct  5 06:00:02 homelabserver root: ----------------------------------------------------------------- Global settings ---------------------------------------------------------------
Oct  5 06:00:02 homelabserver root: Using global moving threshold: 95 %
Oct  5 06:00:02 homelabserver root: Using global freeing threshold: 80 %
Oct  5 06:00:02 homelabserver root: Age: no = 0 ; daysold:
Oct  5 06:00:02 homelabserver root: Clean Folders: yes
Oct  5 06:00:02 homelabserver root: Enable Turbo: yes
Oct  5 06:00:02 homelabserver root: ***************************************************************** FILTERING FILES ***************************************************************
Oct  5 06:00:02 homelabserver root: ------------------------------------------------------ Processing afbeeldingen-immich share -----------------------------------------------------
Oct  5 06:00:02 homelabserver root: Primary storage: cache-docker - size: 1.9TiB - used:  32 % (588GiB)
Oct  5 06:00:02 homelabserver root: Secondary storage: user0
Oct  5 06:00:02 homelabserver root: Share Information: Name: afbeeldingen-immich - Path: /mnt/*/afbeeldingen-immich
Oct  5 06:00:02 homelabserver root: Moving threshold: 95% (1.8TiB) ; Freeing threshold: 80% (1.5TiB)
Oct  5 06:00:02 homelabserver root: Mover action: user0->cache-docker (cache:prefer). Pool is below priming threshold percentage:  32% < 80%.
Oct  5 06:00:02 homelabserver root: => Will smart move newest files from user0 to cache-docker until threshold. Older files will be moved from cache-docker to user0.
Oct  5 06:00:05 homelabserver su[9493]: pam_unix(su:session): session closed for user root
Oct  5 06:00:05 homelabserver sudo: pam_unix(sudo:session): session closed for user root
Oct  5 06:00:12 homelabserver root: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-10-05T060002.list for afbeeldingen-immich

 

Edited October 5, 20242 yr by Soulplayer

On 10/2/2024 at 6:38 PM, Terebi said:

The mover does not appear to be cleaning up empty directories anymore using the age mover.  the "Clean Empty Folders" setting is set to true. 

 

The logs shall tell you why. In example:

Quote

16:33:56.490 270549 files remaining from caches to array  2.5TiB
16:33:56.529 Moving "/mnt/cache/./media/_downloads/torrents/complete/sonarr/Pokemon S18 - AMZN WEB-DL 1920x1080 FR [ABYSS]/Pokemon S18E43 - AMZN WEB-DL 1920x1080 FR [ABYSS].mkv"  to  /mnt/user0/ 
Not deleting dir containing 29 files: /mnt/cache/media/_downloads/torrents/complete/sonarr/Pokemon S18 - AMZN WEB-DL 1920x1080 FR [ABYSS]

[...]

16:44:47.173 270528 files remaining from caches to array  2.5TiB
16:44:47.211 Moving "/mnt/cache/./media/_downloads/torrents/complete/sonarr/Your.Honor.US.S02E02.MULTi.DV.HDR.2160p.WEB-DL.DD5.1.H265-R3MiX/Your.Honor.US.S02E02.MULTi.DV.HDR.2160p.WEB-DL.DD5.1.H265-R3MiX.nfo"  to  /mnt/user0/ 
Deleting empty dir /mnt/cache/media/_downloads/torrents/complete/sonarr/Your.Honor.US.S02E02.MULTi.DV.HDR.2160p.WEB-DL.DD5.1.H265-R3MiX
 

 

Edited October 6, 20242 yr by Reynald

On 10/5/2024 at 6:11 AM, Soulplayer said:

I'm seeing the mover doing something that I do not understand. It is moving the files from my cache to my array, but the share is set Array -> Cache-docker. So that does not make sense? Any idea why it is doing this? Is this a setting I wrongfully set, or is it a bug?

 

edit: and the next run it moves everything back… and then again.  I disabled it for now. 

 

In the log you've shown mover plugin says it will prime your cache up to the threshold. Can you please show several logs of it moving back and forth?

 

I've not extensively tested with age not set. I prefer to use it with 'automatic age' (plugin is extensively tested with this option)

Seems there is a bug in the plugin that causes it to not work with shares that have a space in the name.

Just found this in the logs after failing to figure out why my cache kept filling up no matter what settings I used.

 

Oct  8 16:15:33 Tower move: Primary storage: new_cache - size: 900GiB - used: 49 % (458GiB)
Oct  8 16:15:33 Tower move: Secondary storage: user0
Oct  8 16:15:33 Tower move: Share Information: Name: TV Shows - Path: /mnt/cache/TV Shows
Oct  8 16:15:33 Tower move: Moving threshold: 5% (45GiB) ; Freeing threshold: 5% (45GiB)
Oct  8 16:15:33 Tower move: Mover action: cache->user0 (cache:yes). Pool is above moving threshold percentage:  49% >= 5%.
Oct  8 16:15:33 Tower move: => Will smart move old files from cache to user0. Nothing will be moved from user0 to new_cache
Oct  8 16:15:33 Tower move: find: '/mnt/cache/TV': No such file or directory
Oct  8 16:15:33 Tower move: find: 'Shows': No such file or directory
Oct  8 16:15:33 Tower move: awk: fatal: cannot open file `-v' for reading: No such file or directory
Oct  8 16:15:33 Tower move: Updated Filtered filelist: /tmp/ca.mover.tuning/Filtered_files_2024-10-08T161531.list for TV Shows