[Plugin] Mover Tuning

[Reynald]

2024.09.05.0222

• Fix find not finding hidden files Thanks to solidno8 from forums. (R3yn4ld)

[32wqesdvb5]

Can I get some advice please? I'd like the mover to run quite often, every hour lets say, but it only runs if the cache limit has reached around 70%. If the cache is low then I would want the mover to skip. I think I can do this. The bit I'm stuck on comes next. I then want a second option that runs the mover in the night, and it runs no matter the cache limit. This ensures all data is moved to the array once a day no matter the cache size. The hourly schedule is acting more like an emergency move if cache is high but otherwise it waits till night time.

Is this possible please?

[Reynald]

6 hours ago, 32wqesdvb5 said:

Can I get some advice please? I'd like the mover to run quite often, every hour lets say, but it only runs if the cache limit has reached around 70%. If the cache is low then I would want the mover to skip. I think I can do this. The bit I'm stuck on comes next. I then want a second option that runs the mover in the night, and it runs no matter the cache limit. This ensures all data is moved to the array once a day no matter the cache size. The hourly schedule is acting more like an emergency move if cache is high but otherwise it waits till night time.

Is this possible please?

Hi,

 

It is not natively supported by the plugin, but you could create a user script (with user script plugin), like this one (untested but it's similar to how the share "move now" button works):

#!/bin/bash

sharename=yourshare
/usr/local/emhttp/plugins/ca.mover.tuning/share_mover {$shareName} >> /var/log/syslog &

 

and add the schedule.

If you need several to move several shares, duplicate the script but set different custom schedule so execution don't overlap, else you'll get "mover already running" message and second instance won't move anything.

 

Maybe others have better ideas

Edited September 5 by Reynald

[Notwist]

On 9/4/2024 at 5:06 PM, Reynald said:

I see but... I don't see the point of having the same file with 2 different names/paths on 2 different storages. Moving just the primary file to secondary storage won't save any space on primary storage, so better to keep the hardlinked files on the same place.

However, if I understand your use case correctly you can keep a file on both the cache and the array by using this option:

 

I'd like feedback on this feature from those who use it, regarding deletion in particular.

Will that option respect the option to ignore hidden files and folders?

[Reynald]

Almost: per share options will be honoured

 

After testing and checking: no it won't

Edited September 7 by Reynald

[tehnards]

On 9/4/2024 at 4:06 PM, Reynald said:

I see but... I don't see the point of having the same file with 2 different names/paths on 2 different storages. Moving just the primary file to secondary storage won't save any space on primary storage, so better to keep the hardlinked files on the same place.
 

I have the same setup as @Notwist it seems, where my cache pool has a /data/torrents folder that I have set to ignore so it can continue seeding until a certain time or ratio is met, and then /data/media folder that I'd like to move daily to the array so it is parity protected. The files in the /media folder are renamed so they are organized properly, whereas the /torrents folder are named as released, which is only temporary because it's based on the time / ratio / tracker.

I tried using the Synchronize feature but still get the same error, not moved in order to "prevent breaking hardlinks".

 

Is there another way around this or more efficient way of doing things? I'm using trash-guides for the hardlinks and everything is working great besides now the mover, as I am trying to keep the seeding torrents on the cache only and move only the renamed library files.

Edited September 7 by tehnards

[Reynald]

Oh, I understand now!

The point is that synchronization does not completly work has it does not want to break hardlinks (which is not really as it shall be a copy).

 

I'm going to work on that

[Reynald]

On 9/7/2024 at 8:06 AM, tehnards said:

I have the same setup as @Notwist it seems, where my cache pool has a /data/torrents folder that I have set to ignore so it can continue seeding until a certain time or ratio is met, and then /data/media folder that I'd like to move daily to the array so it is parity protected. The files in the /media folder are renamed so they are organized properly, whereas the /torrents folder are named as released, which is only temporary because it's based on the time / ratio / tracker.

I tried using the Synchronize feature but still get the same error, not moved in order to "prevent breaking hardlinks".

 

Is there another way around this or more efficient way of doing things? I'm using trash-guides for the hardlinks and everything is working great besides now the mover, as I am trying to keep the seeding torrents on the cache only and move only the renamed library files.

 

I'm still working on Synchronization to allow breaking hardkinks and also better cleanup.

 

In the meantime you can sync everything (remove filter), there is no extra used space as you have the counterparts of each links on both array and cache in your use case (plus, in case of SSD failure, you can still finish downloads and seedings)

[unraid userr]

1 hour ago, Reynald said:

 

I'm still working on Synchronization to allow breaking hardkinks and also better cleanup.

 

 

 

@Reynald Rather then fix synchronize first, if you can fix

Ignore files and folders listed inside of a text file: & File list path: you will fix the biggest obstacle to use using the plugin. That would give you more time to fix synchronize. Also, I suspect the ignore files and folders will be an easier fix.

 

[Reynald]

8 hours ago, unraid userr said:

 

 

@Reynald Rather then fix synchronize first, if you can fix

Ignore files and folders listed inside of a text file: & File list path: you will fix the biggest obstacle to use using the plugin. That would give you more time to fix synchronize. Also, I suspect the ignore files and folders will be an easier fix.

 

You're certainly right but... What is not working with ignore files & folder list?

[tehnards]

16 hours ago, Reynald said:

 

I'm still working on Synchronization to allow breaking hardkinks and also better cleanup.

 

In the meantime you can sync everything (remove filter), there is no extra used space as you have the counterparts of each links on both array and cache in your use case (plus, in case of SSD failure, you can still finish downloads and seedings)

Okay, thank you. When you say "remove filter", you mean remove the ignore folder? I have unsorted / unimported files in there too I don't want to move till they are renamed correctly.

[32wqesdvb5]

On 9/5/2024 at 8:25 PM, Reynald said:

Hi,

 

It is not natively supported by the plugin, but you could create a user script (with user script plugin), like this one (untested but it's similar to how the share "move now" button works):

#!/bin/bash

sharename=yourshare
/usr/local/emhttp/plugins/ca.mover.tuning/share_mover {$shareName} >> /var/log/syslog &

 

and add the schedule.

If you need several to move several shares, duplicate the script but set different custom schedule so execution don't overlap, else you'll get "mover already running" message and second instance won't move anything.

 

Maybe others have better ideas

Thanks for the advice. I think the script method will work fine for me. Testing this script I'm not sure if it works for me, should the mover show as running in the "Main" tab, as it doesn't when I attempt to run. My share name is media btw, and I corrected the variable to have a lower case in the shareName to sharename just in case that makes a difference.

I also googled there is a command just called mover, so I tried that but it just runs the mover tuning settings instead and there-fore skips unless at 70%, which kinda makes sense haha.

Any other ideas please?

[Reynald]

10 hours ago, 32wqesdvb5 said:

Testing this script I'm not sure if it works for me, should the mover show as running in the "Main" tab, as it doesn't when I attempt to run. My share name is media btw, and I corrected the variable to have a lower case in the shareName to sharename just in case that makes a difference.

Any other ideas please?

 

Script configuration in your case is like this:

 

#!/bin/bash

 

shareName=media

/usr/local/emhttp/plugins/ca.mover.tuning/share_mover {$shareName} >> /var/log/syslog &

 

(There was a typo in my previous proposal, you've nicely seen it 😊)

 

You can run it from terminal and check for any error.

 

You can check syslog (from UnRaid webui) to see if mover is running and get feeback

[johnsanc]

I'm not sure how feasible this would be... but is it possible to have an "Opportunistic Mover" mode?

 

Basically instead of scheduling things based on time or age or free space on cache, why not move things after array disks have spun down for a certain period of time? You could maybe even get creative and move certain shares based on the disks they have in scope. For example if my Movies share occupies disks 1,2,3 and all 3 of those have been spundown for at least 30 minutes... then kick on mover and move stuff to those disks. If lets say disk 2 is active and spun up, then don't move anything for shares that have disk2 in scope.