[Plugin] Mover Tuning

romain · August 15, 2023

I'm having an issue where if I have this plugin installed mover starts and immediately stops without moving anything, but if I uninstall the plugin mover works fine.

I have 2 cache pools:

- Cache

- Sabnzbd

Any shares that use the Cache pool are set to Cache only, so mover only moves files from the Sabnzbd cache pool to the array.

When I install the Mover Tuning plugin, all I do is update this one setting;

Move Now button follows plug-in filters: Yes

and after that if I click MOVE NOW, mover starts and immediately stops, here is the log:

Aug 14 18:09:33 unRAID move: mover: started
Aug 14 18:09:34 unRAID move: mover: finished

I've tried reinstalling and updating the plugin and have messed around with some other settings, but after whittling settings away it looks like that's all it takes to keep Mover from running for me. I was wondering if it was an issue with my using a cache pool called Sabnzbd instead of Cache, but saw a thread on reddit saying that the plugin should work fine with multiple cache pools so now I'm not really sure what the issue is.

Swarles · August 15, 2023

3 hours ago, romain said:
I'm having an issue where if I have this plugin installed mover starts and immediately stops without moving anything, but if I uninstall the plugin mover works fine.

I have 2 cache pools:

- Cache

- Sabnzbd

Any shares that use the Cache pool are set to Cache only, so mover only moves files from the Sabnzbd cache pool to the array.

When I install the Mover Tuning plugin, all I do is update this one setting;

Move Now button follows plug-in filters: Yes

and after that if I click MOVE NOW, mover starts and immediately stops, here is the log:
Aug 14 18:09:33 unRAID move: mover: started
Aug 14 18:09:34 unRAID move: mover: finished
I've tried reinstalling and updating the plugin and have messed around with some other settings, but after whittling settings away it looks like that's all it takes to keep Mover from running for me. I was wondering if it was an issue with my using a cache pool called Sabnzbd instead of Cache, but saw a thread on reddit saying that the plugin should work fine with multiple cache pools so now I'm not really sure what the issue is.

I can try help with this, first thing to check would be if "share.cfg" exists.

Go to “cd /boot/config” and identify if “share.cfg” file exists.
If it does, verify that the line 'shareCacheEnabled="yes"' is in there.
If no to either 1 or 2, do one of the following:

To have the file created by the system, edit your global share settings (edit, save, revert). "Settings>Global Share Settings"
OR; Paste the following in the terminal:

echo shareCacheEnabled='"yes"' >> /boot/config/share.cfg

If the file did exist and you verified step 2, do the following:

Enable Mover Logging.
Set “Move Now Button Follows Plug-In Filters” to yes.
Set “Test Mode” to yes.
Run mover.
Go to “cd /top/Mover” and share the contents of the most recent ".log" file.

Edited August 15, 2023 by Swarles

romain · August 15, 2023

2 hours ago, Swarles said:
I can try help with this, first thing to check would be if "share.cfg" exists.

Go to “cd /boot/config” and identify if “share.cfg” file exists.

If it does, verify that the line 'shareCacheEnabled="yes"' is in there.

If no to either 1 or 2, do one of the following:
To have the file created by the system, edit your global share settings (edit, save, revert). "Settings>Global Share Settings"

OR; Paste the following in the terminal:
echo shareCacheEnabled='"yes"' >> /boot/config/share.cfg
If the file did exist and you verified step 2, do the following:

Enable Mover Logging.

Set “Move Now Button Follows Plug-In Filters” to yes.

Set “Test Mode” to yes.

Run mover.

Go to “cd /tmp/Mover” and share the contents of the most recent ".log" file.

@Swarles Thanks for responding!

share.cfg exists and shareCacheEnabled="yes" was present in that file, so here is the log after following your other instructions.

Mover_Tuning_2023-08-15T000944.log

romain · August 16, 2023

@Swarles

What does the line below do in the context of Mover? Does it give a list of files to move?

Skipfiletypes string: find "/mnt/nzbget/Media" -depth

I ran it and got nothing, because my share name is "media", not "Media" - I replaced the upper case "M" with a lower case "m" and got a full list of everything on the cache pool.

Could that be the reason that I'm having an issue? If so, I don't know how to fix it if somehow Unraid is capitalizing a letter that it shouldn't be. I took a look around and there are no upper case M's anywhere that I've typed that I can find so I'm not sure why it's capitalized in the log.

Swarles · August 16, 2023

19 minutes ago, romain said:
@Swarles

What does the line below do in the context of Mover? Does it give a list of files to move?
Skipfiletypes string: find "/mnt/nzbget/Media" -depth
I ran it and got nothing, because my share name is "media", not "Media" - I replaced the upper case "M" with a lower case "m" and got a full list of everything on the cache pool.

Could that be the reason that I'm having an issue? If so, I don't know how to fix it if somehow Unraid is capitalizing a letter that it shouldn't be. I took a look around and there are no upper case M's anywhere that I've typed that I can find so I'm not sure why it's capitalized in the log.

Yep that appears to be your issue as I don't see anything else wrong. The process appears to be fully executing but just not finding any files to move.

Check the following location:

ls /boot/config/shares

You must have a file called "Media.cfg". I'm not sure how unraid processes this side of things, did you at any point rename the capital letter? If you directly rename the file to "media.cfg" it will work for the mover tuning plugin, not sure how unraid will handle it. Perhaps try renaming the share (in settings) to something completely different, saving etc, then renaming it back to "media" and see if it changes

romain · August 16, 2023

15 hours ago, Swarles said:
Yep that appears to be your issue as I don't see anything else wrong. The process appears to be fully executing but just not finding any files to move.

Check the following location:
ls /boot/config/shares
You must have a file called "Media.cfg". I'm not sure how unraid processes this side of things, did you at any point rename the capital letter? If you directly rename the file to "media.cfg" it will work for the mover tuning plugin, not sure how unraid will handle it. Perhaps try renaming the share (in settings) to something completely different, saving etc, then renaming it back to "media" and see if it changes

@Swarles That fixed it! Changed the share to media1234 and then back to media and the .cfg file updated itself accordingly and mover started working. Thanks for the help.

privateer · August 18, 2023

Having issues with the Script to run before mover. I've run the script through CLI and it works as expected. When triggered via mover tuner it doesn't seem to have any effect.

Aug 18 11:00:01 Tower root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' '' "/mnt/user/[rest of path]/1.py" '' '' '' '' '' 55

Any idea whats going on? Any additional info I can provide to help?

Swarles · August 21, 2023

On 8/19/2023 at 2:54 AM, privateer said:

Having issues with the Script to run before mover. I've run the script through CLI and it works as expected. When triggered via mover tuner it doesn't seem to have any effect.

Aug 18 11:00:01 Tower root: ionice -c 2 -n 0 nice -n 0 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start 0 0 0 '' '' "/mnt/user/[rest of path]/1.py" '' '' '' '' '' 55

Any idea whats going on? Any additional info I can provide to help?

My first guess will be that perhaps you are missing the shebang at the beginning of your script? The shebang has to be on line one and point to the location of your python interpreter. For example:

#!/usr/bin/python3

But edit it ^^ to make sure it is your location.

This is my best guess and you should test you can otherwise run your script with just "./mnt/user/[rest of path]/1.py" in the terminal. If it does run properly but the shebang doesn't fix it, you will have to go to /tmp/Mover and provided the latest ".log" file in there for me to see what the mover is doing.

Mamaun · August 22, 2023

On 7/28/2023 at 1:11 AM, jch said:

Is there a way to configure the "Move All from Cache-Yes shares when disk is above a certain percentage" / "Move All from Cache-yes shares pool percentage" settings to move only enough files to get the disk below a certain percentage?

The practical use-case is that if the cache is being used for hot-file access, you want to keep as many files as possible in the cache. Ideally you can evict files from the cache drive based on last access or creation time (in addition to the filter settings).

EDIT: I found the comment about using the "keep files under a certain age", but that requires some fairly precise tuning and the workload on the cache disk is fairly variable. It looks like a candidate location to filter the file list would be here https://github.com/hugenbd/ca.mover.tuning/blob/master/source/ca.mover.tuning/usr/local/emhttp/plugins/ca.mover.tuning/age_mover#L753 (to sort the file list by age and then take files until you get to the threshold). I don't have a dev environment setup for doing this otherwise I'd help out :).

This feature would be awesome.

I'm using a disk for hot-file access indeed. This would be THE FEATURE for me.

Edited August 22, 2023 by Mamaun

loond · August 22, 2023

Running 6.12.3

So move ran and while it did move files from a zfs pool to the array as expected, I'm left with a bunch of empty directories in the share created dataset; also have the CA Mover Tuning plugin. I've tried to re run the mover in hope it would clean up the empty directories but no dice. My setup is essentially tiered storage with files moved off of the zfs pool based on age, so I don't want everything moved at once.

Is this a know issue? If yes, any simple solution to clean them up; like a reboot, etc.?

Thanks in advance.

hugenbdd · August 22, 2023

1 hour ago, loond said:

Running 6.12.3

So move ran and while it did move files from a zfs pool to the array as expected, I'm left with a bunch of empty directories in the share created dataset; also have the CA Mover Tuning plugin. I've tried to re run the mover in hope it would clean up the empty directories but no dice. My setup is essentially tiered storage with files moved off of the zfs pool based on age, so I don't want everything moved at once.

Is this a know issue? If yes, any simple solution to clean them up; like a reboot, etc.?

Thanks in advance.

It should delete the empty directories second/next time it's run.

hugenbdd · August 22, 2023

New release from some files Swarles and I have made last month. Sorry for the delay, life got in the way....

2023.08.22

Fixed Cron Job entry

Modified ignore command to include folders (Swarles)

Updated mover cmdline functions (Hugenbdd)

Swarles · August 22, 2023

2 hours ago, loond said:

So move ran and while it did move files from a zfs pool to the array as expected, I'm left with a bunch of empty directories in the share created dataset;

I'm running zfs on my system and it appears to be working as expected, folders on cache are being deleted.
If the folder isn't removed on second run like hugenbdd said, can you specify if these said folders are datasets themselves? If they are this would make sense as datasets need to be destroyed and can't be deleted normally.

Swarles · August 22, 2023

8 hours ago, Mamaun said:

This feature would be awesome.

I'm using a disk for hot-file access indeed. This would be THE FEATURE for me.

I agree this feature would be very nice to have as I use my cache pool in the same manner.

So I am just letting you know that it is certainly in my sights and at some point I would like to implement it

However, there are some factors that need to be considered making it's implementation time consuming which means it likely won't be added by hugenbdd or myself in the near future. I wont have free time for a couple more months.

Some of these considerations are:

When a cache pool starts to exceed it's threshold:
- Should the oldest files be moved first?
- Should removal be based on ctime, mtime, or atime?
  - Should these settings be configurable?
  - Should they be configurable per-share in the pool?
- Should the user be able to have a preference system for which shares on the pool are targeted first?
- Should a limit or threshold be specifiable for individual shares?
  - Eg. on a 1TB pool, should you be able to "allocate" 200GB to "share1" which if exceeded performs the balance.
- What takes precedence when there are other mover settings stopping the oldest files from being moved?
  - What if the other setting stopping files from being moved affect many files to the point where moving all the allowable files doesn't bring it below the threshold?
Then there are considerations on where and how all this should happen in the code which need to be worked out.

This is sort of to just provide an update on where my thought process is at. I can answer many of these questions myself but I know that implementing all of it will take a while + testing, and I won't be starting it for at least 2 months. With that said if you can think of more features I am more than glad to hear them, I'll mull them over in my mind for now.

Robbie G78 · August 23, 2023

Unraid 6.12.3 CA Mover Tuning 2023.08.22

After updating the CA mover tuning plugin, clicking on the "Move" button on the Main screen doesn't do anything except refresh the screen. I doesn't invoke mover. The only way I was able to get the mover to start was by going into the share that I needed the files moved on and invoking "Move" from there. I don't think the schedule for mover is running either.

Edited August 23, 2023 by AWIFI Rob

hugenbdd · August 23, 2023

Can you check to see if the mover file has been created? (It's location was changed between 6.11 and 6.12)

ls -ltr /usr/local/sbin/move

and

ls -ltr /usr/local/bin/move

If it is not in /usr/local/sbin, then run this command to create a link.

ln -s /usr/local/bin/move /usr/local/sbin/move

loond · August 23, 2023

23 hours ago, hugenbdd said:

It should delete the empty directories second/next time it's run.

Configured it to run again last night, manual runs didn't change the situation. The empty folders are still there this morning. Some additional details and responses to questions from other respondents:

It's a zfs pool and the folders not being removed are just directories, not datasets. These are for my movies and tv shares where the pool is the cache for the array. Files/directories are set to move based on age and the mover is set to run on a % of pool used. Basically, newer stuff stays on the pool and older stuff is moved onto the array. I have multiple pools and a couple of legacy btrfs caches that fall under the global mover tuning, however on the above shares I override the age settings. I haven't tried a reboot yet, but will give that a go this evening.

Open to other ideas or did I just find an edge case?

loond · August 23, 2023

Just had another thought, what if the folder has a different date than the file within it. If the folder is newer and doesn't meet the age cut off yet then it isn't moved/removed; even if the files within it do meet the age criteria and are moved, which results in an empty directory.

Not sure on the logic being used, but there should be a catch all that says if directory empty remove regardless of age setting. Basically, a clean up function at the end of the move. I can find all of the empty directories from the terminal via:

find /mnt/mediapool/TV/ -type d -empty

mediapool being the zfs dataset and TV being the directory

loond · August 23, 2023

7 minutes ago, loond said:
Just had another thought, what if the folder has a different date than the file within it. If the folder is newer and doesn't meet the age cut off yet then it isn't moved/removed; even if the files within it do meet the age criteria and are moved, which results in an empty directory.

Not sure on the logic being used, but there should be a catch all that says if directory empty remove regardless of age setting. Basically, a clean up function at the end of the move. I can find all of the empty directories from the terminal via:
find /mnt/mediapool/TV/ -type d -empty
mediapool being the zfs dataset and TV being the directory

Adding "-delete" the above does clean up the empty folders.

find /mnt/mediapool/TV/ -type d -empty -delete

No impact when looking at /mnt/user/* meaning the files and directories are still on the array as desired.

Robbie G78 · August 23, 2023

3 hours ago, hugenbdd said:

Can you check to see if the mover file has been created? (It's location was changed between 6.11 and 6.12)

ls -ltr /usr/local/sbin/move

and

ls -ltr /usr/local/bin/move

If it is not in /usr/local/sbin, then run this command to create a link.

ln -s /usr/local/bin/move /usr/local/sbin/move

Looks like both are there...

image.png.9cbec8d87863064eb19649377653755c.png

trurl · August 24, 2023

23 hours ago, Robbie G78 said:

Looks like both are there...

Looks like one is just a pointer to the other

Robbie G78 · August 24, 2023

6 minutes ago, trurl said:

Looks like one is just a pointer to the other

So, should I still just run that command and that fixes it?

jamikest · August 24, 2023

On 7/12/2023 at 11:26 AM, Swarles said:

Otherwise I'll ask hugendbb if we can change it permanently.

I just wanted to say thanks to you and unr41dus3r. I had this same issue with my forced mover. Hopefully a permanent fix can be implemented.

Kloudz · August 24, 2023

I feel like since the last update. Mover stopped working. I have my scheduler set to every 2 hours and there has been no movement. This happened before when 6.12.0 was released.

Kloudz · August 24, 2023

On 8/23/2023 at 5:44 AM, Robbie G78 said:

Unraid 6.12.3 CA Mover Tuning 2023.08.22

After updating the CA mover tuning plugin, clicking on the "Move" button on the Main screen doesn't do anything except refresh the screen. I doesn't invoke mover. The only way I was able to get the mover to start was by going into the share that I needed the files moved on and invoking "Move" from there. I don't think the schedule for mover is running either.

I have this same issue, I triggered "Move" from the main screen and nothing happens.

I also removed the plugin and now mover isn't working for any zfs pools that are used for cache

Edited August 24, 2023 by Kloudz

[Plugin] Mover Tuning

Recommended Posts

Link to comment

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Squid

hugenbdd

hugenbdd

Posted Images

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation