[Plugin] Mover Tuning


Recommended Posts

5 minutes ago, Andiroo2 said:

 

Hi all,

 

Similar to user @chizll, I am seeing this message in my logs as well, every time mover runs:

 


Feb  1 09:10:42 Tower root: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover: line 171: [: no: integer expression expected
Feb  1 09:10:42 Tower root: Mover not Needed.
Feb  1 09:10:42 Tower root: Share Config: /boot/config/shares/Photos.cfg
Feb  1 09:10:42 Tower root: Pool Pct Used:   5 %
Feb  1 09:10:42 Tower root: Threshold Used: no
Feb  1 09:10:42 Tower root: "0
Feb  1 09:10:42 Tower root:  0
Feb  1 09:10:42 Tower root: 80
Feb  1 09:10:42 Tower root: es
Feb  1 09:10:42 Tower root: no
Feb  1 09:10:42 Tower root: "1
Feb  1 09:10:42 Tower root: no
Feb  1 09:10:42 Tower root: "1
Feb  1 09:10:42 Tower root: no
Feb  1 09:10:42 Tower root: ="
Feb  1 09:10:42 Tower root: no
Feb  1 09:10:42 Tower root: ="
Feb  1 09:10:42 Tower root: es
Feb  1 09:10:42 Tower root: no
Feb  1 09:10:42 Tower root: no
Feb  1 09:10:42 Tower root: no
Feb  1 09:10:42 Tower root: ="
Feb  1 09:10:42 Tower root: es
Feb  1 09:10:42 Tower root: 10

 

This same block of log messages displays for every share in my cache.  If something IS moved from cache to array for that share, the "integer expression expected" message doesn't show up.  

 

The mover appears to be working correctly, so I just wanted to share in case this isn't expected.  I am using cache % used (80%) and file age (10 days) in my Mover settings, and I am on RC2.  Thanks!

Do you still have a Photos share?
ls -ltr /boot/config/shares

 

Seems an upgrade to RC2 may have left some orphan share config files.  I will add some checks in the next release.

Link to comment
Just now, hugenbdd said:

Do you still have a Photos share?
ls -ltr /boot/config/shares

 

Seems an upgrade to RC2 may have left some orphan share config files.  I will add some checks in the next release.

 

Interesting, there are some shares listed in my /boot/config/shares that were long deleted from the system.  Can I just delete the orphaned .cfg files from that shares file?

Link to comment
12 minutes ago, Andiroo2 said:

 

Interesting, there are some shares listed in my /boot/config/shares that were long deleted from the system.  Can I just delete the orphaned .cfg files from that shares file?

I believe you can, based on other users experiences. (page 12 and 13)

 

You may want to move them to a temporary directory just in case they may be needed.

Link to comment

Hi people,

I seem to be having issue getting the 'Ignore files listed inside of a text file' process to work.


My moverignore file is on /boot/moverignore.txt and contains the following:

/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog1.edb
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog1.edb

 

 

However, invoking mover moves all those files, with the following log:

Feb 4 10:11:12 User-HTPC root: Complete Mover Command: find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/moverignore.txt' | /usr/local/sbin/move -d 1
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog1.edb
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog2.edb
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC root: mover: finished

 

Anything obvious I am doing wrong?

Link to comment
22 hours ago, deanpelton said:

Hi people,

I seem to be having issue getting the 'Ignore files listed inside of a text file' process to work.


My moverignore file is on /boot/moverignore.txt and contains the following:

/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog1.edb
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog1.edb

 

 

However, invoking mover moves all those files, with the following log:

Feb 4 10:11:12 User-HTPC root: Complete Mover Command: find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/moverignore.txt' | /usr/local/sbin/move -d 1
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog1.edb
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog2.edb
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC root: mover: finished

 

Anything obvious I am doing wrong?

Few things.

There might be a hidden ^M character in your exclusion file if you created/edited it in windows.  Either remove inside vi editor, try dos2unix (may not work in unraid), or  sed -e "s/^M//" filename > newfilename

 

Try just the find command on it's own from the console and see what it returns. (It should not have the files in the exclude file in the output).

find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/moverignore.txt'

  • Like 1
Link to comment
On 2/5/2021 at 8:56 AM, hugenbdd said:

Few things.

There might be a hidden ^M character in your exclusion file if you created/edited it in windows.  Either remove inside vi editor, try dos2unix (may not work in unraid), or  sed -e "s/^M//" filename > newfilename

 

Try just the find command on it's own from the console and see what it returns. (It should not have the files in the exclude file in the output).

find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/moverignore.txt'

So I ran the script to remove any ^M, (double checked in nano and couldn't see any anyway), ran the find with grep exclusion and still got the following result:

 

root@User-HTPC:~# find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/config/plugins/ca.mover.tuning/moverignore2.txt'
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration
/mnt/cache/Backup/User/User-COMPANY-LAP
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration
/mnt/cache/Backup/User/User-COMPANY-LAP (3)
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog1.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration
/mnt/cache/Backup/User/User-LAPTOP-XPS
/mnt/cache/Backup/

 

The moverignore file contains the following:

image.png.5f36775b25e12b9fa10455acc4b4e918.png

 

Can't figure out why? The Grep command should ignore all of these right, given the -v flag inverts matching?

Link to comment
  • 2 weeks later...
13 minutes ago, binzhu1070 said:

can someone please tell me how I can install this plugin? searching in the communication application doesn't show this for me. I do have an ssd installed as the cache drive.

 

What version of Unraid are you running?  This can affect whether the plugin is available to install via the Apps tab.

Link to comment
23 minutes ago, binzhu1070 said:

Version: 6.8.3  I just installed this about a week ago

 

 

 

the latest version of the plugin is no longer compatible with 6.8.3 as it has been updated for 6.9.0-rc2 compatibility.   If you look back a bit in this thread you will find details on how to install an older version that is compatible.

Link to comment

Found it, thank you! so just to make sure as I'm new to this, I want to setup to move hourly if cache is over 75%, I just need to set the regular mover schedue to hourly and set this plugin to 75% on the threshold and leave everything else as is right?

Edited by binzhu1070
Link to comment
59 minutes ago, binzhu1070 said:

Found it, thank you! so just to make sure as I'm new to this, I want to setup to move hourly if cache is over 75%, I just need to set the regular mover schedue to hourly and set this plugin to 75% on the threshold and leave everything else as is right?


Why is your cache getting that full in the first place?   Since you can always write to the cache much faster than the mover can transfer files to the array you should really have enough space on the cache to handle a day or so worth of files.    Running it hourly may not help if you are putting too much on the cache.   Mover is really designed to be run less frequently in an idle period (e.g. overnight).    Having said that since I do not know the details of your Use Case maybe an hourly schedule will be appropriate.

Link to comment
  • 2 weeks later...

Just upgraded to 6.9 and subsequently upgraded the mover tuning plugin to latest.  Now it seems as if the plugin isn't used at all when the mover is invoked.  Everything gets moved when the mover is invoked even though I have the used space threshold set to 70%.  I see that there are still some debugging echo statements in age_mover script and none of these show up in the syslog (I have mover logging enabled).  I uninstalled and reinstalled the plugin and still just seems to use the base mover.

Link to comment

Working on an update.  Should have something released tonight.

 

If your share config file does not have a "shareCachePool=cache" or another cache pool entry, then it was failing.  Temporary fix would be to change each share to "none" than back to "cache", and hit the apply button, so the entry gets put into the config file.  My update will set default cache name if it's missing in the config.

 

brentdog, will circle back after the release tonight.

Link to comment
On 2/11/2021 at 7:35 PM, deanpelton said:

So I ran the script to remove any ^M, (double checked in nano and couldn't see any anyway), ran the find with grep exclusion and still got the following result:

 

root@User-HTPC:~# find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/config/plugins/ca.mover.tuning/moverignore2.txt'
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration
/mnt/cache/Backup/User/User-COMPANY-LAP
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration
/mnt/cache/Backup/User/User-COMPANY-LAP (3)
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog1.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration
/mnt/cache/Backup/User/User-LAPTOP-XPS
/mnt/cache/Backup/

 

The moverignore file contains the following:

image.png.5f36775b25e12b9fa10455acc4b4e918.png

 

Can't figure out why? The Grep command should ignore all of these right, given the -v flag inverts matching?

 

I tried to recreate with your Config1.xml path. (Minor change, I use Backups for my sharename..)

 

Can't reproduce it.  Maybe try my file ( I changed to Backup, to match your path), but place it in a different area? (/usr/local/emhttp/plugins/ca.mover.tuning/moverignore2.txt)

 

 

moverignore2.txt

Link to comment
44 minutes ago, hugenbdd said:

 

I tried to recreate with your Config1.xml path. (Minor change, I use Backups for my sharename..)

 

Can't reproduce it.  Maybe try my file ( I changed to Backup, to match your path), but place it in a different area? (/usr/local/emhttp/plugins/ca.mover.tuning/moverignore2.txt)

 

 

moverignore2.txt 676 B · 0 downloads

I hadn't considered that the location could be causing the problem!!

Will try moving it to usr/ does this directoy persist after reboot?

 

Will poke around today. Have also upgraded both Mover plugin and unraid OS, so a lot of variables have changed.

Link to comment
On 3/4/2021 at 1:05 AM, deanpelton said:

I hadn't considered that the location could be causing the problem!!

Will try moving it to usr/ does this directoy persist after reboot?

 

Will poke around today. Have also upgraded both Mover plugin and unraid OS, so a lot of variables have changed.

Yes that directory persists.

Link to comment
On 2/23/2021 at 10:39 AM, itimpi said:


Why is your cache getting that full in the first place?   Since you can always write to the cache much faster than the mover can transfer files to the array you should really have enough space on the cache to handle a day or so worth of files.    Running it hourly may not help if you are putting too much on the cache.   Mover is really designed to be run less frequently in an idle period (e.g. overnight).    Having said that since I do not know the details of your Use Case maybe an hourly schedule will be appropriate.

Yeah, I was getting confused with the yes/no/preferred options, now I think I get it, changing around the configs and now there are only some docker files on the cache drive.

 

So yeah you are right I definitely don't need it to run hourly, however the point of the tuner plugin is so that if it ever gets to 75% full mover will run in an hour without wait till the night right? so that's how I want to setup, using this tuner. Without it I probably just set it to run nighly. is my understanding correct?  

Link to comment

Hey - great plugin, much appreciated for all the work from both @Squid and @hugenbdd.

 

I wanted the ability to move files based on minimum number of links. My use case is that I hard link all torrent files downloaded, and want to keep seeding torrents (and their hard linked media) on my cache pool. In combination with the exclude file list (adding the directory my torrents download to) any media file with less than 2 links will be moved to the array. 

 

I made the necessary adjustments to add this functionality and have attached patch here in the event that you want to incorporate this into the plugin. Note that the lowest setting is '2' links, since all files have at least 1 link. Setting to 1 would therefore find no eligible files to move.

 

movertuning.patch

Edited by rinseaid
  • Like 1
Link to comment
6 hours ago, Squid said:

Multiple reports of this:

 

 

Root Cause: Bash code didn't like the space in a share name.   "TV Shows".

Fix: updated code to handle spaces in the share name.

Added: Check to see that the default "/mnt/cache" the codes uses if it cant find sharepoolcachename in the config file is empty.  it will "exit" the mover try for that specific share.

 

New Release available.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.