Hi people,

I seem to be having issue getting the 'Ignore files listed inside of a text file' process to work.

My moverignore file is on /boot/moverignore.txt and contains the following:

/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog1.edb
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog1.edb

 

 

However, invoking mover moves all those files, with the following log:

Feb 4 10:11:12 User-HTPC root: Complete Mover Command: find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/moverignore.txt' | /usr/local/sbin/move -d 1
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog1.edb
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog2.edb
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC root: mover: finished

 

Anything obvious I am doing wrong?

22 hours ago, deanpelton said:

Hi people,

I seem to be having issue getting the 'Ignore files listed inside of a text file' process to work.

My moverignore file is on /boot/moverignore.txt and contains the following:

/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog1.edb
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog1.edb

 

 

However, invoking mover moves all those files, with the following log:

Feb 4 10:11:12 User-HTPC root: Complete Mover Command: find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/moverignore.txt' | /usr/local/sbin/move -d 1
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
Feb 4 10:11:12 User-HTPC move: move: file /mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog1.edb
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Catalog2.edb
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
Feb 4 10:11:13 User-HTPC move: move: file /mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
Feb 4 10:11:13 User-HTPC root: mover: finished

 

Anything obvious I am doing wrong?

Few things.

There might be a hidden ^M character in your exclusion file if you created/edited it in windows.  Either remove inside vi editor, try dos2unix (may not work in unraid), or  sed -e "s/^M//" filename > newfilename

 

Try just the find command on it's own from the console and see what it returns. (It should not have the files in the exclude file in the output).

find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/moverignore.txt'

Is this release (2021.01.08) ok to be installed on 6.8 or no? 

31 minutes ago, cbr600ds2 said:

Is this release (2021.01.08) ok to be installed on 6.8 or no? 

No, it will "error" out if you try to install that release on 6.8 as it checks the version now.

 

For 6.8, follow this post.

 

4 hours ago, hugenbdd said:

No, it will "error" out if you try to install that release on 6.8 as it checks the version now.

 

For 6.8, follow this post.

 

 

Ahhh...ok.  I'm still on the old release so I was wondering.  Thanks. 

On 2/5/2021 at 8:56 AM, hugenbdd said:

Few things.

There might be a hidden ^M character in your exclusion file if you created/edited it in windows.  Either remove inside vi editor, try dos2unix (may not work in unraid), or  sed -e "s/^M//" filename > newfilename

 

Try just the find command on it's own from the console and see what it returns. (It should not have the files in the exclude file in the output).

find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/moverignore.txt'

So I ran the script to remove any ^M, (double checked in nano and couldn't see any anyway), ran the find with grep exclusion and still got the following result:

 

root@User-HTPC:~# find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/config/plugins/ca.mover.tuning/moverignore2.txt'
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration
/mnt/cache/Backup/User/User-COMPANY-LAP
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration
/mnt/cache/Backup/User/User-COMPANY-LAP (3)
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog1.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration
/mnt/cache/Backup/User/User-LAPTOP-XPS
/mnt/cache/Backup/

 

The moverignore file contains the following:

 

Can't figure out why? The Grep command should ignore all of these right, given the -v flag inverts matching?

can someone please tell me how I can install this plugin? searching in the communication application doesn't show this for me. I do have an ssd installed as the cache drive.

13 minutes ago, binzhu1070 said:

can someone please tell me how I can install this plugin? searching in the communication application doesn't show this for me. I do have an ssd installed as the cache drive.

 

What version of Unraid are you running?  This can affect whether the plugin is available to install via the Apps tab.

Version: 6.8.3  I just installed this about a week ago

 

 

23 minutes ago, binzhu1070 said:

Version: 6.8.3  I just installed this about a week ago

 

 

 

the latest version of the plugin is no longer compatible with 6.8.3 as it has been updated for 6.9.0-rc2 compatibility.   If you look back a bit in this thread you will find details on how to install an older version that is compatible.

Found it, thank you! so just to make sure as I'm new to this, I want to setup to move hourly if cache is over 75%, I just need to set the regular mover schedue to hourly and set this plugin to 75% on the threshold and leave everything else as is right?

Edited February 23, 20215 yr by binzhu1070

59 minutes ago, binzhu1070 said:

Found it, thank you! so just to make sure as I'm new to this, I want to setup to move hourly if cache is over 75%, I just need to set the regular mover schedue to hourly and set this plugin to 75% on the threshold and leave everything else as is right?

Why is your cache getting that full in the first place?   Since you can always write to the cache much faster than the mover can transfer files to the array you should really have enough space on the cache to handle a day or so worth of files.    Running it hourly may not help if you are putting too much on the cache.   Mover is really designed to be run less frequently in an idle period (e.g. overnight).    Having said that since I do not know the details of your Use Case maybe an hourly schedule will be appropriate.

Just upgraded to 6.9 and subsequently upgraded the mover tuning plugin to latest.  Now it seems as if the plugin isn't used at all when the mover is invoked.  Everything gets moved when the mover is invoked even though I have the used space threshold set to 70%.  I see that there are still some debugging echo statements in age_mover script and none of these show up in the syslog (I have mover logging enabled).  I uninstalled and reinstalled the plugin and still just seems to use the base mover.

Working on an update.  Should have something released tonight.

 

If your share config file does not have a "shareCachePool=cache" or another cache pool entry, then it was failing.  Temporary fix would be to change each share to "none" than back to "cache", and hit the apply button, so the entry gets put into the config file.  My update will set default cache name if it's missing in the config.

 

brentdog, will circle back after the release tonight.

Version: 2021.03.03 

 

Added a check as described above.  Please update and let me know if you have any issues.  I still have some plans to clean up some new potential issues.

On 2/11/2021 at 7:35 PM, deanpelton said:

So I ran the script to remove any ^M, (double checked in nano and couldn't see any anyway), ran the find with grep exclusion and still got the following result:

 

root@User-HTPC:~# find "/mnt/cache/Backup/" -depth | grep -vFf '/boot/config/plugins/ca.mover.tuning/moverignore2.txt'
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP/Configuration
/mnt/cache/Backup/User/User-COMPANY-LAP
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config1.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration/Config2.xml
/mnt/cache/Backup/User/User-COMPANY-LAP (3)/Configuration
/mnt/cache/Backup/User/User-COMPANY-LAP (3)
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog1.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Catalog2.edb
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config1.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration/Config2.xml
/mnt/cache/Backup/User/User-LAPTOP-XPS/Configuration
/mnt/cache/Backup/User/User-LAPTOP-XPS
/mnt/cache/Backup/

 

The moverignore file contains the following:

 

Can't figure out why? The Grep command should ignore all of these right, given the -v flag inverts matching?

 

I tried to recreate with your Config1.xml path. (Minor change, I use Backups for my sharename..)

 

Can't reproduce it.  Maybe try my file ( I changed to Backup, to match your path), but place it in a different area? (/usr/local/emhttp/plugins/ca.mover.tuning/moverignore2.txt)

 

 

moverignore2.txt

44 minutes ago, hugenbdd said:

 

I tried to recreate with your Config1.xml path. (Minor change, I use Backups for my sharename..)

 

Can't reproduce it.  Maybe try my file ( I changed to Backup, to match your path), but place it in a different area? (/usr/local/emhttp/plugins/ca.mover.tuning/moverignore2.txt)

 

 

moverignore2.txt 676 B · 0 downloads

I hadn't considered that the location could be causing the problem!!

Will try moving it to usr/ does this directoy persist after reboot?

 

Will poke around today. Have also upgraded both Mover plugin and unraid OS, so a lot of variables have changed.

On 3/4/2021 at 1:05 AM, deanpelton said:

I hadn't considered that the location could be causing the problem!!

Will try moving it to usr/ does this directoy persist after reboot?

 

Will poke around today. Have also upgraded both Mover plugin and unraid OS, so a lot of variables have changed.

Yes that directory persists.

On 2/23/2021 at 10:39 AM, itimpi said:

Why is your cache getting that full in the first place?   Since you can always write to the cache much faster than the mover can transfer files to the array you should really have enough space on the cache to handle a day or so worth of files.    Running it hourly may not help if you are putting too much on the cache.   Mover is really designed to be run less frequently in an idle period (e.g. overnight).    Having said that since I do not know the details of your Use Case maybe an hourly schedule will be appropriate.

Yeah, I was getting confused with the yes/no/preferred options, now I think I get it, changing around the configs and now there are only some docker files on the cache drive.

 

So yeah you are right I definitely don't need it to run hourly, however the point of the tuner plugin is so that if it ever gets to 75% full mover will run in an hour without wait till the night right? so that's how I want to setup, using this tuner. Without it I probably just set it to run nighly. is my understanding correct?  

Hey - great plugin, much appreciated for all the work from both @Squid and @hugenbdd.

 

I wanted the ability to move files based on minimum number of links. My use case is that I hard link all torrent files downloaded, and want to keep seeding torrents (and their hard linked media) on my cache pool. In combination with the exclude file list (adding the directory my torrents download to) any media file with less than 2 links will be moved to the array. 

 

I made the necessary adjustments to add this functionality and have attached patch here in the event that you want to incorporate this into the plugin. Note that the lowest setting is '2' links, since all files have at least 1 link. Setting to 1 would therefore find no eligible files to move.

 

movertuning.patch

Edited March 6, 20215 yr by rinseaid

Multiple reports of this:

 

 

6 hours ago, Squid said:

Multiple reports of this:

 

 

Root Cause: Bash code didn't like the space in a share name.   "TV Shows".

Fix: updated code to handle spaces in the share name.

Added: Check to see that the default "/mnt/cache" the codes uses if it cant find sharepoolcachename in the config file is empty.  it will "exit" the mover try for that specific share.

 

New Release available.

On 3/3/2021 at 12:42 PM, brentdog said:

Just upgraded to 6.9 and subsequently upgraded the mover tuning plugin to latest.  Now it seems as if the plugin isn't used at all when the mover is invoked.  Everything gets moved when the mover is invoked even though I have the used space threshold set to 70%.  I see that there are still some debugging echo statements in age_mover script and none of these show up in the syslog (I have mover logging enabled).  I uninstalled and reinstalled the plugin and still just seems to use the base mover.

I had some time to try to debug this myself today.  Turns out you need one of the yes/no criteria (age, min size, etc.) set or the base mover is used.  This is different than previous versions of the plugin where you could just set the percent threshold and it would move everything when you hit that threshold.  I set the age limit to something reasonable and it is working as expected now.

 

Just curious if this change was intentional?

14 minutes ago, brentdog said:

I had some time to try to debug this myself today.  Turns out you need one of the yes/no criteria (age, min size, etc.) set or the base mover is used.  This is different than previous versions of the plugin where you could just set the percent threshold and it would move everything when you hit that threshold.  I set the age limit to something reasonable and it is working as expected now.

 

Just curious if this change was intentional?

I don't think it was. I'll review the test case and try to reproduce tonight or tomorrow.

11 hours ago, brentdog said:

I had some time to try to debug this myself today.  Turns out you need one of the yes/no criteria (age, min size, etc.) set or the base mover is used.  This is different than previous versions of the plugin where you could just set the percent threshold and it would move everything when you hit that threshold.  I set the age limit to something reasonable and it is working as expected now.

 

Just curious if this change was intentional?

It's been that way for as long as I can remember, but then again I've been on some form of 6.9rc for as long as I can remember...ha.