[Plugin] Mover Tuning


Recommended Posts

51 minutes ago, AeonLucid said:

Just found this plugin and it looks great, has anyone tried it on 6.10.0-rc2? Will it work?

Also, am I seeing it right that this is not "tuning" the original mover but a replacement of it?

Mostly a replacement.  It does in some circumstances call the original mover.  There is a "test" mode, so that you can see what your selections will move, as long as logging is enabled.

 

I have not tested/built with 6.10 yet as I'm extremely busy with work and my test box is not up and running.

Link to comment
On 2/19/2022 at 2:20 PM, itimpi said:

It will be nice to get some confirmations that it works OK outside my test environments.

 

i will try to check it this week ,if not i will check it for sure 1.3.22 when my parity will start by sheduler.

(im not really like this parity checks becose of a lot of time of all HDDs usage) 🙂

Link to comment
5 minutes ago, Masterwishx said:

also whanted to ask about parity and mover:

i have mover run every 6 hours and parity start every first day at 0:00  , so they will be start like at same time ,

so its good idea to stay so or maybe to change parity to 0:30 or 23:30 ? should the new option work in this example?

The whole idea of the new option is that it does not matter if they start at the same time.   If the option is enabled (the default is Yes) then the plugins parity check code is continually monitoring whether mover is running and if it finds it is it will apply the pause, and then automatically resume when mover completes (assuming the server is still within the time slot for running a parity check increment).

 

  • Like 1
Link to comment

So every Tuesday morning, I have "Appdata Backup/Restore v2" run a backup of my appdata (among other) folders.  The initial backup is saved on my nvme/cache drive until it gets moved during the the day after successful completion.  Then, when I go to run Mover, it successfully creates the shell folder (ie. the appdata folder titled with the date of the backup) on an available array disk (with sufficient space) however, for some reason, the .tar file doesn't actually get moved.  The last few weeks, i've had to go in and move manually via MC which I'd rather not have to do.  

 

My question is why would this be occurring?  The .tar file is about 465 gb and I have 8 drives on the array with available space that can store this (anywhere from 627 gb to 2.42 TB of available space on each of the 8 drives).  

 

I was able to copy the log here when mover ran:  

 

mvlogger: Log Level: 1
mvlogger: *********************************MOVER -SHARE- START*******************************
mvlogger: Wed Feb 23 01:41:27 EST 2022
mvlogger: Share supplied backups
Sharecfg: /boot/config/shares/backups.cfg
mvlogger: Cache Pool Name: cache_nvme
mvlogger: Share Path: /mnt/cache_nvme/backups
mvlogger: Complete Mover Command: find "/mnt/cache_nvme/backups" -depth | /usr/local/sbin/move -d 1
Feb 23 01:41:27 HawkinsUnraid move: move: skip /mnt/cache_nvme/backups/appdata backup/[email protected]/CA_backup.tar
mvlogger: Wed Feb 23 01:41:27 EST 2022
mvlogger: ********************************Mover Finished*****************************

 

What would be causing the skip?  i assume it has something to do with the Mover command directly above the skip command; however, I cannot discern why it wouldn't move.  In this particular case, mover created the shell folder "[email protected]:00" on my disk 27 which has 2.42TB of space remaining; however, it doesn't actually move the .tar file associated with that folder.  

 

Thanks!

Link to comment

Ignored file types are not working.

 

I see this has been an issue before and was seemingly fixed. I am experiencing this issue again though. I've tried both ".jpg,.jpeg" and "jpg,jpeg" for the rule. Sadly, I cannot add a syslog as my syslog is spitting out write permission errors; the log partition seems to be 0KB in size; sadly, I was unable to resolve this with any of the posts of people having similar issues.

rsyslogd: file '/boot/logs/syslog'[6] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: File too large [v8.2002.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2002.0 try https://www.rsyslog.com/e/2027 ]

 

I have included my plugin settings.

 

Running Unraid 6.9.2 with the plugin version 2021.11.28.

mover settings.PNG

Link to comment
On 2/23/2022 at 9:42 AM, Hawkins12 said:

So every Tuesday morning, I have "Appdata Backup/Restore v2" run a backup of my appdata (among other) folders.  The initial backup is saved on my nvme/cache drive until it gets moved during the the day after successful completion.  Then, when I go to run Mover, it successfully creates the shell folder (ie. the appdata folder titled with the date of the backup) on an available array disk (with sufficient space) however, for some reason, the .tar file doesn't actually get moved.  The last few weeks, i've had to go in and move manually via MC which I'd rather not have to do.  

 

My question is why would this be occurring?  The .tar file is about 465 gb and I have 8 drives on the array with available space that can store this (anywhere from 627 gb to 2.42 TB of available space on each of the 8 drives).  

 

I was able to copy the log here when mover ran:  

 

mvlogger: Log Level: 1
mvlogger: *********************************MOVER -SHARE- START*******************************
mvlogger: Wed Feb 23 01:41:27 EST 2022
mvlogger: Share supplied backups
Sharecfg: /boot/config/shares/backups.cfg
mvlogger: Cache Pool Name: cache_nvme
mvlogger: Share Path: /mnt/cache_nvme/backups
mvlogger: Complete Mover Command: find "/mnt/cache_nvme/backups" -depth | /usr/local/sbin/move -d 1
Feb 23 01:41:27 HawkinsUnraid move: move: skip /mnt/cache_nvme/backups/appdata backup/[email protected]/CA_backup.tar
mvlogger: Wed Feb 23 01:41:27 EST 2022
mvlogger: ********************************Mover Finished*****************************

 

What would be causing the skip?  i assume it has something to do with the Mover command directly above the skip command; however, I cannot discern why it wouldn't move.  In this particular case, mover created the shell folder "[email protected]:00" on my disk 27 which has 2.42TB of space remaining; however, it doesn't actually move the .tar file associated with that folder.  

 

Thanks!

I believe this is because something still has the file locked.  That log entry is from Unraids binary mover file.

 

How long after the backup until mover kicks off.  Could the backup still be writing the file from RAM or write cache to the disks and still has a lock on it?

Link to comment
59 minutes ago, DesertCookie said:

Ignored file types are not working.

 

I see this has been an issue before and was seemingly fixed. I am experiencing this issue again though. I've tried both ".jpg,.jpeg" and "jpg,jpeg" for the rule. Sadly, I cannot add a syslog as my syslog is spitting out write permission errors; the log partition seems to be 0KB in size; sadly, I was unable to resolve this with any of the posts of people having similar issues.

rsyslogd: file '/boot/logs/syslog'[6] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: File too large [v8.2002.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2002.0 try https://www.rsyslog.com/e/2027 ]

 

I have included my plugin settings.

 

Running Unraid 6.9.2 with the plugin version 2021.11.28.

mover settings.PNG

Mostly a guess as Logs are what will pinpoint what is happening.

 

Move All from cache is set very low.  5%...  Could be that your cache is filling up above 5% and that means it's running the original mover command of running all files on the cache drives. I would suggest that be 70% or higher.  It was originally meant for as a "fail safe" so that if someone is filling their cache up, it catches an almost full cache and just empties it over to the array.

 

Link to comment
4 hours ago, hugenbdd said:

Move All from cache is set very low.  5%...

 

I'll try disabling this setting for the moment. I have misunderstood the option based on the description; I assumed it would move all data when only 5% space were remaining.

Maybe it would make more sense to inverse the sorting of this list so that the more recommended options are at the top.

 

Edit: It still moves JPG and JPEG files.

 

Edit 2: Finally got some logs to share. Here it logs the moving of the image files I keep placing on the cache for testing purposes:

Feb 26 21:07:40 STOWER20 move: move: file /mnt/cache/Jellyfin/Serien/EN/Adam Ruins Everything (2015-2019)/Staffel 1 (2015-2016)/folder.jpg
Feb 26 21:07:40 STOWER20 move: move: file /mnt/cache/Jellyfin/Serien/EN/Adam Ruins Everything (2015-2019)/Staffel 2 (2017-2018)/folder.jpg
Feb 26 21:07:41 STOWER20 move: move: file /mnt/cache/Jellyfin/Serien/EN/Adam Ruins Everything (2015-2019)/Staffel 3 (2019)/folder.jpg
Feb 26 21:07:41 STOWER20 move: move: file /mnt/cache/Jellyfin/Serien/EN/Adam Ruins Everything (2015-2019)/Specials (2016)/folder.jpg
Feb 26 21:07:41 STOWER20 move: move: file /mnt/cache/Jellyfin/Serien/EN/Adam Ruins Everything (2015-2019)/folder.jpg

 

Edited by DesertCookie
Link to comment
2 hours ago, hugenbdd said:

I believe this is because something still has the file locked.  That log entry is from Unraids binary mover file.

 

How long after the backup until mover kicks off.  Could the backup still be writing the file from RAM or write cache to the disks and still has a lock on it?

I am not sure -- i suspected it might be that as well but it certainly isn't running the backup still.  I don't auto-run "Mover" so it's whenever I feel the need to hit the button.  Usually its later in the day (12+ hrs after Appdata backup) so it def. is still not writing.  I even tried the next day (and the next) and still same issue. 

 

That being said --  I have Crashplan running on unraid and it's set to auto backup the folder which this .tar is stored in.  Perhaps it is uploading this to Crashplan which is limiting the ability to backup?  I ran mover on Thursday and got the same "skip" notification.  I ran mover last night and it actually moved it....odd....

Link to comment

I need to ignore move of couple folders from a Share (rest of the Share must be moved daily).

Full details in this thread.

Quick example:

/mnt/user/MyShareName/newsShow1 contains:

ShowName1 SXXEXX YYYY-MM-DD.mkv

/mnt/user/MyShareName/newsShow2 contains:

ShowName2 SXXEXX YYYY-MM-DD.mkv

 

I see "Ignore files listed inside of a text file", will this work with a wild card (*) such as:

/mnt/user/MyShareName/newsChannel2/ShowName1*.mkv

Or even better, anything that has "news" in the file path?

Edited by OrdinaryButt
Link to comment

I may have found a bug, or user error. Either way I could use some assistance.

 

I am trying to use this plugin to do the following cases:

- Move files every 6 days

or

- Move all from cache yes if the cache exceeds 50% of drive space

 

The problem is when it hits the default case of "move all from cache" due to exceeding space, it fails to handle a share with a space in the name. The relevant logs are:

Mar  8 09:57:10 Tower root: mvlogger: CACHETHRESH: 50
Mar  8 09:57:10 Tower root: mvlogger: Checking to see if we should all files from share.
Mar  8 09:57:10 Tower root: mvlogger: Move All Cache Threshold: 50
Mar  8 09:57:10 Tower root: mvlogger: /mnt/cache PCT USED:  87
Mar  8 09:57:10 Tower root: mvlogger: Pool /mnt/cache is above 'move all' percentage, moving all cache-yes shares to array
Mar  8 09:57:10 Tower root: mvlogger: Not in Test mode
Mar  8 09:57:10 Tower root: mvlogger: Complete Mover Command: find /mnt/cache/TV Shows -depth | /usr/local/sbin/move -d 1
Mar  8 09:57:10 Tower root: find: '/mnt/cache/TV': No such file or directory
Mar  8 09:57:10 Tower root: find: 'Shows': No such file or directory
Mar  8 09:57:10 Tower root: mvlogger: Done checking Move All from Share

 

I am new to unRAID so am I misunderstanding these features or trying to implement them incorrectly?

Link to comment
On 3/5/2022 at 1:45 PM, OrdinaryButt said:

I need to ignore move of couple folders from a Share (rest of the Share must be moved daily).

Full details in this thread.

Quick example:

/mnt/user/MyShareName/newsShow1 contains:

ShowName1 SXXEXX YYYY-MM-DD.mkv

/mnt/user/MyShareName/newsShow2 contains:

ShowName2 SXXEXX YYYY-MM-DD.mkv

 

I see "Ignore files listed inside of a text file", will this work with a wild card (*) such as:

/mnt/user/MyShareName/newsChannel2/ShowName1*.mkv

Or even better, anything that has "news" in the file path?

I would suggest setting trying your setting and using the "test" option.  Then review the logs to see if the files are moved, or are not moved based on what you want.

Link to comment
9 hours ago, croudebush said:

I may have found a bug, or user error. Either way I could use some assistance.

 

I am trying to use this plugin to do the following cases:

- Move files every 6 days

or

- Move all from cache yes if the cache exceeds 50% of drive space

 

The problem is when it hits the default case of "move all from cache" due to exceeding space, it fails to handle a share with a space in the name. The relevant logs are:

Mar  8 09:57:10 Tower root: mvlogger: CACHETHRESH: 50
Mar  8 09:57:10 Tower root: mvlogger: Checking to see if we should all files from share.
Mar  8 09:57:10 Tower root: mvlogger: Move All Cache Threshold: 50
Mar  8 09:57:10 Tower root: mvlogger: /mnt/cache PCT USED:  87
Mar  8 09:57:10 Tower root: mvlogger: Pool /mnt/cache is above 'move all' percentage, moving all cache-yes shares to array
Mar  8 09:57:10 Tower root: mvlogger: Not in Test mode
Mar  8 09:57:10 Tower root: mvlogger: Complete Mover Command: find /mnt/cache/TV Shows -depth | /usr/local/sbin/move -d 1
Mar  8 09:57:10 Tower root: find: '/mnt/cache/TV': No such file or directory
Mar  8 09:57:10 Tower root: find: 'Shows': No such file or directory
Mar  8 09:57:10 Tower root: mvlogger: Done checking Move All from Share

 

I am new to unRAID so am I misunderstanding these features or trying to implement them incorrectly?

Yup, that looks like an error.  It should have quotes around it.  I'll review the code soon and try to get a bug fix out.

Link to comment

Reporting a bug:

Ignore file types will ignore any file name that also ends with the same letters as the ignored parameter.

Example, ignore ".ts" does not move a file with name "Bets.mp4". Same goes for "ts" filter. Dozens of files ending with "-ts.extension" didn't move.

 

I will only provide logs, if you can't replicate it on your end.

Link to comment
  • 2 weeks later...

This plugin seems to be ignoring the 'Script to run before mover' and 'Script to run after mover' fields.

 

I've them both filled pointing to /mnt/user/system/docker/dockerstop.sh

and /mnt/user/system/docker/dockerstart.sh

 

dockstop.sh:

/etc/rc.d/rc.docker stop

dockerstart.sh:

/etc/rc.d/rc.docker start

 

Please advice on why these might be getting ignored.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.