On 3/6/2021 at 1:05 PM, rinseaid said:

Hey - great plugin, much appreciated for all the work from both @Squid and @hugenbdd.

 

I wanted the ability to move files based on minimum number of links. My use case is that I hard link all torrent files downloaded, and want to keep seeding torrents (and their hard linked media) on my cache pool. In combination with the exclude file list (adding the directory my torrents download to) any media file with less than 2 links will be moved to the array. 

 

I made the necessary adjustments to add this functionality and have attached patch here in the event that you want to incorporate this into the plugin. Note that the lowest setting is '2' links, since all files have at least 1 link. Setting to 1 would therefore find no eligible files to move.

 

movertuning.patch 5.49 kB · 1 download

Haven't forgotten about this.  Will circle back in a few weeks once things settle down with the new release and try and add this in.

3 minutes ago, Glasti said:

Thank you for the info, this makes the Mover button in the `Main` tab obsolete. No big deal for me. 
Would it be possible to add multiple cron jobs? So you can force move multiple times a day if necessary?

Should not be an issue.  adding to cron is a bit different in unRaid.  I can't remember the way they recommend.  However, you would want to add something like /usr/local/sbin/mover.old start

 

Maybe someone else on the thread can give you more specific cron instructions.

8 minutes ago, hugenbdd said:

Should not be an issue.  adding to cron is a bit different in unRaid.  I can't remember the way they recommend.  However, you would want to add something like /usr/local/sbin/mover.old start

 

Maybe someone else on the thread can give you more specific cron instructions.

Add the command to the user scripts plugin with a cron setting.

8 minutes ago, hugenbdd said:

Should not be an issue.  adding to cron is a bit different in unRaid.  I can't remember the way they recommend.  However, you would want to add something like /usr/local/sbin/mover.old start

 

Maybe someone else on the thread can give you more specific cron instructions.

I will poke around and see if it is possible to either setup multiple cron jobs OR add `/usr/local/sbin/mover.old start` to the mover button. I will post it here if i find a solution! Thank you for your work!

After a recent update of Mover Tuning, mover (or something else) is emailing me an hourly mover log. I have been receiving emails from my server for big things like parity checks and errors. Is there a mover setting causing this?

8 hours ago, rbronco21 said:

After a recent update of Mover Tuning, mover (or something else) is emailing me an hourly mover log. I have been receiving emails from my server for big things like parity checks and errors. Is there a mover setting causing this?

I’m having the same problem except my emails are only daily.

1 hour ago, wgstarks said:

I’m having the same problem except my emails are only daily.

It's because I left ECHO statements in the script.  These help me catch bugs for the new release.

 

I will release a new version in the next few days that will comment out the echo statements.

03/17/2021 - New Release

Add's an "Original Move Now" button.  (Bottom right of the "donate" buttons"

removed echo statements.

 

I may add a button in the future to enable the button by a checkbox in the config settings.

19 minutes ago, hugenbdd said:

Bottom right of the "donate" buttons

So this doesn’t replace the dysfunctional move button on Main?

 

Where it says “moves everything” isn’t that a bit dangerous? What happens to dockers and VM’s when appdata and images and such are moved?

48 minutes ago, wgstarks said:

So this doesn’t replace the dysfunctional move button on Main?

 

Where it says “moves everything” isn’t that a bit dangerous? What happens to dockers and VM’s when appdata and images and such are moved?

It does not replace the button/functionality on Main or on Scheduler where you set the time.  That will still look at the settings for the plug-in.

 

This is just calling the original "Move Now" code (now called mover.old under /usr/local/sbin) as if you didn't have the plug-in.  So if you share is cache only, the original move should not move it.  I added "Caution - Moves Everything" as more of a warning.  Probably could have been worded better, but didn't want to put to much in the button.   As I was just trying to give the original functionality back somehow.

5 minutes ago, hugenbdd said:

WIt does not replace the button/functionality on Main or on Scheduler where you set the time.  That will still look at the settings for the plug-in.

 

This is just calling the original "Move Now" code (now called mover.old under /usr/local/sbin) as if you didn't have the plug-in.  So if you share is cache only, the original move should not move it.  I added "Caution - Moves Everything" as more of a warning.  Probably could have been worded better, but didn't want to put to much in the button.   As I was just trying to give the original functionality back somehow.

So you’re saying that the move button on the main tab should still function? I was thinking that it wasn’t working because this plugin had disabled it.

Just now, wgstarks said:

So you’re saying that the move button on the main tab should still function? I was thinking that it wasn’t working because this plugin had disabled it.

It should still work with the plug-in settings.  It will not act as the original mover worked.  

 

I will test this afternoon as my mover is running now.

49 minutes ago, hugenbdd said:

It should still work with the plug-in settings.  It will not act as the original mover worked.  

 

I will test this afternoon as my mover is running now.

Im testing with a 20GB file. I see this in syslog-

Mar 17 15:00:24 Brunnhilde emhttpd: shcmd (211): /usr/local/sbin/mover |& logger &
Mar 17 15:00:24 Brunnhilde kernel: mdcmd (492): set md_write_method 1
Mar 17 15:00:24 Brunnhilde kernel: 
Mar 17 15:00:24 Brunnhilde root: mover: started
Mar 17 15:00:24 Brunnhilde root: mover: finished
Mar 17 15:00:24 Brunnhilde kernel: mdcmd (493): set md_write_method 1

After the move the file is still on the cache.

 

Edit: I also never get “Mover is running” displayed.

Edited March 17, 20215 yr by wgstarks

Current settings (might just be user error 😁)-

 

13 minutes ago, wgstarks said:

Im testing with a 20GB file. I see this in syslog-

Mar 17 15:00:24 Brunnhilde emhttpd: shcmd (211): /usr/local/sbin/mover |& logger &
Mar 17 15:00:24 Brunnhilde kernel: mdcmd (492): set md_write_method 1
Mar 17 15:00:24 Brunnhilde kernel: 
Mar 17 15:00:24 Brunnhilde root: mover: started
Mar 17 15:00:24 Brunnhilde root: mover: finished
Mar 17 15:00:24 Brunnhilde kernel: mdcmd (493): set md_write_method 1

After the move the file is still on the cache.

 

Edit: I also never get “Mover is running” displayed.

Not sure how the "mover is running" is supposed to work.  I do know a PID is setup, if you run mover again while it's still moving it checks for the PID, and will exit if it finds it.  That still takes place, as the PID is setup in the actual mover binary from Limetech.  I just filter what gets to the mover in the find commands.

 

This is the problem when I remove all the echo's.  It becomes really hard to find out what's going on and why a file isn't moving.  If you want to edit the age_mover file and "uncomment" some lines it would be helpful (remove the six #).  But you will have emails again....

 

 /usr/local/emhttp/plugins/ca.mover.tuning/age_mover

Example below will give us the command that is being sent to the mover.

 

                        #Add Mover to find string.
                        FINDSTR+=" | /usr/local/sbin/move -d $LOGLEVEL"

                        ######echo "Complete Mover Command: $FINDSTR"
                        eval "$FINDSTR"
 

 

Love the plugin and I'm fine with the emails if it helps out debugging. Got a couple questions.

 

1. Using tuning, I have mine check for a threshold of 50% each hour on the hour for when I copy large files to the server or download a lot to make sure I don't fill up the cache. Then using the original Mover Settings, I have it move everything every night to make sure everything is protected within 24 hours. Does this sound like a reasonable strategy? Any suggestions or issues?

 

2. I believe it worked this way last time I messed with the settings months ago. Now I don't think the daily full move from Mover Settings is working, so I only get files moved when I hit 50%, which may not happen for days or weeks of normal use, leaving files unprotected. Here are the settings I settled on when I got it running a while ago using this topic. Do I need to make any adjustments to get it working like I stated in #1, specifically the daily move?

 

 

5 hours ago, rbronco21 said:

Love the plugin and I'm fine with the emails if it helps out debugging. Got a couple questions.

 

1. Using tuning, I have mine check for a threshold of 50% each hour on the hour for when I copy large files to the server or download a lot to make sure I don't fill up the cache. Then using the original Mover Settings, I have it move everything every night to make sure everything is protected within 24 hours. Does this sound like a reasonable strategy? Any suggestions or issues?

 

2. I believe it worked this way last time I messed with the settings months ago. Now I don't think the daily full move from Mover Settings is working, so I only get files moved when I hit 50%, which may not happen for days or weeks of normal use, leaving files unprotected. Here are the settings I settled on when I got it running a while ago using this topic. Do I need to make any adjustments to get it working like I stated in #1, specifically the daily move?

 

 

 

1.) Should be nothing wrong with that.  The only possible issue you may run into, is that if your hourly move is not done when the nightly move kicks off.  The nightly move will see the PID for mover, and will just exit.  Therefore not moving your "nightly everything" request.

 

2.) This can still be done, but you have to kick off the script in cron with the renamed "original Mover" mover.old    This will mean that you are running the original "unraid" mover script.  Basically, no filters on the "find" command that my plugin creates.

 

So that means that the Mover Settings section is now ignored if Mover Tuning is enabled? If so, is adding mover.old to cron easy enough that you could show me how to do it? Then I can dig into it and learn more about cron on my own.

11 hours ago, rbronco21 said:

So that means that the Mover Settings section is now ignored if Mover Tuning is enabled? If so, is adding mover.old to cron easy enough that you could show me how to do it? Then I can dig into it and learn more about cron on my own.

no, the opposite.

 

Scheduler and mover tuning w/age section are followed.  

 

If you setup a cron job with user scripts plug-in, for the original mover, it will not follow any settings, and move files off the cache.

The script to call is below. (Original mover)

/usr/local/sbin/mover.old

 

Link to user script plug-in. (With basic instructions)

 

On 3/17/2021 at 7:25 PM, hugenbdd said:

PID is setup

The PID is just written at the start of the mover and removed at the end.

 

echo $$ >/var/run/mover.pid

 

I have submitted some changes to allow mover status to be displayed in the GUI, Pull request is in place for this and I have a new Mover script which I have sent to Limetech, Tom is also looking at the move command to allow more granular reporting as I can only do per share with the standard vers.

 

 

 

Would we need to add the changes to your mover script?

 

Hopefully once more detail is available we can make it look more like Parity check status, showing thruput/end time etc.

35 minutes ago, SimonF said:

The PID is just written at the start of the mover and removed at the end.

 

echo $$ >/var/run/mover.pid

 

I have submitted some changes to allow mover status to be displayed in the GUI, Pull request is in place for this and I have a new Mover script which I have sent to Limetech, Tom is also looking at the move command to allow more granular reporting as I can only do per share with the standard vers.

 

 

 

Would we need to add the changes to your mover script?

 

Hopefully once more detail is available we can make it look more like Parity check status, showing thruput/end time etc.

I probably would need to update the plug-in.  It all depends on where they make the change.  There are two main sections to mover.  The shell script "mover.old" and the binary "mover".  The shell script, does some basic house keeping, and sends a "find" command of the cache piped to the binary mover.  So the binary mover file/code doesn't know everything that's coming in.  Just one file at a time.

4 minutes ago, hugenbdd said:

I probably would need to update the plug-in.  It all depends on where they make the change.  There are two main sections to mover.  The shell script "mover.old" and the binary "mover".  The shell script, does some basic house keeping, and sends a "find" command of the cache piped to the binary mover.  So the binary mover file/code doesn't know everything that's coming in.  Just one file at a time.

I have only updated the shell script, will PM you the changes I sent to Tom. He is also looking to make changes to the move binary also.

 

The GUI changes have not been committed as yet, but you will be able to see the file I create.

Edited March 22, 20215 yr by SimonF

Hello!

 

Mover tuning seems to be broken after an update to 6.9.1 where it prevents the mover from moving files to the array from cache. It is still broken after re-installing the plugin. The error I get from the logs is attached. Removing the plugin fixes my issue.

 

Thank you!

Edited March 24, 20215 yr by redeuxx

5 minutes ago, redeuxx said:

Hello!

 

Mover tuning seems to be broken after an update to 6.9.1 where it prevents the mover from moving files to the array from cache. It is still broken after re-installing the plugin. The error I get from the logs is attached. Removing the plugin fixes my issue.

 

Thank you!

What version were you running?

 

Also, the Line 181 error, should have been fixed in the latest version.  But it's possible there is an error somewhere.  The Share config file does not get updated with the cache pool during an upgrade unless you change the cache pool name in the share GUI screen.  The latest release assumes your cache pool name is cache if it is not found in the config file.

2 minutes ago, hugenbdd said:

What version were you running?

 

Also, the Line 181 error, should have been fixed in the latest version.  But it's possible there is an error somewhere.  The Share config file does not get updated with the cache pool during an upgrade unless you change the cache pool name in the share GUI screen.  The latest release assumes your cache pool name is cache if it is not found in the config file.

 

I was running the latest version from CA, 2021.03.17. The name of my cache is "Cache". It was the same name before I upgraded.