[Plugin] Mover Tuning


568 posts in this topic Last Reply

Recommended Posts

  • Replies 567
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

CA Mover Tuning       A simple little plugin (mainly created for my own purposes) that will let you fine tune the operation of the mover script On scheduled runs of mover, o

I would suggest removing the plug in, then moving everything.   I'm working on getting a Dev box up and running so I can continue working on this for the new Beta.

The flash drive doesn't support executable permissions.  You have to copy the file(s) to somewhere in RAM (say /tmp) and then execute it from there (after applying executable perms to the copy)

Posted Images

So I'm having a weird issue and I've already tried reinstalling the plugin to no avail. For some reason, the plugin is reading the folder names on the cache drive incorrectly. As you can see in the attached log, the mover tries to mount /mnt/cache/STORAGE even though my share name is Storage. It also tries to mount /mnt/cache/media even though my share name is Media. This leads to the mover just not running at all. In an attempt to fix, I tried making a symbolic link for media->Media (I don't actually use Storage right now so I don't really care about that one) which is why you see media in my shares and the log says it finds it, but it still won't move any files. Please help and thanks in advance. Let me know if you need any more info or logs.

 

P.S. For generating the log on command I have the "Move Now button follows plug-in filters:" set to yes but usually have it set to no. When set to no, the mover button works as it should and all files that should be moved get moved.

MoverTuningLog.png

Shares.png

MoverTuningSettings.png

Link to post
1 hour ago, rcm024 said:

So I'm having a weird issue and I've already tried reinstalling the plugin to no avail. For some reason, the plugin is reading the folder names on the cache drive incorrectly. As you can see in the attached log, the mover tries to mount /mnt/cache/STORAGE even though my share name is Storage. It also tries to mount /mnt/cache/media even though my share name is Media. This leads to the mover just not running at all. In an attempt to fix, I tried making a symbolic link for media->Media (I don't actually use Storage right now so I don't really care about that one) which is why you see media in my shares and the log says it finds it, but it still won't move any files. Please help and thanks in advance. Let me know if you need any more info or logs.

 

P.S. For generating the log on command I have the "Move Now button follows plug-in filters:" set to yes but usually have it set to no. When set to no, the mover button works as it should and all files that should be moved get moved.

MoverTuningLog.png

Shares.png

 

Can you post what your share config files are listed as?

 

There is a mis-match somewhere between the config files (Case) and the share names.

 

ls -ltr /boot/config/shares

 

If that doesn't fix it can you post your debug file?

 

original mover loops through the directory on the cache, where I have to loop through by config files.

 

Link to post
Posted (edited)
1 hour ago, hugenbdd said:

 

Can you post what your share config files are listed as?

 

There is a mis-match somewhere between the config files (Case) and the share names.

 

ls -ltr /boot/config/shares

 

If that doesn't fix it can you post your debug file?

 

original mover loops through the directory on the cache, where I have to loop through by config files.

 

 

That was indeed the problem. Thanks! I have fixed the name of the config files and everything is working now. Thanks for the help and keep up the great work you do here!

Edited by rcm024
Link to post

I don't know if I'm the only one, but I'll ask anyways. 

 

When this Mover Script runs it spins up every drive that has a potential share that can be moved from the Cache to its place on the array including my Parity Drive. To make due with this anomaly I currently have my mover script set to run Daily at 5:45 which is 15minutes before Plex Docker scans my media looking for changes.

 

I was considering changing my mover to hourly, but notice it still spins up every drive and for me there is no reason to spin up any drives if nothing is move on an hourly basis. Kinda defeats the purpose of letting my drives spin down if they aren't needed at all. 

 

I'm trying to figure out if its something I'm doing, setting? Or are others simply not noticing or don't care? If its just what it is I'll keep my daily set to the time it is and be done with it. ;)

Link to post
20 minutes ago, kizer said:

I don't know if I'm the only one, but I'll ask anyways. 

 

When this Mover Script runs it spins up every drive that has a potential share that can be moved from the Cache to its place on the array including my Parity Drive. To make due with this anomaly I currently have my mover script set to run Daily at 5:45 which is 15minutes before Plex Docker scans my media looking for changes.

 

I was considering changing my mover to hourly, but notice it still spins up every drive and for me there is no reason to spin up any drives if nothing is move on an hourly basis. Kinda defeats the purpose of letting my drives spin down if they aren't needed at all. 

 

I'm trying to figure out if its something I'm doing, setting? Or are others simply not noticing or don't care? If its just what it is I'll keep my daily set to the time it is and be done with it. ;)

I have heard that before.   I'm not aware of any specific code to the plug-in that would spin up all drives.

 

Three thoughts of what might be causing that.

1.) Cache-Prefer settings.  This might be looking through all drives in order to move files over to the cache.

2.) Mover binary

3.) io priority settings.

 

Can you run in "test mode"?

 

Filter Settings to run in test mode.

Move Now button follows plug-in filters:     YES

Test Mode:    YES

 

Then hit "move now".

 

I just tried this, and it did not spin up my drives.

 

 

Link to post
1 hour ago, kizer said:

I don't know if I'm the only one, but I'll ask anyways. 

 

When this Mover Script runs it spins up every drive that has a potential share that can be moved from the Cache to its place on the array including my Parity Drive. To make due with this anomaly I currently have my mover script set to run Daily at 5:45 which is 15minutes before Plex Docker scans my media looking for changes.

 

I was considering changing my mover to hourly, but notice it still spins up every drive and for me there is no reason to spin up any drives if nothing is move on an hourly basis. Kinda defeats the purpose of letting my drives spin down if they aren't needed at all. 

 

I'm trying to figure out if its something I'm doing, setting? Or are others simply not noticing or don't care? If its just what it is I'll keep my daily set to the time it is and be done with it. ;)

Do you have turbo write enabled?

Link to post

This is what a Typical log looks like for me. Why it does its thing and then at the end it always does a read Smart is beyond me. I'm guessing thats what is trigger the drive spin up or its reading it after its spun up. I'm not 100% sure. I just know its annoying.

 

May 25 11:34:26 Tower root: mvlogger: Adding Age
May 25 11:34:26 Tower root: mvlogger: Age 15
May 25 11:34:26 Tower root: mvlogger: Skipfiletypes string: find "/mnt/cache/Plex-Photos-Hunters" -depth -mtime +15
May 25 11:34:26 Tower root: mvlogger: Not in Test mode
May 25 11:34:26 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/Plex-Photos-Hunters" -depth -mtime +15
May 25 11:34:26 Tower root: find: '/mnt/cache/Plex-Photos-Hunters': No such file or directory
May 25 11:34:26 Tower root: mvlogger: Share Name Only: PlexOptimized
May 25 11:34:26 Tower root: mvlogger: Cache Pool Name: cache
May 25 11:34:26 Tower root: mvlogger: cache Threshold Pct:
May 25 11:34:26 Tower root: mvlogger: OVERALL Threshold: 5
May 25 11:34:26 Tower root: mvlogger: Share Path: /mnt/cache/PlexOptimized
May 25 11:34:26 Tower root: mvlogger: Pool Pct Used: 14 %
May 25 11:34:26 Tower root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 5
May 25 11:34:26 Tower root: mvlogger: Threshold Used: 5
May 25 11:34:26 Tower root: mvlogger: Adding Age
May 25 11:34:26 Tower root: mvlogger: Age 15
May 25 11:34:26 Tower root: mvlogger: Skipfiletypes string: find "/mnt/cache/PlexOptimized" -depth -mtime +15
May 25 11:34:26 Tower root: mvlogger: Not in Test mode
May 25 11:34:26 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/PlexOptimized" -depth -mtime +15
May 25 11:34:26 Tower root: mvlogger: Share Name Only: TV
May 25 11:34:26 Tower root: mvlogger: Cache Pool Name: cache
May 25 11:34:26 Tower root: mvlogger: cache Threshold Pct:
May 25 11:34:26 Tower root: mvlogger: OVERALL Threshold: 5
May 25 11:34:26 Tower root: mvlogger: Share Path: /mnt/cache/TV
May 25 11:34:26 Tower root: mvlogger: Pool Pct Used: 14 %
May 25 11:34:26 Tower root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 5
May 25 11:34:26 Tower root: mvlogger: Threshold Used: 5
May 25 11:34:26 Tower root: mvlogger: Adding Age
May 25 11:34:26 Tower root: mvlogger: Age 15
May 25 11:34:26 Tower root: mvlogger: Skipfiletypes string: find "/mnt/cache/TV" -depth -mtime +15
May 25 11:34:26 Tower root: mvlogger: Not in Test mode
May 25 11:34:26 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/TV" -depth -mtime +15
May 25 11:34:27 Tower root: mover: finished
May 25 11:34:27 Tower root: mvlogger: No After Script to Run.
May 25 11:34:55 Tower emhttpd: read SMART /dev/sdg
May 25 11:34:55 Tower emhttpd: read SMART /dev/sdd
May 25 11:34:55 Tower emhttpd: read SMART /dev/sdf
May 25 11:34:55 Tower emhttpd: read SMART /dev/sdc

 

Link to post
4 minutes ago, kizer said:

This is what a Typical log looks like for me. Why it does its thing and then at the end it always does a read Smart is beyond me. I'm guessing thats what is trigger the drive spin up or its reading it after its spun up. I'm not 100% sure. I just know its annoying.

 


May 25 11:34:26 Tower root: mvlogger: Adding Age
May 25 11:34:26 Tower root: mvlogger: Age 15
May 25 11:34:26 Tower root: mvlogger: Skipfiletypes string: find "/mnt/cache/Plex-Photos-Hunters" -depth -mtime +15
May 25 11:34:26 Tower root: mvlogger: Not in Test mode
May 25 11:34:26 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/Plex-Photos-Hunters" -depth -mtime +15
May 25 11:34:26 Tower root: find: '/mnt/cache/Plex-Photos-Hunters': No such file or directory
May 25 11:34:26 Tower root: mvlogger: Share Name Only: PlexOptimized
May 25 11:34:26 Tower root: mvlogger: Cache Pool Name: cache
May 25 11:34:26 Tower root: mvlogger: cache Threshold Pct:
May 25 11:34:26 Tower root: mvlogger: OVERALL Threshold: 5
May 25 11:34:26 Tower root: mvlogger: Share Path: /mnt/cache/PlexOptimized
May 25 11:34:26 Tower root: mvlogger: Pool Pct Used: 14 %
May 25 11:34:26 Tower root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 5
May 25 11:34:26 Tower root: mvlogger: Threshold Used: 5
May 25 11:34:26 Tower root: mvlogger: Adding Age
May 25 11:34:26 Tower root: mvlogger: Age 15
May 25 11:34:26 Tower root: mvlogger: Skipfiletypes string: find "/mnt/cache/PlexOptimized" -depth -mtime +15
May 25 11:34:26 Tower root: mvlogger: Not in Test mode
May 25 11:34:26 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/PlexOptimized" -depth -mtime +15
May 25 11:34:26 Tower root: mvlogger: Share Name Only: TV
May 25 11:34:26 Tower root: mvlogger: Cache Pool Name: cache
May 25 11:34:26 Tower root: mvlogger: cache Threshold Pct:
May 25 11:34:26 Tower root: mvlogger: OVERALL Threshold: 5
May 25 11:34:26 Tower root: mvlogger: Share Path: /mnt/cache/TV
May 25 11:34:26 Tower root: mvlogger: Pool Pct Used: 14 %
May 25 11:34:26 Tower root: mvlogger: DFTPCT LIMIT USED FOR SETTING: 5
May 25 11:34:26 Tower root: mvlogger: Threshold Used: 5
May 25 11:34:26 Tower root: mvlogger: Adding Age
May 25 11:34:26 Tower root: mvlogger: Age 15
May 25 11:34:26 Tower root: mvlogger: Skipfiletypes string: find "/mnt/cache/TV" -depth -mtime +15
May 25 11:34:26 Tower root: mvlogger: Not in Test mode
May 25 11:34:26 Tower root: mvlogger: Complete Mover Command: find "/mnt/cache/TV" -depth -mtime +15
May 25 11:34:27 Tower root: mover: finished
May 25 11:34:27 Tower root: mvlogger: No After Script to Run.
May 25 11:34:55 Tower emhttpd: read SMART /dev/sdg
May 25 11:34:55 Tower emhttpd: read SMART /dev/sdd
May 25 11:34:55 Tower emhttpd: read SMART /dev/sdf
May 25 11:34:55 Tower emhttpd: read SMART /dev/sdc

 

The "mvlogger: No After Script to Run" is right at the end of my bash script.

 

  rm -f $PIDFILE
  [[ $LOGLEVEL -gt 0 ]] && echo "mover: finished"

  #Run mover after script if specified.
  if [ -z $AFTERSCRIPT ] ; then
        mvlogger "No After Script to Run."
  else
        if [ -f $AFTERSCRIPT ] ; then
                mvlogger "After script will be executed: $AFTERSCRIPT"
                eval "$AFTERSCRIPT"
        fi
  fi

}    <---  closes out the function and there isn't anything else to run...

 

This does run in the controlling php script after the call to the bash script.

        if ( $cfg['enableTurbo'] == "yes" ) {
                logger("Restoring original turbo write mode");
                exec("/usr/local/sbin/mdcmd set md_write_method {$vars['md_write_method']}");
        }
 

But I wouldn't think that would spin up your drives.

 

 

It's also almost 30 seconds after the script has finished also. 

 

Is there a way to see what spins up drives in unraid?  Other than the syslog entry? 

Link to post

I do see my drives being read at 682k, which is typically Plex looking for Drive changes. I can't look at Plex from work because I know I have it set to periodically check for changes, but I know it looks for changes when the drives are spun up. 

 

I'll disable that. Heck I'll shut down Plex all together for the heck of it and try it again. 

Link to post

Hey everyone,

Love the plugin. I'm having trouble getting an "after script" to run. The move runs fine, and the script runs fine by itself when I run it manually, but it just isn't running for some reason when I add it to this plugin. Hopefully something obvious, and sorry if I missed it in this thread.

Thanks.

 

Here is the syslog line where it registers the script:

May 27 21:20:01 Haven root: mvlogger: AFTERSCRIPT: /boot/config/plugins/user.scripts/scripts/SyncRaid50

 

And here is the script itself if that helps:

#!/bin/bash

#MOUNT DISK BY ID (PLUG IN YOUR DISK_ID)
DISK_ID=50_0123456789ABCDEF
THIS_DISK=`ls -l /dev/disk/by-id/ | grep '$DISK_ID' | head -1 | tail -c4`
/usr/local/sbin/rc.unassigned mount "/dev/$THIS_DISK"
if [[ $? -ne 0 ]]; then 
  echo "Exiting due to ERROR."
  exit 1
fi

#RUN SYNC COMMANDS
test -e "/mnt/disks/Mega" && rsync -avh --progress "/mnt/user/Mega/" "/mnt/disks/Mega/2D"
test -e "/mnt/disks/Mega" && rsync -avh --progress "/mnt/user/VR/" "/mnt/disks/Mega/VR"

#UNMOUNT DISK BY ID
/usr/local/sbin/rc.unassigned umount "/dev/$THIS_DISK"

 

Link to post
3 minutes ago, stev067 said:

Hey everyone,

Love the plugin. I'm having trouble getting an "after script" to run. The move runs fine, and the script runs fine by itself when I run it manually, but it just isn't running for some reason when I add it to this plugin. Hopefully something obvious, and sorry if I missed it in this thread.

Thanks.

 

Here is the syslog line where it registers the script:


May 27 21:20:01 Haven root: mvlogger: AFTERSCRIPT: /boot/config/plugins/user.scripts/scripts/SyncRaid50

 

And here is the script itself if that helps:


#!/bin/bash

#MOUNT DISK BY ID (PLUG IN YOUR DISK_ID)
DISK_ID=50_0123456789ABCDEF
THIS_DISK=`ls -l /dev/disk/by-id/ | grep '$DISK_ID' | head -1 | tail -c4`
/usr/local/sbin/rc.unassigned mount "/dev/$THIS_DISK"
if [[ $? -ne 0 ]]; then 
  echo "Exiting due to ERROR."
  exit 1
fi

#RUN SYNC COMMANDS
test -e "/mnt/disks/Mega" && rsync -avh --progress "/mnt/user/Mega/" "/mnt/disks/Mega/2D"
test -e "/mnt/disks/Mega" && rsync -avh --progress "/mnt/user/VR/" "/mnt/disks/Mega/VR"

#UNMOUNT DISK BY ID
/usr/local/sbin/rc.unassigned umount "/dev/$THIS_DISK"

 

I'll try and look more at this tomorrow mornings.  But in the meantime...

 

Whats the permissions of the script and owner?  i.e. ls -ltr /boot/config/plugins/user.scripts/scripts/SyncRaid50

Can you put a few "echo" statements in your script.  i.e. beginning and ending?

Link to post
3 minutes ago, hugenbdd said:

I'll try and look more at this tomorrow mornings.  But in the meantime...

 

Whats the permissions of the script and owner?  i.e. ls -ltr /boot/config/plugins/user.scripts/scripts/SyncRaid50

Can you put a few "echo" statements in your script.  i.e. beginning and ending?

 

Here's what I get from that:

-rw------- 1 root root  10 May 27 20:45 name
-rw------- 1 root root 565 May 27 20:45 script

 

Just tried again with bookmark echoes, and it didn't change the result, and not seeing an echo in syslog. I appreciate your help!

Link to post
13 minutes ago, stev067 said:

 

Here's what I get from that:


-rw------- 1 root root  10 May 27 20:45 name
-rw------- 1 root root 565 May 27 20:45 script

 

Just tried again with bookmark echoes, and it didn't change the result, and not seeing an echo in syslog. I appreciate your help!

You are missing execute permissions on the file.

 

chmod 775 on that file and it should help.

Link to post
26 minutes ago, hugenbdd said:

You are missing execute permissions on the file.

 

chmod 775 on that file and it should help.

I tried both of the below and neither fixed it. Was I missing something? Sorry my linux experience is like 10 years dormant.

 

chmod 775 /boot/config/plugins/user.scripts/scripts/SyncRaid50

chmod -R 775 /boot/config/plugins/user.scripts/scripts/SyncRaid50

 

Link to post
8 hours ago, stev067 said:

I tried both of the below and neither fixed it. Was I missing something? Sorry my linux experience is like 10 years dormant.

The flash drive doesn't support executable permissions.  You have to copy the file(s) to somewhere in RAM (say /tmp) and then execute it from there (after applying executable perms to the copy)

Link to post
1 hour ago, Squid said:

The flash drive doesn't support executable permissions.  You have to copy the file(s) to somewhere in RAM (say /tmp) and then execute it from there (after applying executable perms to the copy)

 

Hmm so I can't just add a 'cp' line to my script though because the script isn't being executed in the first place. Do you mean that I should do a 1-time move of the script into RAM and then leave it there? Would it persist after a reboot?

Link to post

You either have to do the cp via user scripts at startup (or via the go file), or if the plugin supports it, make the path to the script to be something like

 

/bin/bash /path/to/the/script

which will execute it directly from the flash

Link to post
47 minutes ago, Squid said:

You either have to do the cp via user scripts at startup (or via the go file), or if the plugin supports it, make the path to the script to be something like

 


/bin/bash /path/to/the/script

which will execute it directly from the flash

 

Ok so I created a helper script that will run on first array start. Copies the script into the existing RAM scripts folder, and changes the permission.

 

#!/bin/bash
cp -r /boot/config/plugins/user.scripts/scripts/SyncRaid50 /tmp/user.scripts/tmpScripts
chmod -R 775 /tmp/user.scripts/tmpScripts/SyncRaid50

 

Here is the result of 'ls -l' on the script folder and contents after the above helper script:

 

drwxrwxr-x 2 root root 80 May 28 10:57 SyncRaid50/
-rwxrwxr-x 1 root root  10 May 28 10:57 name*
-rwxrwxr-x 1 root root 619 May 28 10:57 script*

 

For mover tuning, I changed the afterscript to the new location.

 

/tmp/user.scripts/tmpScripts/SyncRaid50

 

And still the script isn't running. Something wrong with the permission still? Thanks for all your help.

 

 

Link to post
4 minutes ago, stev067 said:

 

Ok so I created a helper script that will run on first array start. Copies the script into the existing RAM scripts folder, and changes the permission.

 


#!/bin/bash
cp -r /boot/config/plugins/user.scripts/scripts/SyncRaid50 /tmp/user.scripts/tmpScripts
chmod -R 775 /tmp/user.scripts/tmpScripts/SyncRaid50

 

Here is the result of 'ls -l' on the script folder and contents after the above helper script:

 


drwxrwxr-x 2 root root 80 May 28 10:57 SyncRaid50/
-rwxrwxr-x 1 root root  10 May 28 10:57 name*
-rwxrwxr-x 1 root root 619 May 28 10:57 script*

 

For mover tuning, I changed the afterscript to the new location.

 


/tmp/user.scripts/tmpScripts/SyncRaid50

 

And still the script isn't running. Something wrong with the permission still? Thanks for all your help.

 

 

Looks like you are putting the directory into mover tuning.  try the file.

/tmp/user.scripts/tmpScripts/SyncRaid50/script

Link to post
Posted (edited)
4 hours ago, hugenbdd said:

Looks like you are putting the directory into mover tuning.  try the file.

/tmp/user.scripts/tmpScripts/SyncRaid50/script

Eureka! That was the final piece. Here is my final summary if anyone else wants to set this up. And I apologize if there's already documentation on this somewhere.

 

Below is the path to the original script I wrote in Userscripts. You just really need to plug in the script name of this into the SCRIPT_NAME variable of the helper file below. Generally speaking, it's a good idea to avoid spaces in script names.

/boot/config/plugins/user.scripts/scripts/SyncRaid50

 

This script can only be executed as a Mover Tuning afterscript if it is in RAM and has execute permission. Making this helper script which is set to run on array start will ensure it gets placed in RAM when the server boots. This helper script would have to be re-run any time you want to lock in changes to the main script. Below is the code for my helper script.

#!/bin/bash
SCRIPT_NAME="SyncRaid50" #THIS IS THE TARGET SCRIPT THAT YOU WANT TO RUN VIA THIS HELPER SCRIPT
echo "copying $SCRIPT_NAME script to RAM and granting execute permission."
cp -r "/boot/config/plugins/user.scripts/scripts/$SCRIPT_NAME" "/tmp/user.scripts/tmpScripts"
chmod -R 775 /tmp/user.scripts/tmpScripts/$SCRIPT_NAME
echo "RAM copy script complete. Last modify date of RAM file is now the following:"
date -r /tmp/user.scripts/tmpScripts/$SCRIPT_NAME/script

 

Lastly, below is what to put in the Aferscript field of Mover Tuning, where 'SyncRaid50' is the name of your target script.

/tmp/user.scripts/tmpScripts/SyncRaid50/script

 

Thanks a lot to Hugenbdd and Squid for your help, and thanks Squid for another great plugin.

Edited by stev067
Link to post
6 minutes ago, stev067 said:

thanks Squid for another great plugin

Thank huggenbd.  He's long since taken over it.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.