August 14, 201015 yr Is there a way to pause the mover process? I say this because while mover is in progress, performance of the server is diminished when viewing movies especially BD rips. I know the obvious is to schedule mover when the mediapc is not in use but sometimes that is unavoidable.
August 14, 201015 yr How much are you willing to edit the /usr/local/sbin/mover script? Each choice below becomes more involved in what it does, but gives you some ideas to work with. You can add the following line to alter the priority of the script and subsequent children renice +19 -p $$ >/dev/null 2>&1 You can add the following lines and put the IONICE infront of each find if [ -x /usr/bin/ionice ]; then IONICE="/usr/bin/ionice -c3" fi (cd /mnt/cache; $IONICE find -depth -print \ I don't know if the IONICE -c3 works on $$ like the renice command does. Another choice is to add --bwlimit=8192 to each rsync command to slow it down and control how much data it moves to the array. Another choice is to create a new shell called mover.sh and put it somewhere else. This shell will need to update /var/run/mover.pid and stay active until you are done. it activates the duplicate process failsafe. See below. # If a previous invokation of this script is already running, exit if [ -f /var/run/mover.pid ]; then if ps h `cat /var/run/mover.pid` | grep mover ; then echo "mover already running" exit 0 fi fi Then run it, update the /var/run/mover.pid with the mover.sh's pid to keep /usr/local/sbin/mover.sh from running at all. This has the effect of stopping /usr/local/sbin/mover from running that day which means data stays on the cache until the next day. Another choice is to modify the fail safe. Change this line if ps h `cat /var/run/mover.pid` | grep mover ; then to if kill -0 `cat /var/run/mover.pid` ; then Then when you want to stop the mover from running. pidof /usr/local/sbin/emhttp > /var/run/mover.pid When you want to let mover run rm -v /var/run/mover.pid This has the same effect in canceling mover, but canceling it until the pid file is removed. or you could just echo $$ > /var/run/mover.pid Then when you logoff the bash session, the mover.pid is inactive. Again same effect. Cancel mover until the pid is no longer active. None of these are tested or proven to work, it just provides ideas to work with. If it were me I might just work with the priority and bwlimit changes. This insures the mover runs at a slower pace.
August 14, 201015 yr Is there a way to pause the mover process? I say this because while mover is in progress, performance of the server is diminished when viewing movies especially BD rips. I know the obvious is to schedule mover when the mediapc is not in use but sometimes that is unavoidable. Yes there is a way... It was described here http://lime-technology.com/forum/index.php?topic=6033.msg57983#msg57983 and a script is attached to that post. Once you unzip the last_io.sh script to the root of your flash drive, on the "Shares" page in the unRAID interface, change the Cache mover schedule from 40 3 * * * to */5 * * * * /boot/last_io.sh -q %26%26 The %26%26 will be translated to "&&" (On 4.X unraid, you can use "&&", but on 5.X unraid, you'll nee to use "%26%26" to encode the "&&") The resulting command in crontab will be */5 * * * * /boot/last_io.sh -q && /usr/local/sbin/mover 2>&1 | logger last_io.sh will only run mover if all your disks have been idle for 15 minutes or more. (Not spun down, but no IO within that time period) It will move your files from cache as soon as it gets a chance, as it checks every 5 minutes for your server's disks to go idle. If you prefer a shorter interval, you can supply an optional parameter to last_io.sh Try last_io.sh 300 as a test on the command line. the "-q" option suppresses the description of the last IO.
August 14, 201015 yr OK that's much more elegant. Forget what I wrote. heh. I do use the renice, ionice and --bwlimit=8192 for all my rsync moves which is where I got that from. the last_io.sh is much easier and more elegant. I forgot about it.
October 26, 201015 yr i Didn't get this part. If i make the change in the under the share tab and i put the scrip in my flash. Would the mover run automatically every day at certain time or do i need to configure that elsewhere?? Im kind of lost at this mover part. If someone can point me in the right direction, I would like to know where can i modify the time and how often the Mover script should be run. Thanks in advance!
October 26, 201015 yr i Didn't get this part. If i make the change in the under the share tab and i put the scrip in my flash. Would the mover run automatically every day at certain time or do i need to configure that elsewhere?? Im kind of lost at this mover part. If someone can point me in the right direction, I would like to know where can i modify the time and how often the Mover script should be run. Thanks in advance! If you make the change as described in my prior post the "mover" script will check every 5 minutes IF all the disks have been idle for 10 minutes. If the data disks have been idle for 10 minutes, the "mover" script will move files from cache (if there are any to move) No need to configure anything else elsewhere.
October 26, 201015 yr Ok. I have one problem. I did the modifications and i move a small file to test if everything was working properly. I copy the file to the share. Checked the files was copied to the Cache disk and waited for 15 / 20 mins. No write / read in those mins and the file wasnt moved to the share. Both shares had cache enabled and in the cache disk, the folder path is created according to the folder in the User SHare. The script is checking every 5 mins but doesn't start the file move. LAst syslog lines: Oct 26 13:00:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 13:05:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 13:10:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 13:15:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 13:20:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 13:25:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Any idea of what is the problem? Thx.
October 26, 201015 yr Ok. I have one problem. Any idea of what is the problem? Thx. You apparently are using a version of unRAID prior to version 5.0 You need to use */5 * * * * /boot/last_io.sh -q && as your mover schedule since the %26 characters are not converted to "&" in your older version of unRAID.
October 26, 201015 yr Ok. I have one problem. Any idea of what is the problem? Thx. You apparently are using a version of unRAID prior to version 5.0 You need to use */5 * * * * /boot/last_io.sh -q && as your mover schedule since the %26 characters are not converted to "&" in your older version of unRAID. Ok,I am going to give it a try right now, and yes. Im using 4.5.6 Thx Joe
October 27, 201015 yr sorry to keep bothering everyone with the same thing but i have one new question about this mover script. When the script is running and checking every 5 mins the disk seems not spin down at all, I mean ever. Does this make any sense to anyone? Last log lines atached ( I spin down the disk and they came up right away) Oct 26 20:25:03 Tower logger: mover finished Oct 26 20:30:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:35:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:40:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:45:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:45:01 Tower logger: mover started Oct 26 20:45:01 Tower logger: . Oct 26 20:45:03 Tower logger: mover finished Oct 26 20:50:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:55:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:00:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:05:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:05:01 Tower logger: mover started Oct 26 21:05:01 Tower logger: . Oct 26 21:05:03 Tower logger: mover finished Oct 26 21:10:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:15:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:17:21 Tower emhttp: shcmd (38): sync Oct 26 21:17:24 Tower emhttp: Spinning down all drives... Oct 26 21:17:24 Tower kernel: mdcmd (3727): spindown 0 Oct 26 21:17:25 Tower kernel: mdcmd (3728): spindown 1 Oct 26 21:17:25 Tower kernel: mdcmd (3729): spindown 2 Oct 26 21:17:26 Tower emhttp: shcmd (39): /usr/sbin/hdparm -y /dev/sda >/dev/null Oct 26 21:20:01 Tower logger: mover started Oct 26 21:20:01 Tower logger: . Oct 26 21:20:23 Tower emhttp: shcmd (40): /usr/sbin/hdparm -y /dev/sda >/dev/null Oct 26 21:20:25 Tower logger: mover finished
October 27, 201015 yr sorry to keep bothering everyone with the same thing but i have one new question about this mover script. When the script is running and checking every 5 mins the disk seems not spin down at all, I mean ever. Does this make any sense to anyone? Last log lines atached ( I spin down the disk and they came up right away) Oct 26 20:25:03 Tower logger: mover finished Oct 26 20:30:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:35:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:40:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:45:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:45:01 Tower logger: mover started Oct 26 20:45:01 Tower logger: . Oct 26 20:45:03 Tower logger: mover finished Oct 26 20:50:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:55:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:00:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:05:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:05:01 Tower logger: mover started Oct 26 21:05:01 Tower logger: . Oct 26 21:05:03 Tower logger: mover finished Oct 26 21:10:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:15:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:17:21 Tower emhttp: shcmd (38): sync Oct 26 21:17:24 Tower emhttp: Spinning down all drives... Oct 26 21:17:24 Tower kernel: mdcmd (3727): spindown 0 Oct 26 21:17:25 Tower kernel: mdcmd (3728): spindown 1 Oct 26 21:17:25 Tower kernel: mdcmd (3729): spindown 2 Oct 26 21:17:26 Tower emhttp: shcmd (39): /usr/sbin/hdparm -y /dev/sda >/dev/null Oct 26 21:20:01 Tower logger: mover started Oct 26 21:20:01 Tower logger: . Oct 26 21:20:23 Tower emhttp: shcmd (40): /usr/sbin/hdparm -y /dev/sda >/dev/null Oct 26 21:20:25 Tower logger: mover finished when did you reset the "mover" schedule? When you type crontab -l | grep mover what does it say? Mine says this: # Generated mover schedule: */5 * * * * /boot/are_disks_idle.sh && /usr/local/sbin/mover 2>&1 | logger Yours should be the same. Joe L.
October 27, 201015 yr sorry to keep bothering everyone with the same thing but i have one new question about this mover script. When the script is running and checking every 5 mins the disk seems not spin down at all, I mean ever. Does this make any sense to anyone? Last log lines atached ( I spin down the disk and they came up right away) Oct 26 20:25:03 Tower logger: mover finished Oct 26 20:30:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:35:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:40:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:45:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:45:01 Tower logger: mover started Oct 26 20:45:01 Tower logger: . Oct 26 20:45:03 Tower logger: mover finished Oct 26 20:50:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 20:55:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:00:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:05:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:05:01 Tower logger: mover started Oct 26 21:05:01 Tower logger: . Oct 26 21:05:03 Tower logger: mover finished Oct 26 21:10:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:15:01 Tower logger: /boot/last_io.sh: line 62: [: %26%26: integer expression expected Oct 26 21:17:21 Tower emhttp: shcmd (38): sync Oct 26 21:17:24 Tower emhttp: Spinning down all drives... Oct 26 21:17:24 Tower kernel: mdcmd (3727): spindown 0 Oct 26 21:17:25 Tower kernel: mdcmd (3728): spindown 1 Oct 26 21:17:25 Tower kernel: mdcmd (3729): spindown 2 Oct 26 21:17:26 Tower emhttp: shcmd (39): /usr/sbin/hdparm -y /dev/sda >/dev/null Oct 26 21:20:01 Tower logger: mover started Oct 26 21:20:01 Tower logger: . Oct 26 21:20:23 Tower emhttp: shcmd (40): /usr/sbin/hdparm -y /dev/sda >/dev/null Oct 26 21:20:25 Tower logger: mover finished when did you reset the "mover" schedule? When you type crontab -l | grep mover what does it say? Mine says this: # Generated mover schedule: */5 * * * * /boot/are_disks_idle.sh && /usr/local/sbin/mover 2>&1 | logger Yours should be the same. Joe L. Mine said # Generated mover schedule: */5 * * * * /boot/last_io.sh -q %26%26 /usr/local/sbin/mover 2>&1 | logger # Generated mover schedule: */5 * * * * /boot/last_io.sh -q && /usr/local/sbin/mover 2>&1 | logger Clearly not the same but i have no clue where is the problem
October 27, 201015 yr It appears as if the first line was not deleted when the second was added. To fix, type these few commands crontab -l >/tmp/crontab sed -i "/%26/d" /tmp/crontab cp /tmp/crontab /var/spool/cron/crontabs/root- crontab /tmp/crontab Those commands will make a copy of the cron schedule, then use the stream editor (sed) to delete the line with the %26 Once the line is deleted, the newly edited file is re-submitted to replace the existing schedule.
October 27, 201015 yr It appears as if the first line was not deleted when the second was added. To fix, type these few commands crontab -l >/tmp/crontab sed -i "/%26/d" /tmp/crontab cp /tmp/crontab /var/spool/cron/crontabs/root- crontab /tmp/crontab Those commands will make a copy of the cron schedule, then use the stream editor (sed) to delete the line with the %26 Once the line is deleted, the newly edited file is re-submitted to replace the existing schedule. I use all thecomand that you posted but i have no idea how use stream editor to delete the line with the %26. I been here for just 1 week
October 27, 201015 yr It appears as if the first line was not deleted when the second was added. To fix, type these few commands crontab -l >/tmp/crontab sed -i "/%26/d" /tmp/crontab cp /tmp/crontab /var/spool/cron/crontabs/root- crontab /tmp/crontab Those commands will make a copy of the cron schedule, then use the stream editor (sed) to delete the line with the %26 Once the line is deleted, the newly edited file is re-submitted to replace the existing schedule. I use all the comand that you posted but i have no idea how use stream editor to delete the line with the %26. I been here for just 1 week You've already used it...(you are a very quick student) It was the command line starting with "sed -i" If you now type crontab -l | grep mover you will see no line with %26 any longer. (if you typed it correctly)
October 27, 201015 yr OK. I think first time i did get it wrong but after one more try i did get this output # Generated mover schedule: */5 * * * * /boot/last_io.sh -q && /usr/local/sbin/mover 2>&1 | logger So i guess should be ok now. Thx again for the help Joe
October 27, 201015 yr Weird stuff..mover keep spining the disk up even though there isn't any file to move. Oct 26 23:05:01 Tower logger: mover started Oct 26 23:05:01 Tower logger: . Oct 26 23:05:08 Tower logger: mover finished i did the crontab -l | grep mover and i have the same output than you do
October 27, 201015 yr Weird stuff..mover keep spining the disk up even though there isn't any file to move. Oct 26 23:05:01 Tower logger: mover started Oct 26 23:05:01 Tower logger: . Oct 26 23:05:08 Tower logger: mover finished i did the crontab -l | grep mover and i have the same output than you do How do you expect it to learn if there are any files to be moved?
October 27, 201015 yr Weird stuff..mover keep spining the disk up even though there isn't any file to move. Oct 26 23:05:01 Tower logger: mover started Oct 26 23:05:01 Tower logger: . Oct 26 23:05:08 Tower logger: mover finished i did the crontab -l | grep mover and i have the same output than you do How do you expect it to learn if there are any files to be moved? I dont know. Should i Expect it to learn some way?.Is it the normal behavior and the drives are going to be always spined up? I dont really have a problem with the checking, moving, etc...but i would rather let the disk spin down. is this posible with this script enabled?
October 27, 201015 yr I dont know. Should i Expect it to learn some way?.Is it the normal behavior and the drives are going to be always spined up? I dont really have a problem with the checking, moving, etc...but i would rather let the disk spin down. is this posible with this script enabled? Any clue on this??
October 27, 201015 yr The "mover" just runs a "find" command on the cache disk and if it encounters files it moves them if they are not currently being written. If the blocks of memory representing the directory entries on the cache drive are in the disk buffer cache, then the drive does not need to spin up to list the contents of the directories. If the directory entries are not in the disk buffer cache, the disk must spin up for the directories to be read. The disk buffer cache frees (and re-uses) the cache memory of the least recently used data first. If you have 2 Gig of memory and play a 4Gig movie the least recently used buffer could easily be that of the "mover" script at some point. Running it every 5 minutes should keep it in memory, but I have no idea what is happening with the activity on your server. What are you running that might be using the disk buffers? You might try this command and see if it stops the disk spin-ups from continuing: sed -i "s/^sync/#sync/" /usr/local/sbin/mover (I had forgotten about the "sync" that had been added to the mover script. Eliminating it will probably make it act as you desire. You'll need to add the "sed" command described above to the end of your config/go script to have it do the same the next time you reboot. "sync" would spin up all the disks and flush the disk buffer cache to them.) Not too long ago another user posted an alternative "mover" script that used a flag file to conditionally invoke the "sync" command, but only if a file was actually moved. You might try to search for it. Joe L.
October 28, 201015 yr Not too long ago another user posted an alternative "mover" script that used a flag file to conditionally invoke the "sync" command, but only if a file was actually moved. You might try to search for it. Try looking here
December 29, 201015 yr Weird stuff..mover keep spining the disk up even though there isn't any file to move. Oct 26 23:05:01 Tower logger: mover started Oct 26 23:05:01 Tower logger: . Oct 26 23:05:08 Tower logger: mover finished i did the crontab -l | grep mover and i have the same output than you do How do you expect it to learn if there are any files to be moved? I dont know. Should i Expect it to learn some way?.Is it the normal behavior and the drives are going to be always spined up? I dont really have a problem with the checking, moving, etc...but i would rather let the disk spin down. is this posible with this script enabled? I just added a package to unMENU to change "mover" to conditionally sync only if files are actually moved, and to optionally spin down the drives after the sync has occurred. If you have unMENU installed you can check for updates and get it. Joe L.
Archived
This topic is now archived and is closed to further replies.