Hi everybody

I searched the forum, but did not find anything relevant:

I was wondering if there is a modification that could allow for the cache mover to immediately start working when the drives are idle (or have been for some time eg 5mins) and stop again when there is new activity from a user. This would be useful since it would minimize the time that the data remains unprotected on the drive.

 

Also, I was wandering if it is possible to have some cache drive mirroring, ie having 2 drives with identical cached data. This would also give full protection to the cached data since there would be one disk redundancy. I show some mentions in the forums but no follow-ups. I guess this is probably more complicated and might be only possible in a new unRAID release...

 

Many thanks

 

The mover is just scheduled via cron, so you can set it to any time(s) you want..

 

I guess if you set up a mirrored disk via the motherboard or raid card and then assign it as the cache drive, that could work for you..

The mover is just scheduled via cron, so you can set it to any time(s) you want..

of course you can schedule it anytime you want, but you cannot know beforehand when the hard disks will be idle so still my question is valid. to be more specific i can set it, let's say, every day at 03.30. but if the server is doing something else, i dont want it to start. instead i want it to be activated whenever the hard disks are idle

 

I guess if you set up a mirrored disk via the motherboard or raid card and then assign it as the cache drive, that could work for you..

yes this could work, i might try it, although i cannot guess if any problems might arise. i dont know how transparent raid would be to unraid or what problems their interaction may cause...

The mover is just scheduled via cron, so you can set it to any time(s) you want..

of course you can schedule it anytime you want, but you cannot know beforehand when the hard disks will be idle so still my question is valid. to be more specific i can set it, let's say, every day at 03.30. but if the server is doing something else, i dont want it to start. instead i want it to be activated whenever the hard disks are idle

 

There is nothing in place today to run the mover when the cache drive is idle.

There IS some protection in the mover script which will skip open files. Therefore any files currently in use will be left for later.

 

 

I guess if you set up a mirrored disk via the motherboard or raid card and then assign it as the cache drive, that could work for you..

yes this could work, i might try it, although i cannot guess if any problems might arise. i dont know how transparent raid would be to unraid or what problems their interaction may cause...

 

At the current time the only solution is hardware raid.

There are a few options at the current time.

RAID1 with a silicon image chipset on the motherboard, external or with a special PCI bracket adapter. (See addonics).

RAID 1 with a 3ware or ARECA hardware controller.

Software raid1 is not available at the current time.

 

I have tested all 3 hardware raid solutions with success.

There is nothing in place today to run the mover when the cache drive is idle.

 

Put the following commands in a file at /boot.  Name it are_disks_idle.sh

#!/bin/bash
###############################################
# are_disks_idle.sh
# exit status = 0 if all disks idle
# exit status = 1 if any disk spinning
# exit status = 2 if array is not atarted
#
# April 2010 Joe L.

# Check if the array is started
if [ -d /mnt/cache0 ]
then
  # rdevLastIO will be non-zero if a disk is spinning
  last=`/root/mdcmd status | grep -a rdevLastIO | grep -v '=0'`
  if [ "$last}" = "" ]
  then
    # all disks are idle
    exit 0
  else
    # all disks are not idle
    exit 1
  fi
else
  # array is not started
  exit 2
fi

 

Then, on the "Shares" page in the unRAID interface, change the Cache mover schedule

from

40 3 * * *

to

*/5 * * * * /boot/are_disks_idle.sh &&

 

The resulting line in crontab will look like this:

*/5 * * * * /boot/are_disks_idle.sh && /usr/local/sbin/mover 2>&1 | logger

 

The check for idle disks will occur every 5 minutes

The are_disks_idle.sh will test if all the disks are idle, and if they are the "mover" will be run based on its exit status of 0.

The "mover" script has a lock-file to prevent multiple instances of itself from running.

The "mover" script will skip any file that is open for reading or writing.

 

Joe L.

Very nice Joe! I might have use for that in the near future

SWEET.

 

What is the time out period for all disks being idle?

SWEET.

 

What is the time out period for all disks being idle?

You set the spin-down timer yourself on the unRAID web-management page, either globally for all disks on the settings page, or individually per disk on the main page by clicking on the disk name.

 

So, the time-out is whatever you set it to be.

Very nice Joe! I might have use for that in the near future

I'm taking advantage of the fact that unRAID is not checking what you enter to contain only a crontab schedule. 

I use the "&&" to conditionally invoke "mover" based on the test performed  (and exit status) in all_disks_idle.sh

 

I could almost do it all in the crontab field on the unRAID web-form, and not need the shell script at all, but there is a limit of characters the web-field will hold.

 

Joe L.

SWEET.

 

What is the time out period for all disks being idle?

You set the spin-down timer yourself on the unRAID web-management page, either globally for all disks on the settings page, or individually per disk on the main page by clicking on the disk name.

 

So, the time-out is whatever you set it to be.

 

So does the highest one need to be satisfied.

Sorry, I'm not familiar with the rdevLastIO

SWEET.

 

What is the time out period for all disks being idle?

You set the spin-down timer yourself on the unRAID web-management page, either globally for all disks on the settings page, or individually per disk on the main page by clicking on the disk name.

 

So, the time-out is whatever you set it to be.

 

So does the highest one need to be satisfied.

Sorry, I'm not familiar with the rdevLastIO

It contains 0 if the disk is spun down.  It contains a linux date -s string if the disk is spinning.  the date -s "string" is the time, in seconds since Jan 1st 1970, that the disk last had I/O.

 

So, when ever you do any I/O to the disk, the LastIO time is updated to that second.  When the "time between" that LastIO and now exceeds the spin-down delay, the disk is spun down and the value set to 0.

 

There is one LastIO timer per disk in the array.  If all have "=0" in their strings extracted by grep, then all are spun down.

 

Thanks for the explanation.

 

There is one LastIO timer per disk in the array.  If all have "=0" in their strings extracted by grep, then all are spun down.

 

Oh Wait.. so does this mean.. When the drives are all spun down...

They will be spun back up now to do the cache mover script?

There is one LastIO timer per disk in the array.  If all have "=0" in their strings extracted by grep, then all are spun down.

 

Oh Wait.. so does this mean.. When the drives are all spun down...

They will be spun back up now to do the cache mover script?

If there is something to move, yes, a disk will need to be spun up.  Otherwise, nothing should spin up.  It works that way now at 3:40 AM, as most of the time most (all) disks will be idle since you are not actively using the server.

Thanx a lot Joe, this is more than i hoped

 

ideally, it would be even better if i could define idle not as spun down but just not having copied anything for x minutes. But this is a very minor detail...

Thanx a lot Joe, this is more than i hoped

 

ideally, it would be even better if i could define idle not as spun down but just not having copied anything for x minutes. But this is a very minor detail...

OK,

Here is a slightly different script.  (I've attached a zipped copy, just download and un-zip to your flash drive)

I named it last_io.sh

 

#!/bin/bash
#Assume inactive if no read/write in 900 seconds if $1 arg is not given.

if [ "$1" = "-q" ]
then
 quiet_mode=yes
 shift
else
 quiet_mode=no
fi

no_activity_time=${1-900}

#Get the disk spinup status
disk_status=`/root/mdcmd status | strings | grep rdevLastIO`
# Get the current time.
now=`date +%s`
longest_elapsed_time=0
most_recent_disk_activity=100000000
most_recent_disk=""

function hms() {
   seconds=$1
   hours=$(($seconds / 3600))
   seconds=$(($seconds - ($hours * 3600)))
   minutes=$(($seconds / 60))
   seconds=$(($seconds - ($minutes * 60)))
   [ "$hours" = "0" ] && hours="" || hours="$hours hours, "
   [ "$minutes" = "0" ] && minutes="" || minutes="$minutes minutes, "
   echo $hours$minutes$seconds
}

for i in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
do
   last_access=`echo "$disk_status" | grep "rdevLastIO.$i=" | cut -d"=" -f2`
   if [ "$last_access" = "" ]
   then
     continue
   fi

   elapsed_time=$(expr $now - $last_access)
   i=`printf "disk%d" $i`
   if [ "$last_access" != "0" ]
   then
      if [ ${elapsed_time} -le ${most_recent_disk_activity} ]
      then
         most_recent_disk_activity=${elapsed_time}
         most_recent_disk="$i"
      fi
      if [ ${elapsed_time} -gt ${longest_elapsed_time} ]
      then
         longest_elapsed_time=${elapsed_time}
      fi
   fi
done

if [ $longest_elapsed_time = 0 ]
then
   [ $quiet_mode = no ] && echo "No disks are spinning."
   exit 0
else
   if [ $most_recent_disk_activity -gt $no_activity_time ]
   then
       [ $quiet_mode = no ] && echo -n "All disks inactive for at least `hms $no_activity_time` seconds. "
       [ $quiet_mode = no ] && echo "Most recent disk I/O was on $most_recent_disk `hms $most_recent_disk_activity` seconds ago."
       exit 0
   else
       [ $quiet_mode = no ] && echo "Most recent disk I/O was on $most_recent_disk `hms $most_recent_disk_activity` seconds ago."
       exit 1
   fi
fi

 

Invoke it as

last_io.sh 600

The number is in seconds. The example above would test for disk i/o within 10 minutes (600 seconds)

If you do not supply any value as an argument, 900 seconds is used (15 minutes)

 

output looks like this:

root@Tower:/boot/custom/bin# last_io.sh 600

Most recent disk I/O was on disk0 5 minutes, 4 seconds ago.

 

waiting a bit longer and re-running the command:

root@Tower:/boot/custom/bin# last_io.sh 600

All disks inactive for at least 10 minutes, 0 seconds. Most recent disk I/O was on disk0 11 minutes, 6 seconds ago.

 

If all you want is the exit status of 0 or 1, as before, invoke it in quiet mode as

last_io.sh -q 600

 

Exit status = 0 if no disks are spinning or if the last I/O time is greater than the number of seconds specified.

Exit status = 1 if last I/O less than the number of seconds specified.

 

fyi: disk0 = parity disk.

 

Joe L.

last_io.zip

I like the new version better. Thanks Joe, You rock!

well, thats even better  .

Thanx Joe

Great!  The new version will work regardless of whether you've enabled unRAID's spindown function or not.  I like that.