Hi,

 

I have a WDTV set top box for streaming my media from my Unraid server and this has been working great over the last few years.

 

I would like to speed up browsing my movie collection (loading box art, movie info and backdrops) and keep my drives spun down unless I play a movie , so thought I could do that by keeping those files on my SSD cache drive and just keep the actual movie on the Raid.

 

Is there a way I can get mover to ignore certain files/folders and not move them off the Cache drive?

 

As an example. Each movie has the following files:-

/Movies/HD Movies/A_Movie.mkv            (main movie file)

/Movies/HD Movies/A_Movie.metathumb  (movie box art file)

/Movies/HD Movies/A_Movie.xml              (movie info)

/Movies/HD Movies/.A_Movie.backdrop    (folder containing backdrop artwork images)

 

In short I would like all xml & metathumb files to not be moved as well as any folder that starts with an . and ends in .backdrop

If possible I would only like those rules to apply to the Movies share and not my other share called Backup.

 

Many thanks in advance for any help.

 

Cheers

 

 

No - you cannot do what you want!

 

You have the ability to restrict shares to only be on the cache, but not to set up just specific files/folders within a share.

Make separate shares.

Making separate shares won't help. I have no way of setting a paths on the WDTV to have content on 1 share and the xml and thumbs on another.

 

My question was regards to the mover script. If it can't be tweaked then not a big deal and I can live with it.

 

Many thanks anyway.

 

 

Making separate shares won't help. I have no way of setting a paths on the WDTV to have content on 1 share and the xml and thumbs on another.

 

My question was regards to the mover script. If it can't be tweaked then not a big deal and I can live with it.

 

Many thanks anyway.

Having never actually tried to do what you want done, I'm speaking from theory, not practice.

 

You should be able to rewrite the mover script to do what you want, but it's not something that I've seen done, and would require time and effort for somebody to accomplish. If you don't have the skills to do it yourself, I suspect you could hire someone if it's worth that much to you.

Might not be that difficult since it uses rsync which has options to exclude files/folders. You can find the mover script with

which mover

/usr/local/sbin/mover

Google rsync exclude and you might be able to come up with something then if you get it working use the go file to overwrite the stock mover script when you boot.

 

Since the Mover uses rsync, you could add the following to the command and exclude the file types you want:  -- exclude "*.metathumb"  --exclude "*.xml" --exclude "*.backdrop"

 

I presume the space required for these files is not significant relative to the size of your cache drive.

 

 

Many thanks.

 

I tried the --exclude statement by creating an exclude-list.txt in /boot and added --exclude-from '/boot/exclude-list.txt' to the rsync command below. Now while that sort of works in that it does not copy the excluded files types to the main movie share on disks 1/2/3/4, it also does not leave them on the cache drive  . It just deletes them?

 

Any ideas how I can stop the excluded files from being deleted from the Cache drive?

 

Many thanks in advance.

 

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogt --exclude-from '/boot/exclude-list.txt' --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

Create a root directory on the cache that starts with a dot (i.e. ".nomove') and the move will ignore it and its contents.

Progress !!  But no cigar (yet) 

 

I know the basic functionality of the mover is to (a) copy the files in a cache-enabled share from the cache drive to the array and then delete them.    Clearly the excludes for rsync aren't applying to the subsequent delete command.    You'll need someone who's familiar with Linux scripting to provide the appropriate arguments to also exclude those files from the deletion.

 

While putting all the files you don't want moved in a separate folder on the cache won't work (for reasons you noted earlier), it does give a hint about a somewhat "kludgy" workaround:  You could alter the script to first copy all files ending the the suffixes you want to keep (.metathumb,  .xml, and .backdrop)  to a non-moveable folder on the cache;  then let the script run as is (with your exclusions);  and then move the contents of the non-moveable folder back to the original folder.

But surely there's a simple way to just exclude these files from the deletion.

 

 

This is essentially one user case of my exiting idea:

 

http://lime-technology.com/forum/index.php?topic=34434.0

Many thanks.

 

I tried the --exclude statement by creating an exclude-list.txt in /boot and added --exclude-from '/boot/exclude-list.txt' to the rsync command below. Now while that sort of works in that it does not copy the excluded files types to the main movie share on disks 1/2/3/4, it also does not leave them on the cache drive  . It just deletes them?

 

Any ideas how I can stop the excluded files from being deleted from the Cache drive?

 

Many thanks in advance.

 

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogt --exclude-from '/boot/exclude-list.txt' --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

 

 

I haven't see the mover script in years, but you can 'try' some tests if you are brave.

I.E. Remove the -delete and change the rsync and add --remove-sent-files.

This will have the effect of only deleting files that were rsynced.

I do not know what the effect will be by removing the -delete, you'll have to test that.

I'm guessing that empty directories will not be removed.

 

 

However you could pretty much duplicate the find line.

remove the -type -f  chunk

remove the -exec rsync line

so that the only thing being done is the find for an empty directory with -delete.

Many thanks.

 

I tried the --exclude statement by creating an exclude-list.txt in /boot and added --exclude-from '/boot/exclude-list.txt' to the rsync command below. Now while that sort of works in that it does not copy the excluded files types to the main movie share on disks 1/2/3/4, it also does not leave them on the cache drive  . It just deletes them?

 

Any ideas how I can stop the excluded files from being deleted from the Cache drive?

 

Many thanks in advance.

 

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogt --exclude-from '/boot/exclude-list.txt' --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

 

Instead of excluding files from the rsync, prevent them from being found. Then none of the following actions (-print,-exec rsync,-delete) will be performed. For example:

root@tower:/mnt/cache/iso# find . -depth \( \( -type f \! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print
./test.txt
./test.mkv
./test.thumb
./test.nfo
root@tower:/mnt/cache/iso# find . -depth \( \( -type f \! \( -name '*.thumb' -o -name '*.nfo' -o -exec fuser -s {} \; \) \) -o \( -type d -empty \) \) -print
./test.txt
./test.mkv

That is a much better way of dealing with it, however be careful of the syntax.

Many thanks guys for all the help. I decided to leave the main Mover script rsync code untouched so as not to impact the moving of any data copied to my other shares.

 

In the end I did this. Probably not the best or most efficient way, but it achieves what I want.

 

Modified the Mover script to:-

1) Back up my metadata from my Movie share into a hidden .Metadata folder in the root of the cache drive

2) Delete all the stuff on my Movie share I don't want moved to the array when mover runs

3) Let the original Mover rsync code run to move any new movies and data added to my other shares off the cache drive

4) Restore Movie metadata from .Metadata to my Movie share

 

I only have about 1.5GB of metadata and it takes about 90 seconds to do the above if there are no Movies to move. I know I am putting wear and tear on my SSD by doing this every 24 hours,but drives are cheap and the data is not critical.

 

 

# Only run script if cache disk enabled and in use
if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid

echo " "
echo "+=============================================================================="
echo "+ My mover started"
echo "+=============================================================================="
echo " "

echo "+=============================================================================="
echo "+ Backup movie metadata to .Metadata"
echo "+=============================================================================="
echo " "

rsync -av --exclude '*.mkv' --exclude '*.avi' --exclude '*.mp4' --exclude '*.wmv' --exclude '*.iso' --delete /mnt/user/Movies /mnt/cache/.Metadata

echo " "
echo "+=============================================================================="
echo "+ Delete movie metadata from Movie share"
echo "+=============================================================================="
echo " "

find /mnt/user/Movies/. -name '*.jpg' -delete

find /mnt/user/Movies/. -name '*.xml' -delete

find /mnt/user/Movies/. -name '*.bmk' -delete

find /mnt/user/Movies/. -name '*.metathumb' -delete

find /mnt/user/Movies/. -name 'flixster_lookup_cache*' -delete

find /mnt/user/Movies/. -name '.wd_tv' -delete

find /mnt/user/Movies/. -name '*.backdrop' -delete


echo " "
echo "+=============================================================================="
echo "+ Move content to array"
echo "+=============================================================================="
echo " "

cd /mnt/cache

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogt --exclude-from '/boot/exclude-list.txt' --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

echo " "
echo "+=============================================================================="
echo "+ Restore movie metadata to Movie share"
echo "+=============================================================================="
echo " "

rsync -av  /mnt/cache/.Metadata/Movies/ /mnt/user/Movies/

rm /var/run/mover.pid

echo " "
echo "+=============================================================================="
echo "+ My mover finished"
echo "+=============================================================================="

Many thanks guys for all the help. I decided to leave the main Mover script rsync code untouched so as not to impact the moving of any data copied to my other shares.

 

In the end I did this. Probably not the best or most efficient way, but it achieves what I want.

 

Modified the Mover script to:-

1) Back up my metadata from my Movie share into a hidden .Metadata folder in the root of the cache drive

2) Delete all the stuff on my Movie share I don't want moved to the array when mover runs

3) Let the original Mover rsync code run to move any new movies and data added to my other shares off the cache drive

4) Restore Movie metadata from .Metadata to my Movie share

 

I only have about 1.5GB of metadata and it takes about 90 seconds to do the above if there are no Movies to move. I know I am putting wear and tear on my SSD by doing this every 24 hours,but drives are cheap and the data is not critical.

 

 

# Only run script if cache disk enabled and in use
if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid

echo " "
echo "+=============================================================================="
echo "+ My mover started"
echo "+=============================================================================="
echo " "

echo "+=============================================================================="
echo "+ Backup movie metadata to .Metadata"
echo "+=============================================================================="
echo " "

rsync -av --exclude '*.mkv' --exclude '*.avi' --exclude '*.mp4' --exclude '*.wmv' --exclude '*.iso' --delete /mnt/user/Movies /mnt/cache/.Metadata

echo " "
echo "+=============================================================================="
echo "+ Delete movie metadata from Movie share"
echo "+=============================================================================="
echo " "

find /mnt/user/Movies/. -name '*.jpg' -delete

find /mnt/user/Movies/. -name '*.xml' -delete

find /mnt/user/Movies/. -name '*.bmk' -delete

find /mnt/user/Movies/. -name '*.metathumb' -delete

find /mnt/user/Movies/. -name 'flixster_lookup_cache*' -delete

find /mnt/user/Movies/. -name '.wd_tv' -delete

find /mnt/user/Movies/. -name '*.backdrop' -delete


echo " "
echo "+=============================================================================="
echo "+ Move content to array"
echo "+=============================================================================="
echo " "

cd /mnt/cache

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogt --exclude-from '/boot/exclude-list.txt' --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

echo " "
echo "+=============================================================================="
echo "+ Restore movie metadata to Movie share"
echo "+=============================================================================="
echo " "

rsync -av  /mnt/cache/.Metadata/Movies/ /mnt/user/Movies/

rm /var/run/mover.pid

echo " "
echo "+=============================================================================="
echo "+ My mover finished"
echo "+=============================================================================="

 

 

Sounds like it works fine ... you could have avoided the extra deletes by just doing what I had noted earlier ...

a somewhat "kludgy" workaround:  You could alter the script to first copy all files ending the the suffixes you want to keep (.metathumb,  .xml, and .backdrop)  to a non-moveable folder on the cache;  then let the script run as is (with your exclusions);  and then move the contents of the non-moveable folder back to the original folder.

 

You're doing the same initial thing (copying the metadata to another folder);  but then you're deleting all the stuff you don't want moved.    That isn't necessary -- the excludes you added earlier had the same impact, and since it then deleted all the files, everything was already cleaned up.    Then just moving the metadata back is the same as what I'd suggested.

 

But the good news is you've got it all working well.  By the way, I don't think this will have any notable impact on your SSD's longevity.  Modern SSD's have FAR better write cycle tolerance than those of just a few years ago.

 

That is true, but if I added the excludes then they would also apply to the data on my other shares. Also the way I am deleting is quick, leaving the excludes in like I had it it was much slower as the mover rsync script processes 1 files at a time in a loop (15 minutes to delete my metadata vs. a couple of seconds using find).

 

Either way does the job though. I have also added a back up routine to back up my metadata as well just in case my SDD dies.

 

 

 

It may be simpler to make 2 shares and then add links. The share with the movable items should be linked in the cache-only share. The share which is cache-only will be exported to the player. The files which need to be moved are accessible via the links. The share which is not cache-only need not be exported at all.