Is there a way, via command line, to move a massive amount of data from one share to another. I do not care what physical disk the data is on.

 

I am consolidating all my media into a 'Media' folder and am starting with my movies. I have a share of Movies with a split level of 1 that I want to move into my Media share with a split level of 2. So the Folder structure would appear as /Media/Movies. Music, TV Shows, etc to come later.

 

The caveat is it has to be a one shot deal. No duplicates as there isn't enough room to have 2 copies of ALL my movies.

 

Up to this point I have been moving from folder A to folder B, in small chunks, via copy/ paste and deleting the moved files from folder A once complete. Slow going to say the least.... and I have to constantly monitor it.

 

TIA!

log in via telnet using putty and type mc at the prompt.

 

You can move and select till your heart is content.  It may take a little getting use to but you can do what you need to do via mc (midnight commander).

I have a command line shell that I use to move files with rsync.

 

It's called rsyncmvd

 

rsync - mv -delete.

I wrote this so I could have a controlled speed and priority of moving files between disks.

It can be between any source or destination.

 

The only difference is the destination directory is first.

 

example:

rsyncmvd /mnt/disk1/someshare  /mnt/disk2/data1/ /mnt/disk3/data2/  

 

This lets me do something like.

 

rsyncmvd /mnt/disk1/towatch *.iso

 

You can set the BWLIMIT to any default you like.

When my system is busy I do 8192 like this.

 

BWLIMIT=8192 rsyncmvd /mnt/disk1/towatch *.iso

 

#!/bin/sh

if [ -z "${1}" ] 
  then echo "$0: Usage [destination dir] file[s]..."
       exit
fi

DIR="$1"
shift

if [ ! -d "${DIR}" ]
  then echo "$0: argumment 1:${DIR} is not a directory"
       exit
fi

if [ -x /usr/bin/nice ]; then
  NICE="/usr/bin/nice -19"
fi

if [ -x /usr/bin/ionice ]; then
  IONICE="/usr/bin/ionice -c3"
fi

exec ${NICE} ${IONICE} rsync -avP --remove-sent-files --bwlimit=${BWLIMIT:=30000} "$@" "${DIR}" 
exec ${NICE} ${IONICE} rsync -avP --remove-sent-files "$@" "${DIR}"

I have a command line shell that I use to move files with rsync.

 

It's called rsyncmvd

 

rsync - mv -delete.

I wrote this so I could have a controlled speed and priority of moving files between disks.

It can be between any source or destination.

 

The only difference is the destination directory is first.

 

example:

rsyncmvd /mnt/disk1/someshare  /mnt/disk2/data1/ /mnt/disk3/data2/  

 

This lets me do something like.

 

rsyncmvd /mnt/disk1/towatch *.iso

 

You can set the BWLIMIT to any default you like.

When my system is busy I do 8192 like this.

 

BWLIMIT=8192 rsyncmvd /mnt/disk1/towatch *.iso

 

#!/bin/sh

if [ -z "${1}" ] 
  then echo "$0: Usage [destination dir] file[s]..."
       exit
fi

DIR="$1"
shift

if [ ! -d "${DIR}" ]
  then echo "$0: argumment 1:${DIR} is not a directory"
       exit
fi

if [ -x /usr/bin/nice ]; then
  NICE="/usr/bin/nice -19"
fi

if [ -x /usr/bin/ionice ]; then
  IONICE="/usr/bin/ionice -c3"
fi

exec ${NICE} ${IONICE} rsync -avP --remove-sent-files --bwlimit=${BWLIMIT:=30000} "$@" "${DIR}" 
exec ${NICE} ${IONICE} rsync -avP --remove-sent-files "$@" "${DIR}"

What is the purpose of the second rsync line? (the one without the bwlimit option)  It seems identical otherwise?

 

Joe L.

What is the purpose of the second rsync line? (the one without the bwlimit option)  It seems identical otherwise?

 

Joe L.

 

When I want, I eliminate the bwlimit by moving one line above the other.

The second one never gets executed because the first one is exec'ed and replaces the current process.

 

I use it sometimes to benchmark my system in moving stuff as fast as possible without any throttling.

It was how I determined the kernel tuning issues and how to beef up the cache tunings for my system to function at maximum throughput.

What is the purpose of the second rsync line? (the one without the bwlimit option)  It seems identical otherwise?

 

Joe L.

 

When I want, I eliminate the bwlimit by moving one line above the other.

The second one never gets executed because the first one is exec'ed and replaces the current process.

 

I use it sometimes to benchmark my system in moving stuff as fast as possible without any throttling.

It was how I determined the kernel tuning issues and how to beef up the cache tunings for my system to function at maximum throughput.

I saw the exec, but did not think of the consequences.  You are correct, the second line will not be executed.

Thanks Weebo, I am going to try that our this weekend!