All:

 

I recently built a new unRaid server to replace my existing 11TB one.  What is the fastest way to copy from one server to the other?  I am thinking I may need to smb mount from unRaid to unRaid and rsync?  Is this supported?

 

It is my understanding that copying via a windows client will be slower because it copies from unRaid to Windows to unRaid?  is that correct?

This is how I did it (where T2 is my old server):

 

mkdir /mnt/T2

mount -o ro,nolock,tcp 192.168.0.2:/mnt/user/Movies /mnt/T2

 

Then mc (Midnight Commander) to copy stuff.

Something tells me rsync without samba will be faster.

But as you probably know you'd have to set up /etc/rsync.conf and start the rsync daemon on one of them.

If your disks are mostly full, you can move them all to the new server, remove one, let system 'Disable' it, replace with new drive, and then let system re-build onto new drive.  Do this for each drive

Another option is install ssh package (unmenu has it) on both, enable root login on the old on, then, being on the new server, run the following:

 

ssh old_server tar -C /mnt/user -cf - . | tar -C /mnt/user -xf -

 

This will probably faster than any other methods.

 

To enable root login:

 

sed -i -e 's/#PermitEmptyPasswords.*/PermitEmptyPasswords yes/' /etc/ssh/sshd_config; pkill -1 sshd

 

 

direct rsync to an rsync --server would be fast.

It is a socket to socket connection without any intermediate protocols.

 

The tar over ssh does work, but there's other protocols and windowing there.

if the socket drops for any reason you have to start over.

 

with rsync -avP (source) rsync://server/sharename/directory

it;s direct. If there is a failure, you will only move that which is missing or changed.

 

Tom's method of just moving the drive will probably be the fastest. ;-)

 

With the rsync, once you set it up, it can be a cron job that does it disk by disk automatically on schedule.

 

 

 

The tar over ssh does work, but there's other protocols and windowing there.

if the socket drops for any reason you have to start over.

 

........

 

Tom's method of just moving the drive will probably be the fastest. ;-)

 

......

 

 

We are tailking about LAN connection here. Or you mean somebody trips over the network cable?

I know ssh is picky to the network and will drop connection if it detects a corrupted MAC (ssh specific checksum).

But if you see this you better look at your NIC/switches since it would likely be an indication ofi a hardware problem.

 

As for the swapping drives, I personally doubt it will be the fastest method, just because you spend time (and effort) to manipulate the drives

Unless you have very few of them.

 

I've used ssh-tar-untar to transfer hundreds of gigabytes (not specifically with unRAID though) and never had problems with it.

 

 

The tar over ssh does work, but there's other protocols and windowing there.

if the socket drops for any reason you have to start over.

 

 

We are tailking about LAN connection here. Or you mean somebody trips over the network cable?

I know ssh is picky to the network and will drop connection if it detects a corrupted MAC (ssh specific checksum).

But if you see this you better look at your NIC/switches since it would likely be an indication ofi a hardware problem.

 

 

If there are hundreds of small files, then it's going to take allot of time to create them.

As your drive fills up, the time to create the directory entries and free space takes longer and longer which leads to timeouts.

 

With rsync direct (or over ssh), if there is any kind of failure, you can pick up from where you left off at.

 

With tar over ssh, if there is a break for any reason (who knows) then the procedure will have to be started over again from scratch.

 

Also, I think rsync creates all of the directories first.

On some file systems this has the advantage of putting them near the start of the file system this making it faster to look up files in the future.

 

 

Create an /etc/rsyncd.conf as in this example.

 

uid             = root
gid             = root
use chroot      = no
max connections = 4
pid file        = /var/run/rsyncd.pid
timeout         = 600

[mnt]
    path = /mnt
    comment = /mnt files
    read only = FALSE

 

run /usr/bin/rsync --daemon on the destination server.

 

On the source server, I believe the syntax is

 

root@Atlas ~ #cd /mnt/disk1

root@Atlas /mnt/disk1 #rsync -avP . rsync://servername/mnt/disk1

 

Someone correct me if I'm wrong, right now I do not have two servers running to test this.

 

add -n to the switches just to test it out first (dry run).

 

 

You can also add something like

 

--bwlimit=10240 to limit the throughput to keep the transmission going smoothly.. or you could let it go full speed.

You will see lots of pauses then pickups, but it will still run.

 

If the transfer breaks for any reason, run the same command.

 

This is meant for a local lan. It is not secure over the internet.

rsync can also be done over ssh which would be secure but there would be other protocol overhead.

 

FWIW try this out on a small disk/directory first just to be sure the suggested syntax is correct.

I wrote this from memory.

 

In response to one of the original questions

 

>> It is my understanding that copying via a windows client will be slower because it copies from unRaid to Windows to unRaid?  is that correct?

 

This is correct.

 

 

The tar over ssh does work, but there's other protocols and windowing there.

if the socket drops for any reason you have to start over.

 

 

We are tailking about LAN connection here. Or you mean somebody trips over the network cable?

I know ssh is picky to the network and will drop connection if it detects a corrupted MAC (ssh specific checksum).

But if you see this you better look at your NIC/switches since it would likely be an indication ofi a hardware problem.

 

 

If there are hundreds of small files, then it's going to take allot of time to create them.

As your drive fills up, the time to create the directory entries and free space takes longer and longer which leads to timeouts.

 

 

 

Nothing can beat tar when it comes to handling large directory trees. It streams them really well, especially over a pipeline.

In case of many files it aggregates the I/O and does not perform intensive operations on directory trees.

No doubt rsync will beat it on the second run (on an existing tree), but this is not the case we discuss here.

And to reduce ssh overhead on encryption you could use blowfish cypher (-c blowfish).

Though these day you would not likely notice a big difference in the speed.