Best/fastest way to move data between unRAID servers?

[ThatDude]

Hi

 

I have an existing unRAID 5 server with 14TB of data in various users shares. I've just built a new unRAID 6 server and would like to copy the data across by the fastest means possible.

 

So far as I can see my options are either to use rsync or to mount the new server as a share on the old server and use CP to copy the data accross from usershare to usershare.

 

Is that correct or is the there a better/faster way to achieve this?

 

Many thanks,

 

Dan

 

 

 

 

[BRiT]

The only faster way is to move the actual data drives over, so its more of a motherboard/case/ram upgrade.

[ThatDude]

Thanks Brit, that's not really an option as I need to keep the old server running until the migration is complete.

 

I'll give rsync a try for a large usershare and then try NFS and CP. I'll report back with the speed of each, hopefully it'll help someone else out in the future.

[sureguy]

If you have checksums for your files you could do the transfer before assigning the parity disk.  I don't know if that will be faster as you then have to calculate parity and check your check sums.

[WeeboTech]

In your rsync server add these options before any path definitions.

 

...<snip>...
socket options  = SO_SNDBUF=524288,SO_RCVBUF=524288

[mnt]
        path = /mnt
        comment = /mnt files
        read only = FALSE
        list      = yes
...<snip>...

 

You can use the same options on the rsync client using rsync to rsync (do not use rsync over ssh if you can avoid it).

 

root@unRAIDb:/# rsync --help | grep sockopts

    --sockopts=OPTIONS      specify custom TCP options

 

So an example command might be.

 

cd /mnt/disk1

rsync -avXP --sockopts=SO_SNDBUF=524288,SO_RCVBUF=524288 . rsync://destinationhost/mnt/disk1

If you can disable the parity while the disks are mirrored (at least for the initial population) parity will not slow it down.

 

I get near line speed when doing it this way.

 

If you use the bunker or bitrot script to attach hashes to each file the -X will transfer those hashes to the destination.

You can then use the respective scripts to verify them.

[ThatDude]

Thank you WeeboTech, that's incredibly helpful.

 

One question, does rsync work in the same way on user shares? My shares are large and span multiple drives.

[Fireball3]

WeeboTech, can you elaborate on the figures you advise?

Are they valid for every configuration? (RAM, NIC)

[ThatDude]

This is the rsync command line that I'm using on my unRAID5 server to copy data to my unRAID6b14b server:

 

rsync -avhP --stats --progress --sockopts=SO_SNDBUF=524288,SO_RCVBUF=524288 /mnt/user/public/ rsync://tower/mnt/user/public

 

I'm achieving transfer speeds of around 75MB/s, is that good? I was expecting higher speeds (copying large ISO's / managed GB switch etc).

 

NB: Those sockopts that Weebo suggested made a HUGE difference, literally doubling the transfer speeds.

[Fireball3]

NB: Those sockopts that Weebo suggested made a HUGE difference, literally doubling the transfer speeds.

 

I'm sure it has an effect.

As always with such tweaks, the question is: what is the best value?

Is it already optimal?

[ThatDude]

NB: Those sockopts that Weebo suggested made a HUGE difference, literally doubling the transfer speeds.

 

I'm sure it has an effect.

As always with such tweaks, the question is: what is the best value?

Is it already optimal?

 

No I don't think it's optimal. Just working on the basis that I can transfer a file from either of my unRAID servers to my Mac via NFS at >100MB/s, I would expect a server to server rsync to be faster.

[Fireball3]

Are you bypassing the parity calculation on the receiving machine?

75MB/s is not too bad.

 

[ThatDude]

Yes I've disabled the parity drive, it was ~35MB/s with it enabled.

 

If I stop and start the sync the first ~30 seconds run at around 119MB/s before dropping to 75MB/s, I'm guessing this is the buffers filling up. I can see the destination server's drive light is idling every 4 seconds or so for around 3 seconds. I'm sure there's a way to optimise this further.

[ThatDude]

Small update: When I execute rsync on the new server and pull data from the old server I get a speed boost to just over 100MB/s.

 

I think that running rsync on the old server to push the data was creating some kind of CPU or RAM bottleneck (it's an old AMD x2 CPU with 2GB RAM).

 

The new server is an Intel QuadCore with 16GB RAM, this is the command line I'm executing to pull the datat:

 

rsync -avh --stats --progress --sockopts=SO_SNDBUF=524288,SO_RCVBUF=524288 rsync://oldnas/mnt/user/iso/ /mnt/user/iso

 

I'm sure that there is still room for improvement but breaking the 100MB/s barrier is good enough for my needs.

[WeeboTech]

I'm sure that there is still room for improvement but breaking the 100MB/s barrier is good enough for my needs.

 

That's about the top end unless you were to add compression, which may or may not help.

You could try disabling compression with the --skip-compress=LIST skip compressing files with suffix in LIST option.

 

compression is highly CPU dependent.

 

The closer you get to 90MB/s the better you are, 90-100MB/s is going to be the top end speed.

 

 

Frankly, I would avoid the user share if possible.

That's going to slow things down and disorganize the primary and backup servers if you have the same disks configuration.

 

You can schedule it overnight in cron very simply without interfering with day to day usage.

That's just me and my preference when doing these type of backups. I like to have data compartmentalized and managed well.

[ThatDude]

Thanks Weebo

 

User Shares are essential to my workflow, I can't imagine not using them.

 

I'm a commercial photographer IRL and I use unraid as my storage server for photo and video projects, I generate about 2TB of data per year so I just add drives as necessary. I then have a Windows box with CrashPlan installed which automagically takes all of that data and saves it in the cloud.

 

I'm only switching to a new server because I've run out of sata ports and space to put drives in my original server (9x2TB). I'm making the leap to unRAID6, a 4U rack server and 7x4TB drives, I think 24TB should keep me in storage for another couple of years :-)

[WeeboTech]

Do what works best for you.

 

 

If the two machines do not have duplicate configurations for disk layout, then by all means, use the usershare luke! 

[darkside40]

Could someone point me into the right direction how to temporary enable the rsync daemon on the receiving site?

[trurl]

As for rsync daemon, here is the guide I used. Weebotech put this up several years ago and it is still working for me on v6.

[darkside40]

Thanks for that!