Hello –

 

Anyone have experience with getting rsync to use a dedicated NIC  to another dedicated NIC (rsync-ing peer-to-peer)?

 

Configuration:

unRAID Server #1 – basic Gb network (eth0 & eth1 bonded) with a 192.168.1.x schema with a gateway

AND a 10Gbe NIC with a 10.10.10.X schema and NO gateway.

 

unRAID Server #2 (same config as above) with different values for “X” of course.

 

My goal is to get rsync to exclusively use the 10Gbe NIC’s to do my backups. So, the rsyc action would only occur on the 10.10.10.X network.

 

----------------------------------------------------------------------------------------------

Here is what I have been messing around with:

rsync -u -v -r -P --address=10.10.10.221:/mnt/user/* 10.10.10.225:/mnt/user/vault

---------------------------------------------------------------------------------------------

 

If anyone can help me get this first part to work, I can figure there rest out from other posts I’ve read so far.

 

Thanks in advance everyone!

You just need to ssh to the dest address using the 10GbE IP, e.g., this is what I used:

rsync -av /mnt/path/to/source/ 10.10.10.1:/mnt/path/to/dest/

 

Note however that rsync isn't really made for performance, I could never get it over 150MB/s, even using faster ssh ciphers, that's one of the reasons I stopped using it.

What did you replace it with?

How fast can you transfer now?

Sent from my chisel, carved into granite

I'm using btrfs send/receive, get up to 300MB/s, also has other advantages for me, like only metadata is updated on for example a folder rename/move, though obviously only works for btrfs filesystems.

Hi @johnnie.black,

 

Appreciate the reply. And @tr0910 beat me to the obvious question!

 

I did a bunch of digging on the whole rsync backing up, including the post by tr0910 and his remote to China (there has to be a story there)

 

I, more or less, get the concept of SSH, but have not used it yet.

So, let me start by asking: does SSH-ing in 'make the connection' between the two machines?

In other words, does using SSH give me the ability to specify that I want to connect from NIC-A (local) to NIC-B (remote)?

2 minutes ago, TechMed said:

So, let me start by asking: does SSH-ing in 'make the connection' between the two machines?

When using an IP address on the command line rsync will default to ssh connection, and as long as you use the correct destination 10GbE IP it will use that NIC, and use it also on source since normally it's the only one that can connect to the other.

 

 

3 minutes ago, johnnie.black said:

rsync will default to ssh connection

Okay, I thought so.

 

I did not bring any of this up (in original post) as I wanted to make sure I had the basics down.

 

I did lay my rsync string out (very similar to what you suggested) with the assumption data would flow OUT the 10Gbe as it was the only NIC with that addressing schema. It just did not seem like it was working as the speed was atrociously slow.

 

So, based on what your saying...

On 3/20/2019 at 3:45 AM, johnnie.black said:

I could never get it over 150MB/s

The 150 MB/s is the best I can hope for?

 

 

I could never get it faster than that, and after googling it found various users complaining of the same, only way I found to speed it up was by running multiple instances, but that's not a good option with Unraid because of how parity works.

<Sigh>

 

I've been looking around too and cannot find an alternative backup solution for unRAID.

I went down this whole 10Gbe road just to try to p/u some speed (have about 20Tb I need to backup).

I couldn't be more disappointed; I wish I would have saved my money.

 

I really appreciate your chatting with me about it.

Props Bud... you help a lot of people. Thanks!

 

46 minutes ago, TechMed said:

I went down this whole 10Gbe road just to try to p/u some speed (have about 20Tb I need to backup).

If this is for the initial backup you could disable parity and rsync multiple disks at the same time, I did this when I switched from rsync to send/receive, I could do 3 to 4 disks at the same time, it maxed out at around 375MB/s for multiple transfers, in my case it appeared to be the CPUs couldn't handle more, still initial backup was much faster, now with parity it it's only worth doing one disk at a time, but incremental backups are much smaller so I can live with that.

 

You mention

5 minutes ago, johnnie.black said:

I could do 3 to 4 disks

I get the disabling Parity, but you mention "disks" instead of files/folders.

 

If you have the time, would mind elaborating or pointing me to a post(s) that explain?

I was using btrfs send/receive from disk to disk, but you can do the same with rsync, disk shares must be enable, then e.g.

 

rsync -av --progress /mnt/disk1/share/ 10.10.10.1:/mnt/disk1/share/

 

got it... thanks!

@johnnie.black ran a test and had an interesting outcome.

Any thoughts on why I went from +/- 115Mb/s to 50Mb/s?

Thermal?

Test bkup disk is an - HGST HUS726060ALA640 Ultrastar He6 (Helium Platform) 6TB 7200RPM 64MB Cache SATA 6.0Gb/s

NO cache, but parity is still enabled...

64GB ram and processors did not seem to be working hard at all.

Just now, TechMed said:

Any thoughts on why I went from +/- 115Mb/s to 50Mb/s?

That's about right with parity enable and default writing mode, it should be faster with turbo write (only relevant while using parity)

Hmm okay.

So to disable parity, temporarily, just stop the array, 'remove' the parity drive (remove it from the current config), restart the array, ignore the warnings and roll with a test?

Do you think that will at least get me a steady +/- 115Mb/s?

 

After the test just re-enable parity by stopping the array, 're-installing' the parity drive (adding it back to the config), restarting the array and let parity run (for days and days and days... 😵)

 

**I am being verbose in case someone else needs this info**

6 minutes ago, TechMed said:

So to disable parity, temporarily, just stop the array, 'remove' the parity drive (remove it from the current config), restart the array, ignore the warnings and roll with a test?

Yes.

 

6 minutes ago, TechMed said:

Do you think that will at least get me a steady +/- 115Mb/s?

It should, unless you're using very old and slow disks.

 

7 minutes ago, TechMed said:

After the test just re-enable parity by stopping the array, 're-installing' the parity drive (adding it back to the config), restarting the array and let parity run

Yes, just re-assign it and start the array to begin the parity sync.

Cool. Thanks!

I'll give it a go...

 

3 hours ago, TechMed said:

I did a bunch of digging on the whole rsync backing up, including the post by tr0910 and his remote to China (there has to be a story there)

Not much of a story, other than I started using unRaid for storing massive raw camera image data, and got so accustomed to it that I decided to use it in a business solution where web server backups needed to flow internationally.  (give a bloke a hammer and from then on, every problem is a nail)

 

I don't doubt that other solutions might be better for that use case, however unRaid was already in place and it scaled to meet the need.  The only problem I have is that the community of awesome volunteers here are expert on media hoarding, but use cases like mine are on the fringe to the general community.  It wouldn't matter so much if I were a bit more of a Linux gearhead.  I only hope my adventures might spur some more capable coders to clean up my mess and make something that isn't so hacked together and difficult to replicate for future users.

Hi @tr0910 - it is a story nonetheless and I thank you for sharing it (and all of your rsync work). Who knows, in some far off day maybe many others will discover the utility of unRAID for something other than being a media server. At present, I am guilty of the latter, but do have designs on being a "Contributor" someday!

 

 

@johnnie.black - Worked like a charm!

One thing was rather concerning though, during the 'sync' CPU 8 and 12, out of 24, were pegged at 100% usage while the others were only 10-20% usage.

Normal? Is there a way to spread the love around?

 

(And for those wondering, yes, I DO own the discs! )

 

27 minutes ago, TechMed said:

Normal?

Yep, not much you can do about that.

Good to know.

Looked at the 'parallel' command, but that is a bit over my head so I will stick to the basics for now.

Thanks again!