Hi everyone,

 

I'm running rsync on a target Unraid server to pull data from a source Unraid server.  Running rsync once after a fresh reboot works without issue.  Subsequent rsync calls slow down and stop at 0kbps.

 

The source dataset is ~7tb without compression, and the target server has previously successfully completed full syncs, and is running zstd-9 compression.

 

Both unraid servers are 6.12.14.  Source is an i7-4770S w/32gb of DDR3, and target is a i5-13600k, with 96gb DDR5.

 

Rsync is run on the target server.

 

Rsync command:

rsync -avzhP --info=progress2 root@server:/mnt/liveFiles/ /mnt/ur2pool/hotBackup/

 

This is what I see 20 min after running the above command after having completed the same call successfully earlier this evening:

 

 

 

I see the following in the target server logs:

Jan  2 00:12:19 ur2 unassigned.devices: Waiting 5 secs before mounting Remote Shares...
Jan  2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered blocking state
Jan  2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered disabled state
Jan  2 00:12:20 ur2 kernel: device vnet0 entered promiscuous mode
Jan  2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered blocking state
Jan  2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered forwarding state
Jan  2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered blocking state
Jan  2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered disabled state
Jan  2 00:12:20 ur2 kernel: device vnet1 entered promiscuous mode
Jan  2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered blocking state
Jan  2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered forwarding state
Jan  2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered blocking state
Jan  2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered disabled state
Jan  2 00:12:20 ur2 kernel: device vnet2 entered promiscuous mode
Jan  2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered blocking state
Jan  2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered forwarding state
Jan  2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered blocking state
Jan  2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered disabled state
Jan  2 00:12:20 ur2 kernel: device vnet3 entered promiscuous mode
Jan  2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered blocking state
Jan  2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered forwarding state
Jan  2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered blocking state
Jan  2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered disabled state
Jan  2 00:12:21 ur2 kernel: device vnet4 entered promiscuous mode
Jan  2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered blocking state
Jan  2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered forwarding state
Jan  2 00:12:28 ur2 kernel: x86/split lock detection: #AC: CPU 7/KVM/24553 took a split_lock trap at address: 0xfffff8047ce5701c
Jan  2 00:14:31 ur2 kernel: x86/split lock detection: #AC: CPU 1/KVM/24482 took a split_lock trap at address: 0xfffff80404f2b303
Jan  2 00:14:34 ur2 kernel: x86/split lock detection: #AC: CPU 0/KVM/24546 took a split_lock trap at address: 0xfffff8047ce13be8

 

I see similar entries on the source target logs:

 

r0: port 2(veth39c102c) entered blocking state
Jan  2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered forwarding state
Jan  2 00:03:27 ur kernel: eth0: renamed from veth2448e22
Jan  2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth39c102c: link becomes ready

 

And a bit earlier on the source server I see the following, which is a bit concerning?

Jan  2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth6e0b602: link becomes ready
Jan  2 00:03:27 ur kernel: br-04a65ab93d16: port 1(veth6e0b602) entered blocking state
Jan  2 00:03:27 ur kernel: br-04a65ab93d16: port 1(veth6e0b602) entered forwarding state
Jan  2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): br-04a65ab93d16: link becomes ready
Jan  2 00:03:27 ur rc.docker: swag: started succesfully!
Jan  2 00:03:27 ur kernel: L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
Jan  2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered blocking state
Jan  2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered disabled state
Jan  2 00:03:27 ur kernel: device veth39c102c entered promiscuous mode
Jan  2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered blocking state
Jan  2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered forwarding state
Jan  2 00:03:27 ur kernel: eth0: renamed from veth2448e22
Jan  2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth39c102c: link becomes ready

 

This is a 10gb connection that I used for a few years now in my previous server that is now installed on this new server.  The source server also has a 10gb connection and they are connected via a Mikrotik CRS326-24G-2S+RM that again, has been around for a few years.  Both servers are in the same vlan.

 

The source server has not been modified in quite some time, asides from regular docker and Unraid updates.  No hardware changes there.  The target server is new, but is using the same 10gb SFP+ card that was moved across from the old server this new one replaces. 

 

I've only recently starting using rsync, so I'm unsure if this same issue would have presented on the old target server.

 

Both target and source server diagnostics are attached.

 

Any idea what's going on here, or how to be able to run rsync subsequent times without needing a full server reboot inbetween rsync calls?

 

Thanks everyone,

Cal.

 

UPDATE:  Ok, even a reboot isn't solving it any longer.  I'm starting to think this may be a physical problem.  This is 5 min after calling rsync after a fresh reboot of both source and target servers, and a reboot of the Mikrotik switch, and ER-x router:

 

 

 

And 5 min after that screenshot:

 

 

 

Those two files being synced are the two diag files attached to this post.

 

Subsequent attempts without rebooting anything are even worse:

 

 

 

Target network:

 

 

 

Source network:

 

 

 

Any thoughts as to how to resolve this?

 

Thanks everyone,

Cal.

 

 

 

Edited January 5, 20251 yr by calvados

If you try using for exaple windows explorer to copy the data from one server to the other, is it successful?

7 hours ago, JorgeB said:

If you try using for exaple windows explorer to copy the data from one server to the other, is it successful?

Hi @JorgeB,

 

Yes I'm able to copy via windows from source to target via SMB shares.  I neglected to note above, that both source and target datasets reside inside a zpool.

 

Thanks @JorgeB,

Cal.

Try rsync without the z flag, compression is only useful for very low bandwidth scenarios, though it should never cause such a slowdown, but worth trying.

Thanks @JorgeB.  I get the same issue without the z flag:

 

 

Edited January 5, 20251 yr by calvados

I ran iperf between the two servers.  Looks like decent speeds for a 10gb connection?

 

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.44 Gbits/sec   24    515 KBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.43 Gbits/sec    8    549 KBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0    520 KBytes
[  5]   3.00-4.00   sec  1.09 GBytes  9.41 Gbits/sec    0    535 KBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0    532 KBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0    526 KBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec    9    585 KBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0    532 KBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0    557 KBytes
[  5]   9.00-10.00  sec  1.09 GBytes  9.41 Gbits/sec    0    563 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.42 Gbits/sec   41             sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec                  receiver

iperf Done.

 

Edited January 5, 20251 yr by calvados

Not really sure what it could be if SMB works fine, try rsync with the other server's share mounted over SMB with UD, to rule out any SSH issues.

1 minute ago, JorgeB said:

Not really sure what it could be if SMB works fine, try rsync with the other server's share mounted over SMB with UD, to rule out any SSH issues.

Sorry, what is UD?

The Unassigned Devices plugin, you can use it to mount an NFS or SMB share on the other server and then use the remote path with rsync, it will be like:

 

rsync -av /mnt/remotes/UD_share_name/ /mnt/path/to/destination/

 

On 1/2/2025 at 10:10 AM, JorgeB said:

The Unassigned Devices plugin, you can use it to mount an NFS or SMB share on the other server and then use the remote path with rsync, it will be like:

 

rsync -av /mnt/remotes/UD_share_name/ /mnt/path/to/destination/

 

Thanks.  Tried that and am getting the same result:

 

 

 

Is there anything in my network config near the bottom of my first post that is miss-configured?

Edited January 5, 20251 yr by calvados

Sounds like an rsync problem with that data, try using a different util to copy

Looks like I spoke too soon on my last post.  It did start, and started with transfers at ~14MBps, but has promptly slowed down to ~2MBps, and is slowly going slower over time...Now at ~1.9MBPS:

 

And now ~30 min later, speeds have picked up: 

 

 

Given I'm getting faster speeds with the UD share, does that provide any insight into why the SSH copy was so slow?  Any thoughts on the inconsistency I'm seeing all over the place?

 

Thanks again for all your help and insight as always @JorgeB,

Cal.

Sorry for the multiple posts.  I'm confused by what I'm seeing, and am not making any progress on solving this issue.  0%, and 0kbps for everything?  WTH?

 

Is the copy still going? Very small files can show basically null stats

 

 

It completed after quite some time.  Restarting show slow transfers even on larger .iso or VM files. 

Sorry, no more ideas.

Thanks for all your help as always @JorgeB.  I really appreciate you taking the time to look at this.