calvados

Thanks for all your help as always @JorgeB. I really appreciate you taking the time to look at this.

It completed after quite some time. Restarting show slow transfers even on larger .iso or VM files.

Sorry for the multiple posts. I'm confused by what I'm seeing, and am not making any progress on solving this issue. 0%, and 0kbps for everything? WTH?

And now ~30 min later, speeds have picked up: Given I'm getting faster speeds with the UD share, does that provide any insight into why the SSH copy was so slow? Any thoughts on the inconsistency I'm seeing all over the place? Thanks again for all your help and insight as always @JorgeB, Cal.

Looks like I spoke too soon on my last post. It did start, and started with transfers at ~14MBps, but has promptly slowed down to ~2MBps, and is slowly going slower over time...Now at ~1.9MBPS:

Thanks. Tried that and am getting the same result: Is there anything in my network config near the bottom of my first post that is miss-configured?

Sorry, what is UD?

I ran iperf between the two servers. Looks like decent speeds for a 10gb connection? [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.44 Gbits/sec 24 515 KBytes [ 5] 1.00-2.00 sec 1.10 GBytes 9.43 Gbits/sec 8 549 KBytes [ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 520 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.41 Gbits/sec 0 535 KBytes [ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 532 KBytes [ 5] 5.00-6.00 sec 1.10 GBytes 9.42 Gbits/sec 0 526 KBytes [ 5] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 9 585 KBytes [ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 532 KBytes [ 5] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec 0 557 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.41 Gbits/sec 0 563 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.0 GBytes 9.42 Gbits/sec 41 sender [ 5] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec receiver iperf Done.

Thanks @JorgeB. I get the same issue without the z flag:

Hi @JorgeB, Yes I'm able to copy via windows from source to target via SMB shares. I neglected to note above, that both source and target datasets reside inside a zpool. Thanks @JorgeB, Cal.

Hi everyone, I'm running rsync on a target Unraid server to pull data from a source Unraid server. Running rsync once after a fresh reboot works without issue. Subsequent rsync calls slow down and stop at 0kbps. The source dataset is ~7tb without compression, and the target server has previously successfully completed full syncs, and is running zstd-9 compression. Both unraid servers are 6.12.14. Source is an i7-4770S w/32gb of DDR3, and target is a i5-13600k, with 96gb DDR5. Rsync is run on the target server. Rsync command: rsync -avzhP --info=progress2 root@server:/mnt/liveFiles/ /mnt/ur2pool/hotBackup/ This is what I see 20 min after running the above command after having completed the same call successfully earlier this evening: I see the following in the target server logs: Jan 2 00:12:19 ur2 unassigned.devices: Waiting 5 secs before mounting Remote Shares... Jan 2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered disabled state Jan 2 00:12:20 ur2 kernel: device vnet0 entered promiscuous mode Jan 2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered forwarding state Jan 2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered disabled state Jan 2 00:12:20 ur2 kernel: device vnet1 entered promiscuous mode Jan 2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered forwarding state Jan 2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered disabled state Jan 2 00:12:20 ur2 kernel: device vnet2 entered promiscuous mode Jan 2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered forwarding state Jan 2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered disabled state Jan 2 00:12:20 ur2 kernel: device vnet3 entered promiscuous mode Jan 2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered forwarding state Jan 2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered blocking state Jan 2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered disabled state Jan 2 00:12:21 ur2 kernel: device vnet4 entered promiscuous mode Jan 2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered blocking state Jan 2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered forwarding state Jan 2 00:12:28 ur2 kernel: x86/split lock detection: #AC: CPU 7/KVM/24553 took a split_lock trap at address: 0xfffff8047ce5701c Jan 2 00:14:31 ur2 kernel: x86/split lock detection: #AC: CPU 1/KVM/24482 took a split_lock trap at address: 0xfffff80404f2b303 Jan 2 00:14:34 ur2 kernel: x86/split lock detection: #AC: CPU 0/KVM/24546 took a split_lock trap at address: 0xfffff8047ce13be8 I see similar entries on the source target logs: r0: port 2(veth39c102c) entered blocking state Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered forwarding state Jan 2 00:03:27 ur kernel: eth0: renamed from veth2448e22 Jan 2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth39c102c: link becomes ready And a bit earlier on the source server I see the following, which is a bit concerning? Jan 2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth6e0b602: link becomes ready Jan 2 00:03:27 ur kernel: br-04a65ab93d16: port 1(veth6e0b602) entered blocking state Jan 2 00:03:27 ur kernel: br-04a65ab93d16: port 1(veth6e0b602) entered forwarding state Jan 2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): br-04a65ab93d16: link becomes ready Jan 2 00:03:27 ur rc.docker: swag: started succesfully! Jan 2 00:03:27 ur kernel: L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details. Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered blocking state Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered disabled state Jan 2 00:03:27 ur kernel: device veth39c102c entered promiscuous mode Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered blocking state Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered forwarding state Jan 2 00:03:27 ur kernel: eth0: renamed from veth2448e22 Jan 2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth39c102c: link becomes ready This is a 10gb connection that I used for a few years now in my previous server that is now installed on this new server. The source server also has a 10gb connection and they are connected via a Mikrotik CRS326-24G-2S+RM that again, has been around for a few years. Both servers are in the same vlan. The source server has not been modified in quite some time, asides from regular docker and Unraid updates. No hardware changes there. The target server is new, but is using the same 10gb SFP+ card that was moved across from the old server this new one replaces. I've only recently starting using rsync, so I'm unsure if this same issue would have presented on the old target server. Both target and source server diagnostics are attached. Any idea what's going on here, or how to be able to run rsync subsequent times without needing a full server reboot inbetween rsync calls? Thanks everyone, Cal. UPDATE: Ok, even a reboot isn't solving it any longer. I'm starting to think this may be a physical problem. This is 5 min after calling rsync after a fresh reboot of both source and target servers, and a reboot of the Mikrotik switch, and ER-x router: And 5 min after that screenshot: Those two files being synced are the two diag files attached to this post. Subsequent attempts without rebooting anything are even worse: Target network: Source network: Any thoughts as to how to resolve this? Thanks everyone, Cal.

Thank again @bmartino1. I'll give this a try and let you know how it goes.

Thanks @bmartino1. I'm currently running Unraid 6.12.14. I'm in the middle of migrating to a new server, so I have a chance to create a brand new zpool, so based on what you're saying I should be able to, from the terminal, create the zpool with 2 vdevs, one with raidz1, and one with raidz2. I have other backups in place, and am comfortable with the risk associated with having one of the vdevs being raidz1. Would I do this simply by running the following? zpool create -O casesensitivity=insensitive -m /mnt/ur2pool ur2pool raidz1 {4 drives here} zpool create -O casesensitivity=insensitive -m /mnt/ur2pool ur2pool raidz2 {4 drives here} Then creating the dataset within the zpool as normal? Thanks again @bmartino1, Cal.

Thanks @JorgeB. I don't use the traditional unraid shares. I have them turned off as per the screenshot below. If I'm understanding what you're saying, I could do it, (use a raidz1 vdev and a raidz2 vdev in the same zpool), via terminal and it would work? Could I automate the import in the go file so the mixed vdev zpool is online after a reboot? Do you happen to know if this would work in the upcoming Unraid 7? Thanks again @JorgeB, Cal.

Follow up question to my post above that is slightly different in nature, hence a separate post: Is it possible to add a 2nd vdev to an existing zpool with a different raidz level? e.g. adding a 4 disk wide raidz1 vdev to an existing pool made up of a 4 disk wide raidz2 vdev? I'm currently running 6.12.14. If this is not possible in 6.12.14, would it be possible in the upcoming version 7? Thanks again everyone, Cal.