calvados

Thanks for all your help as always @JorgeB. I really appreciate you taking the time to look at this.

It completed after quite some time. Restarting show slow transfers even on larger .iso or VM files.

Sorry for the multiple posts. I'm confused by what I'm seeing, and am not making any progress on solving this issue. 0%, and 0kbps for everything? WTH?

And now ~30 min later, speeds have picked up: Given I'm getting faster speeds with the UD share, does that provide any insight into why the SSH copy was so slow? Any thoughts on the inconsistency I'm seeing all over the place? Thanks again for all your help and insight as always @JorgeB, Cal.

Looks like I spoke too soon on my last post. It did start, and started with transfers at ~14MBps, but has promptly slowed down to ~2MBps, and is slowly going slower over time...Now at ~1.9MBPS:

Thanks. Tried that and am getting the same result: Is there anything in my network config near the bottom of my first post that is miss-configured?

Sorry, what is UD?

I ran iperf between the two servers. Looks like decent speeds for a 10gb connection? [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.44 Gbits/sec 24 515 KBytes [ 5] 1.00-2.00 sec 1.10 GBytes 9.43 Gbits/sec 8 549 KBytes [ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 520 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.41 Gbits/sec 0 535 KBytes [ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 532 KBytes [ 5] 5.00-6.00 sec 1.10 GBytes 9.42 Gbits/sec 0 526 KBytes [ 5] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 9 585 KBytes [ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 532 KBytes [ 5] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec 0 557 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.41 Gbits/sec 0 563 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.0 GBytes 9.42 Gbits/sec 41 sender [ 5] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec receiver iperf Done.

Thanks @JorgeB. I get the same issue without the z flag:

Hi @JorgeB, Yes I'm able to copy via windows from source to target via SMB shares. I neglected to note above, that both source and target datasets reside inside a zpool. Thanks @JorgeB, Cal.

Hi everyone, I'm running rsync on a target Unraid server to pull data from a source Unraid server. Running rsync once after a fresh reboot works without issue. Subsequent rsync calls slow down and stop at 0kbps. The source dataset is ~7tb without compression, and the target server has previously successfully completed full syncs, and is running zstd-9 compression. Both unraid servers are 6.12.14. Source is an i7-4770S w/32gb of DDR3, and target is a i5-13600k, with 96gb DDR5. Rsync is run on the target server. Rsync command: rsync -avzhP --info=progress2 root@server:/mnt/liveFiles/ /mnt/ur2pool/hotBackup/ This is what I see 20 min after running the above command after having completed the same call successfully earlier this evening: I see the following in the target server logs: Jan 2 00:12:19 ur2 unassigned.devices: Waiting 5 secs before mounting Remote Shares... Jan 2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered disabled state Jan 2 00:12:20 ur2 kernel: device vnet0 entered promiscuous mode Jan 2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 2(vnet0) entered forwarding state Jan 2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered disabled state Jan 2 00:12:20 ur2 kernel: device vnet1 entered promiscuous mode Jan 2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 3(vnet1) entered forwarding state Jan 2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered disabled state Jan 2 00:12:20 ur2 kernel: device vnet2 entered promiscuous mode Jan 2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 4(vnet2) entered forwarding state Jan 2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered disabled state Jan 2 00:12:20 ur2 kernel: device vnet3 entered promiscuous mode Jan 2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered blocking state Jan 2 00:12:20 ur2 kernel: br0: port 5(vnet3) entered forwarding state Jan 2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered blocking state Jan 2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered disabled state Jan 2 00:12:21 ur2 kernel: device vnet4 entered promiscuous mode Jan 2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered blocking state Jan 2 00:12:21 ur2 kernel: br0: port 6(vnet4) entered forwarding state Jan 2 00:12:28 ur2 kernel: x86/split lock detection: #AC: CPU 7/KVM/24553 took a split_lock trap at address: 0xfffff8047ce5701c Jan 2 00:14:31 ur2 kernel: x86/split lock detection: #AC: CPU 1/KVM/24482 took a split_lock trap at address: 0xfffff80404f2b303 Jan 2 00:14:34 ur2 kernel: x86/split lock detection: #AC: CPU 0/KVM/24546 took a split_lock trap at address: 0xfffff8047ce13be8 I see similar entries on the source target logs: r0: port 2(veth39c102c) entered blocking state Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered forwarding state Jan 2 00:03:27 ur kernel: eth0: renamed from veth2448e22 Jan 2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth39c102c: link becomes ready And a bit earlier on the source server I see the following, which is a bit concerning? Jan 2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth6e0b602: link becomes ready Jan 2 00:03:27 ur kernel: br-04a65ab93d16: port 1(veth6e0b602) entered blocking state Jan 2 00:03:27 ur kernel: br-04a65ab93d16: port 1(veth6e0b602) entered forwarding state Jan 2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): br-04a65ab93d16: link becomes ready Jan 2 00:03:27 ur rc.docker: swag: started succesfully! Jan 2 00:03:27 ur kernel: L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details. Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered blocking state Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered disabled state Jan 2 00:03:27 ur kernel: device veth39c102c entered promiscuous mode Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered blocking state Jan 2 00:03:27 ur kernel: docker0: port 2(veth39c102c) entered forwarding state Jan 2 00:03:27 ur kernel: eth0: renamed from veth2448e22 Jan 2 00:03:27 ur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth39c102c: link becomes ready This is a 10gb connection that I used for a few years now in my previous server that is now installed on this new server. The source server also has a 10gb connection and they are connected via a Mikrotik CRS326-24G-2S+RM that again, has been around for a few years. Both servers are in the same vlan. The source server has not been modified in quite some time, asides from regular docker and Unraid updates. No hardware changes there. The target server is new, but is using the same 10gb SFP+ card that was moved across from the old server this new one replaces. I've only recently starting using rsync, so I'm unsure if this same issue would have presented on the old target server. Both target and source server diagnostics are attached. Any idea what's going on here, or how to be able to run rsync subsequent times without needing a full server reboot inbetween rsync calls? Thanks everyone, Cal. UPDATE: Ok, even a reboot isn't solving it any longer. I'm starting to think this may be a physical problem. This is 5 min after calling rsync after a fresh reboot of both source and target servers, and a reboot of the Mikrotik switch, and ER-x router: And 5 min after that screenshot: Those two files being synced are the two diag files attached to this post. Subsequent attempts without rebooting anything are even worse: Target network: Source network: Any thoughts as to how to resolve this? Thanks everyone, Cal.

Thank again @bmartino1. I'll give this a try and let you know how it goes.

Thanks @bmartino1. I'm currently running Unraid 6.12.14. I'm in the middle of migrating to a new server, so I have a chance to create a brand new zpool, so based on what you're saying I should be able to, from the terminal, create the zpool with 2 vdevs, one with raidz1, and one with raidz2. I have other backups in place, and am comfortable with the risk associated with having one of the vdevs being raidz1. Would I do this simply by running the following? zpool create -O casesensitivity=insensitive -m /mnt/ur2pool ur2pool raidz1 {4 drives here} zpool create -O casesensitivity=insensitive -m /mnt/ur2pool ur2pool raidz2 {4 drives here} Then creating the dataset within the zpool as normal? Thanks again @bmartino1, Cal.

Thanks @JorgeB. I don't use the traditional unraid shares. I have them turned off as per the screenshot below. If I'm understanding what you're saying, I could do it, (use a raidz1 vdev and a raidz2 vdev in the same zpool), via terminal and it would work? Could I automate the import in the go file so the mixed vdev zpool is online after a reboot? Do you happen to know if this would work in the upcoming Unraid 7? Thanks again @JorgeB, Cal.

Follow up question to my post above that is slightly different in nature, hence a separate post: Is it possible to add a 2nd vdev to an existing zpool with a different raidz level? e.g. adding a 4 disk wide raidz1 vdev to an existing pool made up of a 4 disk wide raidz2 vdev? I'm currently running 6.12.14. If this is not possible in 6.12.14, would it be possible in the upcoming version 7? Thanks again everyone, Cal.

Is this still the case? I'd like to add a raidz1 vdev to my existing pool containing a raidz2 vdev...both vdevs would contain 4 physical disks. If I did it all via terminal would it work?

Turns out this value "5" should not be set, as it is the quota for the dataset:

Looks like I overlooked this part of your post. I checked the zpool.cache file and it does exist, and looks like it has a timestamp that means it's been updated this eve. When I Nano it, it looks like a binary file that I cannot read. Is there something I should be doing to ensure it's not corrupted, and that is being updated correctly as you note? On the second part, I ran "zfs get all | grep compression" and I see: zfs get all | grep compression ur2pool compression off local ur2pool/hotBackup compression zstd-5 local When I run "modinfo zfs" I get a few pages of text. Changing the command to be "modinfo zfs | grep comp" gets me: modinfo zfs | grep comp parm: zio_slow_io_ms:Max I/O completion time (milliseconds) before marking it as slow (int) parm: zfs_sync_pass_dont_compress:Don't compress starting in this pass (int) parm: zfs_rebuild_scrub_enabled:Automatically scrub after sequential resilver completes (int) parm: zfs_compressed_arc_enabled:Disable compressed arc buffers (int) If I try grep zstd, or grep lz4 I get no results. Is there something specific I should be verifying or looking for here? EDIT: Can you explain what you mean by "otherwise you may need a go file / first array start script to set the compression."? EDIT2: I placed "zfs set compression=zstd-5 ur2pool" into my go file, but it had no impact: zfs get compression NAME PROPERTY VALUE SOURCE ur2pool compression on local ur2pool/hotBackup compression zstd-5 local Thanks again @bmartino1, Cal.

Hi everyone, I'm looking for steps to set zstd-5 compression at both the zpool level, and at the dataset level. Thanks to help from @bmartino1 I managed to get the dataset compression set, but the zpool level still shows as "on". I checked and Unraid 6.12.14 is running > version 2 of zfs, so zstd-5 should be supported. zfs -V zfs-2.1.15-1 zfs-kmod-2.1.15-1 With help from @bmartino1 I did the following: Created the zpool via the command line: zpool create -O casesensitivity=insensitive -m /mnt/ur2pool ur2pool raidz1 ata-HUH721212ALE601_8DJ2S4ZH ata-HUH721212ALE601_8HKS73MY ata-HUH721212ALE601_AAGAX74H ata-HUH721212ALE601_AAGHK6WH zfs set compression=zstd-5 ur2pool zfs set atime=off ur2pool I then added a pool via the "Add Pool" button at the top left: I populated the newly created pool with the 4 drives that makeup my raidz1 zpool. I then clicked on the pool and edited the values with red arrows: I then created a dataset via the create dataset button in the ZFS Master plugin as per the screenshot above, and set compression to zstd, and "Quota" to 5 MiB: Do not set quota to 5, as this limits the data allowed in the dataset. However upon reboot, I see the following instead of "zstd-5" for both: zfs get compression NAME PROPERTY VALUE SOURCE ur2pool compression on local ur2pool/hotBackup compression zstd local I then set compression via the command line for both the zpool, and the dataset: zfs set compression=zstd-5 ur2pool zfs set compression=zstd-5 ur2pool/hotBackup Upon reboot, I see the following: zfs get compression NAME PROPERTY VALUE SOURCE ur2pool compression on local ur2pool/hotBackup compression zstd-5 local My understanding is that "on" means lz4 compression. Ideally I'd like this to be zstd-5. This is pretty close. Is there a way to set zstd-5 at the pool level? Is there anything I've done in the steps below that should be adjusted before I populate the pool/dataset with data? Thanks everyone, Cal.

Thanks again @bmartino1. Setting the compression again after reboot, then rebooting again has me at the closest point yet, that is probably good enough. Ideally I'd have the same zstd-5 at the pool level, but I guess this will work. zfs get compression NAME PROPERTY VALUE SOURCE ur2pool compression on local ur2pool/hotBackup compression zstd-5 local

Getting closer. I followed these steps and now I'm getting: zfs get compression NAME PROPERTY VALUE SOURCE ur2pool compression on local ur2pool/hotBackup compression zstd local Any other thought as to how can I get zstd-5 or another zstd-# compression? Thanks again @bmartino1, Cal.

Ahh thanks again @bmartino1. I'll delete the dataset I created via terminal, then recreate it via the GUI. I only see the basic zstd, but no option to set zstd-5. Is there a way to set zstd-5?

Where do you see this page? I'm not seeing it. I have ZFS master installed, and I see the following: I got here by hitting "Add Pool" (below). I didn't see any options under ZFS Master to create a new pool, so I did it via command line. zpool create -O casesensitivity=insensitive -m /mnt/ur2pool ur2pool raidz1 ata-HUH721212ALE601_8DJ2S4ZH ata-HUH721212ALE601_8HKS73MY ata-HUH721212ALE601_AAGAX74H ata-HUH721212ALE601_AAGHK6WH zfs set compression=zstd-5 ur2pool zfs set atime=off ur2pool zfs create ur2pool/hotBackup The zpool then showed up under ZFS Master: What am I missing? Thanks again @bmartino1, Cal.

Hi everyone, Maybe I'm daft, but I'm not able to set zstd-5 compression on my zpool, or the datasets contained within the zpool. I run "zfs set compression=zstd-5 mypool" and it runs without error. I then run "zfs get compression" and it shows "zstd-5" After reboot, I get a mixed bag. I've seen "zfs get compression" return "on", and "off" seemingly without a repeatable pattern. I checked and Unraid 6.12.14 is running > version 2 of zfs, so zstd-5 should be supported. zfs -V zfs-2.1.15-1 zfs-kmod-2.1.15-1 What am I missing here? Thanks everyone, Cal.

Thank you for pointing that out. I was not aware that was the case. I'm not sure if on my latest reboot I exported it first, but now it seems that the zpool is mounting correctly upon reboot. In fact, I'm pretty sure, but not certain, that I did not export it before my last reboot, but either way, I've done two reboots, and it is now working. Thank you @JorgeB for your help as always. You've saved my bacon more than a few times, and I am very grateful for your help and insight. I hope you have a great holiday season @JorgeB, Cal.