testing Unraid on a ZimaCube Pro (1235U w 64Gb RAM). V7.0.0 beta 4.

 

All drives are new.

I have 4 WD Red Plus 8-TB (1 is parity) - formatted ZFS

2 WD_Black SN850x NVMe- 2-TB as a pool, btrfs

2 Samsung 970 Evo Plus NVMe 1-TB as another pool - zfs 

Lan is single 2.5GbE

Client - Windows 11 workstation on 10GbE

 

I've got a few large files I was using as testing. As I send them to either (windows copy, TO Unraid host) of the ZFS destinations (array or pool) - it screams along at 280 MB/s, until I get 4-5GB complete (of 8.0 GB file), then it drops off to 0. It will usually eventually ramp up enough to finish, but  I do get timeout with 'unexpected network error' about a third of the time.  One of my CPU threads is 100% during the copy .  When the ethernet throughput hits the wall, I notice a second thread on the CPU (out of 12) goes to 100%. RAM never gets about 20% utilized. When that thread eases back down , the transfer resumes. I was reading some similar issues with ZFS, so I redid one of the pools as btrfs - this one I get about 2GB into transfer, and then it drops from 280 MB/s to about 60 MB/s. CPU never hits 100%, and it completes - just slowly . I tried locking my workstation ethernet down to from 10GbE to 2.5GbE. Turning on and off NIC offload and flow-control, no change. I see no ethernet errors on the switch at either port. I test this same network path and test files to my production NAS, a little Asustor AS5304T w single RAID-5 array, and no such issues. Solid transfer rate the entire time. While not scientific, I see the NVMe drives in the pool still showing a steady write-speed even when my copy has stalled to 0.

 

Testing same file payload FROM Unraid to Windows host, I get 270-280 MB/s the entire time, no problem. If I use a smaller file, say 2.8 GB - from windows TO unraid ZFS Pool, it is 270-280 MB/s the entire time. Only when I get to about that 4-5 GB mark does it tank. Sooner w/ the btrfs pool (2.5 - 3.5 GB). I've attached diags that are from when the copy speed falls off (one for ZFS, one for btrfs). Am I hitting some cache limit or RAM config?   

 

These are user shares, but I've tried it with disk shares - same effect. I also notice its about the same when I test to the ZFS array disks, but  they show an active write speed to the disk and parity disk long after I've either completed or gotten unexpected timeout. 

 

No docker or VM engines started.               

zcube1-diagnostics-zfs-20241122-1906.zip zcube1-diagnostics-btrfs-20241122-1903.zip

 

 

https://nascompares.com/2024/08/19/zimacube-pro-nas-review/

 

Looks like it may be low on processor compute. Its not bad hardware for casaos / base nas duties, might be a bit underpowered for all of unraids features....

 

From the extracted diagnostics, key files for investigating performance issues include:

System Folder:

top.txt: CPU load and running processes.

memory.txt and meminfo.txt: Memory usage details.

zfs-info.txt and btrfs-usage.txt: Filesystem-specific statistics.

ethtool.txt and ifconfig.txt: Network configuration and status.

Logs Folder:

syslog.txt: General system activity logs for error messages or unusual activity.

 

Analysis of Logs:

top.txt (CPU and Memory Usage):

Load Average: Moderate, not overly taxed (1.72, 0.99, 0.58).

CPU Usage:

~80% idle, with ~18.5% waiting for I/O (wa), which could indicate a bottleneck related to disk I/O.

Memory Usage: Ample free memory (~37 GB free, ~17 GB cached), with no swap in use.

zfs-info.txt (ZFS Status):

All pools (cache, disk1, disk2, disk3) are in an ONLINE state with no read, write, or checksum errors.

ZFS metadata indicates the pools are healthy, and there's no immediate corruption or degradation.

syslog.txt (System Logs):

No specific warnings or errors related to file transfers in the preview.

Kernel logs indicate a clean boot, and no anomalies were captured in this snippet.

Potential Causes for Performance Drop-Off:

Cache Limits:

The high performance at the beginning of the transfer (~280 MB/s) could be due to the ZFS ARC (Adaptive Replacement Cache).

Once the ARC is filled, ZFS resorts to slower disk writes, leading to the observed drop.

Disk I/O Bottlenecks:

The ~18.5% I/O wait (wa) from top.txt suggests possible contention with disk operations during transfers.

Single-Thread Limitation:

ZFS can sometimes rely heavily on a single thread for certain tasks (e.g., compression, check summing), leading to CPU bottlenecks even with low overall usage.

Network Interface Constraints:

The network logs (ethtool.txt) and interface configuration should be checked for any MTU mismatches or offloading settings impacting sustained transfer rates.

Next Steps:

Increase ZFS ARC Size:

Use zfs set primary cache=all on your pool to prioritize caching for metadata and data.

Allocate more memory to ARC if it's not already using the available free memory.

Check Disk I/O and Pool Configurations:

Verify if your parity disk is slowing down writes, especially for small files or unaligned writes.

Network Adjustments:

Inspect ethtool.txt for network interface settings, such as offloading and MTU size.

Test transfers with direct connections to eliminate potential switch-related issues.

Advanced Logging:

Enable verbose logging for ZFS to monitor cache behavior during transfers.

 

Edited November 23, 20241 yr by bmartino1
typo

review:
 

comes down to what you want it to do and how you want to interact with it. What kind of tuning are you looking for?

If the transfers start fast and then slow down, it suggests the network is fine, just that the devices are not able to keep up, except for the array with zfs, that suffers from a known issue.

 

Array disks appear to be empty, if they are, reformat disk1 to xfs, enable turbo write, start a large file transfer directly to the array, post the Windows explorer graph and the diags during the transfer.

11 hours ago, bmartino1 said:

review:

comes down to what you want it to do and how you want to interact with it. What kind of tuning are you looking for?

To be able to transfer more than 3-4 Gb without it timing out.  It's going to do some basic NAS duties, but primarily *arr suite and Plex. Transcoding via A2000 Ada GPU. So I need to be able to get large media files to it reliably. 

8 hours ago, JorgeB said:

If the transfers start fast and then slow down, it suggests the network is fine, just that the devices are not able to keep up, except for the array with zfs, that suffers from a known issue.

 

Array disks appear to be empty, if they are, reformat disk1 to xfs, enable turbo write, start a large file transfer directly to the array, post the Windows explorer graph and the diags during the transfer.

Yeah, network is fine. I've tested the same path to with another NAS device with no issue.  All disks/pools are empty except for the files I bounced around for this testing. I'll try one as XFS and post results. 

8 hours ago, JorgeB said:

 

Okay, redoing disk1 as XFS, it behaves very much like the btrfs nvme pool. Starts out at wire speed ( ~280 MB/s) then about 2.5 GB into the 8GB test file, it drops down to 70-80 MB/s for the rest of the xfer.   Then enabling reconstructive write - it maintains 280 MB/s longer, about 3.8GB into the transfer, then dips to about 180-190 MB/s for the remainder. That's the fastest yet. I wasn't expecting a single spinning disk  - with parity, to outperform the btrfs NVMe 2 slot pool. Is there a big overhead difference between btfrs and xfs? Or COW duties vs parity? My goal is to get the NVMe cache pools to sustain 280 MB/s. The 4 NVMe drives share a PCIe 3.0 x 4 connection, but only hitting one pool at a time I'd think it could hold 280 MB/s.  Doing reads (copy from) Unraid to the client sustain 200-230 MB/s indefinitely.  I'll go play with ZFS cache and see if I can move the needle any. Is there any tuning that can be done w/ btrfs? . With 64Gb ram and just a few containers to run, I've got some ram to play with I think. Also, here's current disk cache settings:

 

Current Disk Cache Settings

vm.dirty_background_ratio = 3%

vm.dirty_ratio = 5%

vm.dirty_writeback_centisecs = 200

vm.dirty_expire_centisecs = 1000

 

48 minutes ago, RJT_2k said:

with parity, to outperform the btrfs NVMe 2 slot pool. Is there a big overhead difference between btfrs and xfs?

There's shouldn't be, not like that, try creating two separate pools with both NVMe devices and retest, in case there's an issue with one of them, you can also test with different filesystem, but they show all perform very similar for a single device pool.

try a diffeent copy portcal like ftp... windows is nortious for louse results as windows decides and makes a trap to not copy higher or lower. This may require smb extra options for perforamce tuning to unraids samba....

To diagnose and address these performance and stability issues, let’s break the situation down and suggest a few approaches tailored to the described challenges:

 

Analyzing Bottlenecks

From your description, here are potential causes for the observed performance degradation:

Write Cache Saturation: A pattern emerges where transfers degrade after a certain point (~4-5GB for ZFS, ~2-3GB for BTRFS). This indicates that the write cache or disk I/O buffer is being saturated.

CPU Utilization: A single thread hits 100%, which could indicate that the system is CPU-bound for certain operations, such as parity calculations, file system metadata handling, or protocol overhead (e.g., SMB).

PCIe Bandwidth Contention: The NVMe drives sharing a single PCIe 3.0 x4 connection could face contention, especially with simultaneous reads/writes.

File System Overheads: Differences between BTRFS, XFS, and ZFS (e.g., Copy-on-Write mechanisms, metadata management) may also contribute to performance disparities.

Network Layer: Intermittent stalls with “unexpected network error” might involve network interface card (NIC) offloading or driver issues.

 

Configuration Suggestions

Disk Cache Tuning

Current settings for vm.dirty_* parameters seem conservative for a system with 64GB of RAM. Adjusting these values may improve sustained write performance:

sysctl -w vm.dirty_background_ratio=10
sysctl -w vm.dirty_ratio=20
sysctl -w vm.dirty_writeback_centisecs=100
sysctl -w vm.dirty_expire_centisecs=500

Explanation:

dirty_background_ratio and dirty_ratio control how much dirty data can exist in RAM before being flushed to disk.

Higher values delay the flushing process, allowing larger bursts of writes to sustain performance.

 

Testing Disk Configurations

XFS vs BTRFS:

XFS is generally more predictable for high-throughput workloads. Stick with XFS for spinning disks unless snapshots or specific BTRFS features are required.

ZFS Pools:

Verify ZFS ARC (Adaptive Replacement Cache) size with

cat /proc/spl/kstat/zfs/arcstats

Reduce ARC size to prevent ZFS from consuming too much memory, especially if Docker or other services are running:

echo "4G" > /sys/module/zfs/parameters/zfs_arc_max

Cache Pools:

Ensure NVMe cache pools use the latest firmware and drivers.

Optimize ZFS recordsize (128K is a common optimal size for media files).

 

Networking Adjustments

NIC Offloading and Flow Control

Disable problematic offloading features and flow control settings on the Unraid server:

ethtool -K eth0 tso off gso off gro off
ethtool -A eth0 rx off tx off

MTU Settings

Ensure consistent MTU (maximum transmission unit) across all devices. If using jumbo frames (e.g., MTU 9000), test with standard MTU (1500) to rule out packet fragmentation.

SMB Tuning

For SMB transfers, modify the SMB configuration (/etc/samba/smb.conf) to optimize large file handling:

[global]
   aio read size = 1
   aio write size = 1
   socket options = TCP_NODELAY IPTOS_LOWDELAY
   write cache size = 1048576
   min receivefile size = 16384
   use sendfile = yes

 

Monitoring and Testing

Identify Bottlenecks

Disk I/O: Use iotop to monitor disk activity and identify if specific drives are slowing down.

CPU Utilization: Use htop or top to confirm which process is spiking CPU usage during transfers.

Network: Run iftop or nload to monitor real-time network usage and identify stalls.

Synthetic Benchmarks

Use tools like iperf (for network) and dd (for disk) to isolate performance issues:

 

#network test:
iperf3 -s # on Unraid
iperf3 -c <unraid-ip> # on client

#Disk test
dd if=/dev/zero of=/mnt/cache/testfile bs=1G count=10 oflag=direct

 

Advanced Troubleshooting

Check PCIe Bus Saturation

Tools like nvme-cli can be used to monitor real-time NVMe performance. Ensure no single device saturates the PCIe bus:

nvme top

Driver Updates

Ensure the NIC, storage, and GPU drivers are up-to-date. Unraid often ships with updated kernels, but verifying driver versions can help.

Parity Disk Optimization

If using reconstructive write, ensure the parity disk isn't a bottleneck (e.g., slower HDD compared to other array drives).

 

Summary

Key recommendations:

Adjust disk cache settings (vm.dirty_*).

Use XFS for array disks and test ZFS tuning for cache pools.

Optimize network settings (disable offloading, adjust SMB).

Monitor I/O and CPU to pinpoint bottlenecks.

Run synthetic benchmarks to isolate performance issues.

Implement these changes step-by-step and test each to observe improvements. Let me know how it goes or if you need clarification!

 

So, I went and enabled exclusive shares, and rebooted. Testing to cache pools (btrfs and zfs) - 283 MB/s the whole way through on 8GB file. Even two of them back to back (16GB total). Same - 283 MB/s entire copy, all the way on disk2 (zfs array disk). I disabled exclusive shares, rebooted, and still getting 283 MB/s all the way through.

 

I did have my pools shares set to the array as secondary storage early in. Maybe it left a config with fuse involved until the reboot? After disabling exclusive shares, I still had the pool shares set to no secondary storage. No other changes were made other than exclusive shares on then off, and 2 reboots. 

 

Odd, but it's cruising along now.  I'll do some more testing later this weekend, maybe switch the unraid host over to 10GbE to see where where the different drives/pools choke out. 2.5GbE is my bottleneck right now, which is fine for my needs.

 

if you set secondary storage yes you will see a penalty hit as it has to write and read in 2 locations.

the first location and then how mover decides adn when it moves the files from the 1st location to the second.

I don't recommend using second storage unless its array spinning disk to nvme cache disk for the performance benefit. (more on read then write....)

You are most lilkley hitting nvme pcie bottlenecks by trying to copy to multiple nvme disk on the pcie bus and the network copy at the same time.

Windows copy is detecting a similar to waht a HDD smr cahce read write that drops the performance to gurantee a file write...

Other disk settings may need to be adjusted like md write method:


review:

 

1 hour ago, RJT_2k said:

Maybe it left a config with fuse involved until the reboot?

Reboot is not needed, and exclusive shares don't work with secondary storage set, but if I understood correctly it's still fast with exclusive shares disabled, if true, something else was causing the issue.

19 hours ago, JorgeB said:

Reboot is not needed, and exclusive shares don't work with secondary storage set, but if I understood correctly it's still fast with exclusive shares disabled, if true, something else was causing the issue.

correct. I had secondary locations set up originally. Encountered issue. I removed secondary locations, had issue. Set exclusive shares to yes, and rebooted, issue gone. Set exclusive shares to no, but still have no secondary location - still no issue.  I'll move it over to 10GbE today and test some larger files to baseline and see where my breakpoints are, then maybe tune a little more zfs / performance options. But as long as nothings regressed, I'm already above my goal on performance. Not sure what the issue was, but thanks for everyone's help in tuning recommendations and general 'how does it work' pointers. It's much appreciated.

I made some small tweaks , but really, whatever was going on was solved somewhere in that reboot. Testing on 10GbE , I was able to push 1.09 GB/s for an entire 35.1 GB write operation to either of my two cache pools (disk share). 

 

Putting it back to the normal 2.5 GbE link, it can push 283 MB/s solid to either cache pool (user share). So all good. It meets or exceeds my current production NAS, and that was my goal for throughput. I moved everything back over to ZFS (both cache pools, and all of the array HDDs) and will keep testing.

 

Thanks again for everyone's help.