I'm seeing a write speed issue on a ZFS array disk (no parity) when the source disk is able to provide data faster than the destination disk can write it. In a controlled test, write speed was about 80 MB/s, whereas by limiting the source speed, write speed was over 170 MB/s.
I would expect this to be reproducible on at least some systems. My system has dual Xeon E5-2450s (not V2). Their age and fairly weak single-core performance may have something to do with the issue, but given how drastic the difference is, I don't think that CPU limitations fully explains it.
--
Here are the details:
I added a new drive (12TB Red Plus) to my unprotected array. It had no previous partition, and I let unRAID format it as ZFS (no compression). I created a share called "Test" on only that disk with no secondary storage.
On my existing SSD cache pool, I have a "Download" share (shows as exclusive access). In that share, I copied several very large media files into a "_test" folder (83 GB total).
At the command line, I navigated to the Download test folder and ran the following:
rsync -h --progress --stats -r -tgo -p -l * /mnt/user/Test/
This took 17m57s, for speed of 79 MB/s (calculated, not just taking rsync's word for it). As you can see from the first image, write activity was very sporadic. During the dips, one or two CPU threads were pegged at 100% (usually two).
Next, I removed the files from the Test share and reran the command, this time adding the --bwlimit option:
rsync -h --progress --stats -r -tgo -p -l --bwlimit=180000 * /mnt/user/Test/
I may have been able to go higher, but I wanted to make sure to use a speed under the max capabilities of the destination disk. It completed in just 8m12s, for a speed of 173 MB/s. The second image shows a drastically different profile for the write activity.
Finally, just in case the order of operations had an impact, I removed the files again and reran the original command without the limit. The speed & activity profile was the same as before.
--
I'm thinking of reformatting the test disk to BTRFS to see how it behaves (I still want to use ZFS long-term). I also don't know what will happen once I have dual-parity. Although using reconstruct write, I would expect to still be limited by this issue.
I'm actually not that concerned about array write speed once I have the data populated, but the performance loss is so substantial that I thought I should report it in case anything can be done. Let me know if there's any additional testing I could do that would help.
Recommended Comments
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.