Hello Unraid-Community!

 

I have recently upgraded my motherboard, CPU and RAM and I have noticed that the cache write speed over 10 GbE is slower than it used to be.

Old hardware: write 500-550mbs / read 600-650mbs

New hardware: write 300mbs / read 600mbs

 

I have tried to find a solution, but no luck so far.

 

Really appreciate if anyone could look into my diagnostics, to help me find a solution.

unraid-diagnostics-20241102-0108.zip

Edited November 2, 20241 yr by parazit

Start by running a single stream iperf test in both directions.

Hi,

I did the iperf test and these are the results.

 

Also what I see is that are writes and reads every 5-10 seconds to the cache disk. Is that a normal behaviour?

Edited November 3, 20241 yr by parazit

The iperf speeds look right.

Cache drive activity is normal.

thank you for your answer.

 

Yes, the speeds look good but I still think there is something wrong. Before I changed hardware the speeds were better. When I was testing the speed using blackmagic disk speed test, I go following results.

Old hardware: write 500-550mbs / read 600-650mbs

New hardware: write 300mbs / read 600mbs

Something like that is tough to nail down.  My guess is something about your new HW.  Maybe the memory or could be individual core speed of the CPU.  Was your nvme connected directly to CPU on previous config and now shared thru a MB chipset?  Probably more possible explanations along that line.

I do not think unraid would harbor old drivers causing an issue within itself.  Could fire up a trial USB and test.  If that's faster, you could have your answer.

On 11/2/2024 at 12:09 AM, parazit said:

500-550mbs / read 600-650mbs

This seems too good for a SATA pool, the current results seem more normal, post a screenshot from the Windows explorer transfer graph from a large file transfer.

Well that's weird, but if that's normal, I won't do any more tests, because I want to upgrade the SATA SSDs anyway. I will get 2x SN770 tomorrow and post the results.

Would you recommend btrfs or zfs for a cache pool?

 

Here is the screenshot of a 1gb file.

For multidevice pools recommend zfs over btrfs, it's better a recovering from a dropped device for example.

 

Regarding the speed, looks OK to me for the existing pool, but it may be a little faster with disk hares (or exclusive shares)

I have swapped the SATA SSDs with 2x M.2 SN770 1TB but the result is still the same, which is very weird.

It actually doesn't matter if primary storage of a share is set to cache or not - the speeds are almost equal.

Any ideas, what the bottleneck could be?

Did you test with an exclusive or disk share?

Yes, all test that I have done so far were with a disk share. I am using blackmagic disk speed test for the tests.

 

Edit: Sorry, I thought I was using a disk share but it was a user share. So I have did a test by writing and reading directly into the cache drive and the result is way better

write: 500mbs

read: 800-950mbs

Edited November 4, 20241 yr by parazit

That's better, if your use case allows it, try using exclusive/disk shares whenever possible, in a future kernel with FUSE passthrough support, the performance penalty with user shares will hopefully be much smaller.