Unsatisfactory transfer speeds on NVMe ZFS pool

[leutnantlurch]

Hello everyone

 

I have an interesting problem with Unraid and ZFS. Briefly about my setup, I have a Unifi network at home. My PC is connected with a 2.5 GbE NIC to a 2.5 GbE switch. My Unraid server is connected to the same switch with an Intel X520-DA2 via 10 GbE. I have a normal HDD array with single parity, a ZFS cache, a ZFS HDD pool and a brand new NVMe ZFS pool.

 

The platform is an X99 mainboard with 32 GB DDR4 Ram and an Intel Xeon E5 2699 v3 CPU.

 

Now to my problem, I have a rather modest transfer speed for all ZFS pools. My tests have shown that the transfer speed is dependent on my CPU generator. On "power saving" I get transfer speeds of about 145 Mbyte/s, on "on demand" I get speeds of about 180 Mbyte/s and on "performance" the speed increases to about 250-280 Mbyte/s from transfer start to transfer end. I find this particularly irritating with NVMe pools, as these SSDs are absolutely capable of a speed of around 280 Mbyte/s from the start.

Now I wanted to ask whether the CPU is perhaps no longer capable of delivering such speeds?

 

Thank you for the answers.

 

Greetings from Switzerland!

[itimpi]

You do not mention how you are measuring this and what the transfer is to/from.   If it is the PC then that has a theoretical maximum of around 250MB/sec if your LAN is 2.5 Gb/sec.

[leutnantlurch]

Yes, you're right, I need to provide more data here I have now tested with iperf3.

 

CPU Governor "power safe"

 

"on demand"

 

"conservative"

 

"performance"

 

WIndows filetransfer on ZFS NVMe Pool (testfile generatet with fsutil)

 

"power save"

 

"on demand"

 

"conservative"

 

"performance"

 

And the tests with the HDD ZFS Pool (only on demand and performance)

 

"on demand"

 

"performance"

 

After the tests, I'm no longer sure whether the transfer speeds are really not satisfactory. Can you take a look at it and tell me whether this is complaining on a high level?

[Vr2Io]

Xeon E5 2699 is core / thread monster, but clock frequency turbo only 3.6Ghz, in my experience if clock frequency running lower then 3Ghz then large storage performance degrade would be expect.

 

For consumer CPU due to less core count, in lower power governor would still keep at rather high clock frequency. So change to some less core count CPU should solve the problem.

 

Or you could try disable half core in BIOS to check any different.

[leutnantlurch]

Hey!

Thanks a lot for your answer. More and more I have the feeling that my CPU wants to retire...

 

I have currently observed that the urBackup backup 1.) takes a long time and 2.) drives the CPU to the limit in places, the Unraid WebUI does not respond and the other applications that I run on the server become very sluggish or crash (in the case of Authentik).

 

I haven't had any major problems so far, but until recently I didn't have a 2.5 GbE connection either.

 

However, since a stronger CPU would automatically mean a platform change, I would like to be sure that it is a "performance problem".

[leutnantlurch]

What do you guys think about the i9-10900X? I would have the opportunity to buy a X299 Motherboard and this CPU + 64GB of Ram for a reasonable price.

[Vr2Io]

My main storage build are x299 9800x, main reason was it support 8 memory slot ( 32GB x8 ), I setting its power limit PL2 at 66W, so clock frequency maintain to ~3.3Ghz, so far so good.

It have two ZFS pool, 8 disk raidz1 and 12 disk raidz1x2.

 

If you not need much PCIe lane and memory, then no reason to go in those platform. My 2nd and 3rd build ( not storage purpose ) are 9700k and 5600h.

Edited November 22, 2023 by Vr2Io

[leutnantlurch]

After upgrading to x299 and a i9 10900X with 64GB of RAM (yes i need the pcie lanes ) these are the results with iperf:

 

 

I would say, this is perfect! Thanks to @Vr2Io for the tipp, I will tweak my cpu.

 

Case closed! 😃