Hi.

I have a rig with multiple M.2's and I must admit I was expecting a better throughput than what I am seeing now. I have the following rig:

• ASRock X570 Pro4
• Ryzen 5800X
• 64 GB ram (out of which 50% is allocated to zfs via zfs.conf override)
• Graphics in Slot 3
• Asus Hyper M.2 x16 Gen 4 in slot 1 (bifurcated 4x4)
• 2 x Samsung 990 Pro (in slot 1 and 2 on Asus)
• Samsung 980 Pro (in slot 1 on mb)
• Samsung 980 Pro (in slot 2 on mb)

 

Now the response I'm getting from the system is very good, but when I tested the rig just now I belive I should be seeing better speeds. The following screenshot shows the following rsync jobs:

• 980-Slot-1 to 990-Slot-1
• 980-Slot-1 to 990-Slot-2
• 980-Slot-1 to 980-Slot-2

 

When testing all combinations of copying between the M.2 the numbers are almost identical (to within 1%). The screenshot shows a test with disks formated with zfs-encrypted (with compression), but I have mirrored the tests using xfs with the same results.

Are there any hardware limitations I'm hitting or is this the expected level?

 

//Thomas

Edited February 29, 20242 yr by Ulf Thomas Johansen
typo

Rsync is not really built for speed, having said that, in my experience don't expect much more the between 1 to 2GB/s, mostly depending on the CPUs single thread performance, try using PV with a large file to see if it's the same:

 

pv /path/to/source > /path/to/dest

 

Good point on rsync, and I've redone with pv as suggested. Giving the following observations:

• 1.79 GiB/s  between the 990's (on the asus card)
• 1.19 GiB/s between the 980's (on the mb, one of which will be capped at gen3)
• 1.86 GiB/s from one 980 to one 990

 

Better then rsync, but I am not getting much more effect from the Asus and the 990's.

Edited February 29, 20242 yr by Ulf Thomas Johansen

I'm afraid that's around what I would expect to see with that CPU, you might be able to get around 2GB/s+ with a faster single threaded CPU, like an i5 14600K.

A more structured test using a 128 GB file:

 

I am not sure I follow your argument that the cpu is the bottle neck because when I run two pv's in parallell I am getting an average of 3 GiB/s throughput? Watching htop I can see that the system is hitting 4 GB/s in disk R/W pr pv job. Wouldn't this suggest something else is capping?

 

Edited February 29, 20242 yr by Ulf Thomas Johansen

22 minutes ago, Ulf Thomas Johansen said:

hen I run two pv's in parallell I am getting an average of 3 GiB/s throughput?

That's normal, that's why I mentioned 

 

29 minutes ago, JorgeB said:

faster single threaded CPU

 

Each transfer will only use one CPU thread.

Aha ... Makes sense.