Ok I have tried everything I can think of and still this pool crawls performance wise and I cannot for the file of me figure out why.

 

I have 4x 1TB Samsung 970 Pro NVME drives in a ZFS Z1 Pool. but I can only get about 40-50 MB/sec write and about 400-500 MB/sec read speed from it. This set up should produce well over 2GB a second in write speed and prolly couple that for read. I have 40GB fiber between the machines locally. When I made the share I set it on the nvme-pool with no secondary storage. There has to be something I'm missing here.

 

One user suggested enabling disk shares and I did the test using that. The performance about doubled but is still very low for the hardware.

Also...

 

iperf3 to the server from the windows box and it shows:
1.56 Gbits/sec

iperf3 from server to windows box:
12Gbits/sec

 

I have jumbo frames set to 9000 on both ends and on the Mellenox SX6036 switch.

I have seen others with almost the same setup but with slower drives in the system get about 8GB/sec of transfer over SMB so does anyone know what I'm doing wrong?
"though I don't think that drives come into play in the iperf test, I may be wrong."

Also, the Windows box has the drivers from the Nvidia website installed. latest version.

in unraid it says that the info for the card is :

FW Version: 2.43.7010
FW Release Date: 6.5.2019
Product Version: 02.43.70.10
Rom Info: type=PXE version=3.4.662
type=UEFI version=14.9.90 cpu=AMD64
Device ID: 4103
Description: Node Port1 Port2 Sys image
GUIDs: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs: e41****62720 e41d2d762721
VSD:
PSID: MT_1090111023

Which is a little weird. since according to the nvidia website the latest firmware is:
fw-ConnectX3Pro-rel-2_42_5000-MCX314A-BCC_Ax-FlexBoot-3.4.752.bin

Am I wrong or is that not older firmware? Am I reading the version wrong?

Systems are: 

 

Windows 11 Pro box is as follows:

i9-13900K
64GB DDR5

Z790 chipset

Samsung 990 SSD 2TB

Unraid is on:

HP ProLiant ML350 Gen9

2X Intel® Xeon® CPU E5-2640 v4 @ 2.40GHz 20 cores total

192GB DDR4 ECC Memory

4x Samsung 980 PRO 1T nvme drives in a Z1 pool 4TB total

 

 

 

 

Any insight would be very much appreciated. I have been at this for like 3 days and cannot figure it out.

 

Thank you in advance

 

Edited January 2, 20242 yr by calmasacow
Additional info

Suspect CPU clock rate too low / too weak CPU

 

CPU is Intel Xeon CPU D-1541 @ 2.10GHz 8 core 16 threads

 

 

Edited January 1, 20242 yr by Vr2Io

22 hours ago, Vr2Io said:

Suspect CPU clock rate too low / too weak CPU

 

CPU is Intel Xeon CPU D-1541 @ 2.10GHz 8 core 16 threads

 

 

CPU is dual Intel® Xeon® CPU E5-2640 v4 @ 2.40GHz 20 cores total. and 192GB of ram. it shouldn't be an issue and shouldn't be this slow. it is a little faster when I use NFS instead of SMB but still seems very low. Seems should be able to get at least the single drive performance.

23 hours ago, calmasacow said:

40-50 MB/sec write

especially seems too slow, how are use testing this? Also, did you ran an iperf test to confirm the LAN is performing normally?

3 hours ago, JorgeB said:

especially seems too slow, how are use testing this? Also, did you ran an iperf test to confirm the LAN is performing normally?

I'm using Blackmagic disk speed tester. My only concern is actually read and write performance via via network nfs/smb. I don't know how to perform iperf test between unraid and windows. I switched to using NFS and now I get about 130MB write and about 550MB read. But still it is well below the performance that I should be getting of these drives. individually these drives get about 2GB/ in both write and read. I have 4 of them in a Z1 pool. I would think I would get at least single drive performance.


Ok I found out how to run iperf3 to the server from the windows box and it shows:
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.82 GBytes  1.56 Gbits/sec                  sender
[  4]   0.00-10.00  sec  1.82 GBytes  1.56 Gbits/sec                  receiver

 

When I run it back the other way from unraid server to the windows box I get about 11-12 Gbit. which seems odd.

This also seems low both have 40GB fiber cards in them. is there something I can alter that may be contributing to this low throughput. I have jumbo frames set to 9000 on both.

Edited January 2, 20242 yr by calmasacow

1.56Gbit is pretty low, and suggests something is not working as it should with the LAN.

On 1/3/2024 at 2:19 AM, JorgeB said:

1.56Gbit is pretty low, and suggests something is not working as it should with the LAN.

Agreed and also why is it 8x fast the other direction. and even 8x is slow I have seem almost same hardware typically is in the neighborhood of like 20-25Gbits. gonna try to boot to Linux on the windows box and see if that makes any difference. some have suggested that it is a windows issue.

On 1/2/2024 at 6:57 PM, calmasacow said:

CPU is dual Intel® Xeon® CPU E5-2640 v4 @ 2.40GHz 20 cores total. and 192GB of ram. it shouldn't be an issue and shouldn't be this slow. it is a little faster when I use NFS instead of SMB but still seems very low. Seems should be able to get at least the single drive performance.

Well i learned my self that many cores doesn't mean that the machine is capable of handling the tasks you throw at them. The single core perfomance is also important.

yes perhaps. but I have seen many users with systems of lower spec getting around 20Gbits. Nor is there any significant CPU load during testing

gonna try plain linux on both machines and retest this weekend

So I installed Ubuntu 22.04 on the unraid box and interestingly the speed is not 3-4Gbits both ways between it and the windows box.

going to try to install windows on it and try that

 

Did you go to unraid main and toggle read/write view on?

This will show the on drive speed rather than the blackmagic disk speedster running via something else.

 

Edited January 13, 20242 yr by dopeytree