10GB Cache speed (slow)


Recommended Posts

Hey guys,

 

What kind of speeds are folks getting on their cache drives when copying to them over a 10GB network via SMB (no jumbo frames)?  I've got 2 Samsung PM953 nvme drives setup in brtfs raid 0 and can only get it to write at 400 - 500MB/s from my Windows desktop.  I have a Windows 2019 server on the same 10GB network that I'm able copy to at 1GB/s to a pair of SATA3 SSDs in raid 0.  Does anyone have any tuning advice?   Diags attached.  Thanks!

 

 

 

 

tower-diagnostics-20190514-0022.zip

Link to comment
4 hours ago, johnnie.black said:

I only use jumbo frames and Direct IO (Settings -> Global Share Settings) and get practically line speed.

 

What kind of cache are you using?  I have Direct IO enabled.   iperf leads me to believe the network side shouldn't be an issue.

 

Thanks for the response!

 

 

 

C:\temp\iperf-3.1.3-win64>iperf3.exe -P 10 -c 192.168.1.15
Connecting to host 192.168.1.15, port 5201

[  4] local 192.168.1.8 port 8667 connected to 192.168.1.15 port 5201
[  6] local 192.168.1.8 port 8668 connected to 192.168.1.15 port 5201
[  8] local 192.168.1.8 port 8669 connected to 192.168.1.15 port 5201
[ 10] local 192.168.1.8 port 8670 connected to 192.168.1.15 port 5201
[ 12] local 192.168.1.8 port 8671 connected to 192.168.1.15 port 5201
[ 14] local 192.168.1.8 port 8672 connected to 192.168.1.15 port 5201
[ 16] local 192.168.1.8 port 8673 connected to 192.168.1.15 port 5201
[ 18] local 192.168.1.8 port 8674 connected to 192.168.1.15 port 5201
[ 20] local 192.168.1.8 port 8675 connected to 192.168.1.15 port 5201
[ 22] local 192.168.1.8 port 8676 connected to 192.168.1.15 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  93.2 MBytes   782 Mbits/sec
[  6]   0.00-1.00   sec  92.8 MBytes   778 Mbits/sec
[  8]   0.00-1.00   sec  93.0 MBytes   780 Mbits/sec
[ 10]   0.00-1.00   sec  93.5 MBytes   784 Mbits/sec
[ 12]   0.00-1.00   sec  93.4 MBytes   783 Mbits/sec
[ 14]   0.00-1.00   sec  94.2 MBytes   790 Mbits/sec
[ 16]   0.00-1.00   sec  95.5 MBytes   801 Mbits/sec
[ 18]   0.00-1.00   sec  93.1 MBytes   781 Mbits/sec
[ 20]   0.00-1.00   sec  94.6 MBytes   793 Mbits/sec
[ 22]   0.00-1.00   sec  93.1 MBytes   781 Mbits/sec
[SUM]   0.00-1.00   sec   936 MBytes  7.85 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   1.00-2.00   sec   113 MBytes   946 Mbits/sec
[  6]   1.00-2.00   sec   113 MBytes   946 Mbits/sec
[  8]   1.00-2.00   sec   113 MBytes   946 Mbits/sec
[ 10]   1.00-2.00   sec   113 MBytes   946 Mbits/sec
[ 12]   1.00-2.00   sec   113 MBytes   946 Mbits/sec
[ 14]   1.00-2.00   sec   113 MBytes   945 Mbits/sec
[ 16]   1.00-2.00   sec   113 MBytes   946 Mbits/sec
[ 18]   1.00-2.00   sec   113 MBytes   945 Mbits/sec
[ 20]   1.00-2.00   sec   113 MBytes   945 Mbits/sec
[ 22]   1.00-2.00   sec   112 MBytes   944 Mbits/sec
[SUM]   1.00-2.00   sec  1.10 GBytes  9.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   2.00-3.00   sec   112 MBytes   936 Mbits/sec
[  6]   2.00-3.00   sec   112 MBytes   936 Mbits/sec
[  8]   2.00-3.00   sec   112 MBytes   936 Mbits/sec
[ 10]   2.00-3.00   sec   112 MBytes   936 Mbits/sec
[ 12]   2.00-3.00   sec   112 MBytes   937 Mbits/sec
[ 14]   2.00-3.00   sec   112 MBytes   936 Mbits/sec
[ 16]   2.00-3.00   sec   112 MBytes   936 Mbits/sec
[ 18]   2.00-3.00   sec   112 MBytes   936 Mbits/sec
[ 20]   2.00-3.00   sec   112 MBytes   936 Mbits/sec
[ 22]   2.00-3.00   sec   111 MBytes   935 Mbits/sec
[SUM]   2.00-3.00   sec  1.09 GBytes  9.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   3.00-4.00   sec   113 MBytes   945 Mbits/sec
[  6]   3.00-4.00   sec   113 MBytes   945 Mbits/sec
[  8]   3.00-4.00   sec   113 MBytes   947 Mbits/sec
[ 10]   3.00-4.00   sec   113 MBytes   947 Mbits/sec
[ 12]   3.00-4.00   sec   113 MBytes   947 Mbits/sec
[ 14]   3.00-4.00   sec   113 MBytes   945 Mbits/sec
[ 16]   3.00-4.00   sec   113 MBytes   947 Mbits/sec
[ 18]   3.00-4.00   sec   113 MBytes   945 Mbits/sec
[ 20]   3.00-4.00   sec   113 MBytes   944 Mbits/sec
[ 22]   3.00-4.00   sec   113 MBytes   944 Mbits/sec
[SUM]   3.00-4.00   sec  1.10 GBytes  9.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   4.00-5.00   sec   113 MBytes   946 Mbits/sec
[  6]   4.00-5.00   sec   113 MBytes   946 Mbits/sec
[  8]   4.00-5.00   sec   113 MBytes   945 Mbits/sec
[ 10]   4.00-5.00   sec   113 MBytes   945 Mbits/sec
[ 12]   4.00-5.00   sec   113 MBytes   945 Mbits/sec
[ 14]   4.00-5.00   sec   113 MBytes   945 Mbits/sec
[ 16]   4.00-5.00   sec   113 MBytes   945 Mbits/sec
[ 18]   4.00-5.00   sec   113 MBytes   945 Mbits/sec
[ 20]   4.00-5.00   sec   113 MBytes   944 Mbits/sec
[ 22]   4.00-5.00   sec   113 MBytes   944 Mbits/sec
[SUM]   4.00-5.00   sec  1.10 GBytes  9.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   5.00-6.00   sec   113 MBytes   946 Mbits/sec
[  6]   5.00-6.00   sec   113 MBytes   946 Mbits/sec
[  8]   5.00-6.00   sec   113 MBytes   946 Mbits/sec
[ 10]   5.00-6.00   sec   113 MBytes   945 Mbits/sec
[ 12]   5.00-6.00   sec   113 MBytes   946 Mbits/sec
[ 14]   5.00-6.00   sec   113 MBytes   945 Mbits/sec
[ 16]   5.00-6.00   sec   113 MBytes   945 Mbits/sec
[ 18]   5.00-6.00   sec   113 MBytes   945 Mbits/sec
[ 20]   5.00-6.00   sec   113 MBytes   945 Mbits/sec
[ 22]   5.00-6.00   sec   113 MBytes   945 Mbits/sec
[SUM]   5.00-6.00   sec  1.10 GBytes  9.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   6.00-7.00   sec   113 MBytes   946 Mbits/sec
[  6]   6.00-7.00   sec   113 MBytes   946 Mbits/sec
[  8]   6.00-7.00   sec   113 MBytes   946 Mbits/sec
[ 10]   6.00-7.00   sec   113 MBytes   946 Mbits/sec
[ 12]   6.00-7.00   sec   113 MBytes   946 Mbits/sec
[ 14]   6.00-7.00   sec   113 MBytes   945 Mbits/sec
[ 16]   6.00-7.00   sec   113 MBytes   946 Mbits/sec
[ 18]   6.00-7.00   sec   113 MBytes   946 Mbits/sec
[ 20]   6.00-7.00   sec   113 MBytes   945 Mbits/sec
[ 22]   6.00-7.00   sec   112 MBytes   944 Mbits/sec
[SUM]   6.00-7.00   sec  1.10 GBytes  9.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   7.00-8.00   sec   113 MBytes   946 Mbits/sec
[  6]   7.00-8.00   sec   113 MBytes   946 Mbits/sec
[  8]   7.00-8.00   sec   113 MBytes   946 Mbits/sec
[ 10]   7.00-8.00   sec   113 MBytes   946 Mbits/sec
[ 12]   7.00-8.00   sec   113 MBytes   946 Mbits/sec
[ 14]   7.00-8.00   sec   113 MBytes   946 Mbits/sec
[ 16]   7.00-8.00   sec   113 MBytes   946 Mbits/sec
[ 18]   7.00-8.00   sec   113 MBytes   945 Mbits/sec
[ 20]   7.00-8.00   sec   113 MBytes   945 Mbits/sec
[ 22]   7.00-8.00   sec   113 MBytes   945 Mbits/sec
[SUM]   7.00-8.00   sec  1.10 GBytes  9.46 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   8.00-9.00   sec   113 MBytes   946 Mbits/sec
[  6]   8.00-9.00   sec   113 MBytes   946 Mbits/sec
[  8]   8.00-9.00   sec   113 MBytes   946 Mbits/sec
[ 10]   8.00-9.00   sec   113 MBytes   946 Mbits/sec
[ 12]   8.00-9.00   sec   113 MBytes   946 Mbits/sec
[ 14]   8.00-9.00   sec   113 MBytes   945 Mbits/sec
[ 16]   8.00-9.00   sec   113 MBytes   946 Mbits/sec
[ 18]   8.00-9.00   sec   113 MBytes   945 Mbits/sec
[ 20]   8.00-9.00   sec   113 MBytes   945 Mbits/sec
[ 22]   8.00-9.00   sec   113 MBytes   945 Mbits/sec
[SUM]   8.00-9.00   sec  1.10 GBytes  9.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]   9.00-10.00  sec  96.4 MBytes   808 Mbits/sec
[  6]   9.00-10.00  sec  97.4 MBytes   817 Mbits/sec
[  8]   9.00-10.00  sec  96.1 MBytes   806 Mbits/sec
[ 10]   9.00-10.00  sec  96.5 MBytes   809 Mbits/sec
[ 12]   9.00-10.00  sec  96.1 MBytes   806 Mbits/sec
[ 14]   9.00-10.00  sec  97.1 MBytes   815 Mbits/sec
[ 16]   9.00-10.00  sec  98.5 MBytes   826 Mbits/sec
[ 18]   9.00-10.00  sec  96.2 MBytes   807 Mbits/sec
[ 20]   9.00-10.00  sec  97.6 MBytes   819 Mbits/sec
[ 22]   9.00-10.00  sec  96.2 MBytes   807 Mbits/sec
[SUM]   9.00-10.00  sec   968 MBytes  8.12 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.06 GBytes   915 Mbits/sec                  sender
[  4]   0.00-10.00  sec  1.06 GBytes   915 Mbits/sec                  receiver
[  6]   0.00-10.00  sec  1.07 GBytes   915 Mbits/sec                  sender
[  6]   0.00-10.00  sec  1.07 GBytes   915 Mbits/sec                  receiver
[  8]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec                  sender
[  8]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec                  receiver
[ 10]   0.00-10.00  sec  1.07 GBytes   915 Mbits/sec                  sender
[ 10]   0.00-10.00  sec  1.07 GBytes   915 Mbits/sec                  receiver
[ 12]   0.00-10.00  sec  1.06 GBytes   915 Mbits/sec                  sender
[ 12]   0.00-10.00  sec  1.06 GBytes   915 Mbits/sec                  receiver
[ 14]   0.00-10.00  sec  1.07 GBytes   916 Mbits/sec                  sender
[ 14]   0.00-10.00  sec  1.07 GBytes   916 Mbits/sec                  receiver
[ 16]   0.00-10.00  sec  1.07 GBytes   918 Mbits/sec                  sender
[ 16]   0.00-10.00  sec  1.07 GBytes   918 Mbits/sec                  receiver
[ 18]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec                  sender
[ 18]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec                  receiver
[ 20]   0.00-10.00  sec  1.07 GBytes   916 Mbits/sec                  sender
[ 20]   0.00-10.00  sec  1.07 GBytes   916 Mbits/sec                  receiver
[ 22]   0.00-10.00  sec  1.06 GBytes   913 Mbits/sec                  sender
[ 22]   0.00-10.00  sec  1.06 GBytes   913 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  10.7 GBytes  9.15 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  10.7 GBytes  9.15 Gbits/sec                  receiver

iperf Done.

 

Link to comment
2 minutes ago, johnnie.black said:

Do you get 10GbE speeds or close with a single thread? Multiple threads can give better results than single and one should be enough.

 

Single thread yields about half the speed.  Maybe this is my problem (although I don't know how to fix it)?  Two threads yields similar results to 10 threads like above.

 

Thanks!

 

C:\temp\iperf-3.1.3-win64>iperf3.exe -P 1 -c 192.168.1.15
Connecting to host 192.168.1.15, port 5201
[  4] local 192.168.1.8 port 9056 connected to 192.168.1.15 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   639 MBytes  5.36 Gbits/sec
[  4]   1.00-2.00   sec   640 MBytes  5.37 Gbits/sec
[  4]   2.00-3.00   sec   662 MBytes  5.55 Gbits/sec
[  4]   3.00-4.00   sec   655 MBytes  5.49 Gbits/sec
[  4]   4.00-5.00   sec   640 MBytes  5.37 Gbits/sec
[  4]   5.00-6.00   sec   636 MBytes  5.34 Gbits/sec
[  4]   6.00-7.00   sec   778 MBytes  6.52 Gbits/sec
[  4]   7.00-8.00   sec   632 MBytes  5.30 Gbits/sec
[  4]   8.00-9.00   sec   643 MBytes  5.39 Gbits/sec
[  4]   9.00-10.00  sec   682 MBytes  5.72 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  6.45 GBytes  5.54 Gbits/sec                  sender
[  4]   0.00-10.00  sec  6.45 GBytes  5.54 Gbits/sec                  receiver

iperf Done.
 

Link to comment
4 minutes ago, johnnie.black said:

I think so, I get get close to max with a single thread, and a copy works like a single thread.

 

Jumbo frames might help with that, if not it's likely the NIC hardware/cables.

 

I'll do some science tonight with jumbo frames.  I've also go another NIC I can try if that doesn't yield and results.  Wish me luck!  Thanks for the help.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.