May 14, 20197 yr Hey guys, What kind of speeds are folks getting on their cache drives when copying to them over a 10GB network via SMB (no jumbo frames)? I've got 2 Samsung PM953 nvme drives setup in brtfs raid 0 and can only get it to write at 400 - 500MB/s from my Windows desktop. I have a Windows 2019 server on the same 10GB network that I'm able copy to at 1GB/s to a pair of SATA3 SSDs in raid 0. Does anyone have any tuning advice? Diags attached. Thanks! tower-diagnostics-20190514-0022.zip
May 14, 20197 yr Community Expert I only use jumbo frames and Direct IO (Settings -> Global Share Settings) and get practically line speed.
May 14, 20197 yr Author 4 hours ago, johnnie.black said: I only use jumbo frames and Direct IO (Settings -> Global Share Settings) and get practically line speed. What kind of cache are you using? I have Direct IO enabled. iperf leads me to believe the network side shouldn't be an issue. Thanks for the response! C:\temp\iperf-3.1.3-win64>iperf3.exe -P 10 -c 192.168.1.15 Connecting to host 192.168.1.15, port 5201 [ 4] local 192.168.1.8 port 8667 connected to 192.168.1.15 port 5201 [ 6] local 192.168.1.8 port 8668 connected to 192.168.1.15 port 5201 [ 8] local 192.168.1.8 port 8669 connected to 192.168.1.15 port 5201 [ 10] local 192.168.1.8 port 8670 connected to 192.168.1.15 port 5201 [ 12] local 192.168.1.8 port 8671 connected to 192.168.1.15 port 5201 [ 14] local 192.168.1.8 port 8672 connected to 192.168.1.15 port 5201 [ 16] local 192.168.1.8 port 8673 connected to 192.168.1.15 port 5201 [ 18] local 192.168.1.8 port 8674 connected to 192.168.1.15 port 5201 [ 20] local 192.168.1.8 port 8675 connected to 192.168.1.15 port 5201 [ 22] local 192.168.1.8 port 8676 connected to 192.168.1.15 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 93.2 MBytes 782 Mbits/sec [ 6] 0.00-1.00 sec 92.8 MBytes 778 Mbits/sec [ 8] 0.00-1.00 sec 93.0 MBytes 780 Mbits/sec [ 10] 0.00-1.00 sec 93.5 MBytes 784 Mbits/sec [ 12] 0.00-1.00 sec 93.4 MBytes 783 Mbits/sec [ 14] 0.00-1.00 sec 94.2 MBytes 790 Mbits/sec [ 16] 0.00-1.00 sec 95.5 MBytes 801 Mbits/sec [ 18] 0.00-1.00 sec 93.1 MBytes 781 Mbits/sec [ 20] 0.00-1.00 sec 94.6 MBytes 793 Mbits/sec [ 22] 0.00-1.00 sec 93.1 MBytes 781 Mbits/sec [SUM] 0.00-1.00 sec 936 MBytes 7.85 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 6] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 8] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 10] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 12] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 14] 1.00-2.00 sec 113 MBytes 945 Mbits/sec [ 16] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 18] 1.00-2.00 sec 113 MBytes 945 Mbits/sec [ 20] 1.00-2.00 sec 113 MBytes 945 Mbits/sec [ 22] 1.00-2.00 sec 112 MBytes 944 Mbits/sec [SUM] 1.00-2.00 sec 1.10 GBytes 9.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 6] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 8] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 10] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 12] 2.00-3.00 sec 112 MBytes 937 Mbits/sec [ 14] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 16] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 18] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 20] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 22] 2.00-3.00 sec 111 MBytes 935 Mbits/sec [SUM] 2.00-3.00 sec 1.09 GBytes 9.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-4.00 sec 113 MBytes 945 Mbits/sec [ 6] 3.00-4.00 sec 113 MBytes 945 Mbits/sec [ 8] 3.00-4.00 sec 113 MBytes 947 Mbits/sec [ 10] 3.00-4.00 sec 113 MBytes 947 Mbits/sec [ 12] 3.00-4.00 sec 113 MBytes 947 Mbits/sec [ 14] 3.00-4.00 sec 113 MBytes 945 Mbits/sec [ 16] 3.00-4.00 sec 113 MBytes 947 Mbits/sec [ 18] 3.00-4.00 sec 113 MBytes 945 Mbits/sec [ 20] 3.00-4.00 sec 113 MBytes 944 Mbits/sec [ 22] 3.00-4.00 sec 113 MBytes 944 Mbits/sec [SUM] 3.00-4.00 sec 1.10 GBytes 9.46 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 4.00-5.00 sec 113 MBytes 946 Mbits/sec [ 6] 4.00-5.00 sec 113 MBytes 946 Mbits/sec [ 8] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 10] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 12] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 14] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 16] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 18] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 20] 4.00-5.00 sec 113 MBytes 944 Mbits/sec [ 22] 4.00-5.00 sec 113 MBytes 944 Mbits/sec [SUM] 4.00-5.00 sec 1.10 GBytes 9.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 5.00-6.00 sec 113 MBytes 946 Mbits/sec [ 6] 5.00-6.00 sec 113 MBytes 946 Mbits/sec [ 8] 5.00-6.00 sec 113 MBytes 946 Mbits/sec [ 10] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 12] 5.00-6.00 sec 113 MBytes 946 Mbits/sec [ 14] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 16] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 18] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 20] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 22] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [SUM] 5.00-6.00 sec 1.10 GBytes 9.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 6] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 8] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 10] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 12] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 14] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 16] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 18] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 20] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 22] 6.00-7.00 sec 112 MBytes 944 Mbits/sec [SUM] 6.00-7.00 sec 1.10 GBytes 9.46 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 6] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 8] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 10] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 12] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 14] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 16] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 18] 7.00-8.00 sec 113 MBytes 945 Mbits/sec [ 20] 7.00-8.00 sec 113 MBytes 945 Mbits/sec [ 22] 7.00-8.00 sec 113 MBytes 945 Mbits/sec [SUM] 7.00-8.00 sec 1.10 GBytes 9.46 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 6] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 8] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 10] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 12] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 14] 8.00-9.00 sec 113 MBytes 945 Mbits/sec [ 16] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 18] 8.00-9.00 sec 113 MBytes 945 Mbits/sec [ 20] 8.00-9.00 sec 113 MBytes 945 Mbits/sec [ 22] 8.00-9.00 sec 113 MBytes 945 Mbits/sec [SUM] 8.00-9.00 sec 1.10 GBytes 9.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 9.00-10.00 sec 96.4 MBytes 808 Mbits/sec [ 6] 9.00-10.00 sec 97.4 MBytes 817 Mbits/sec [ 8] 9.00-10.00 sec 96.1 MBytes 806 Mbits/sec [ 10] 9.00-10.00 sec 96.5 MBytes 809 Mbits/sec [ 12] 9.00-10.00 sec 96.1 MBytes 806 Mbits/sec [ 14] 9.00-10.00 sec 97.1 MBytes 815 Mbits/sec [ 16] 9.00-10.00 sec 98.5 MBytes 826 Mbits/sec [ 18] 9.00-10.00 sec 96.2 MBytes 807 Mbits/sec [ 20] 9.00-10.00 sec 97.6 MBytes 819 Mbits/sec [ 22] 9.00-10.00 sec 96.2 MBytes 807 Mbits/sec [SUM] 9.00-10.00 sec 968 MBytes 8.12 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec sender [ 4] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec receiver [ 6] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec sender [ 6] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec receiver [ 8] 0.00-10.00 sec 1.06 GBytes 914 Mbits/sec sender [ 8] 0.00-10.00 sec 1.06 GBytes 914 Mbits/sec receiver [ 10] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec sender [ 10] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec receiver [ 12] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec sender [ 12] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec receiver [ 14] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec sender [ 14] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec receiver [ 16] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec sender [ 16] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec receiver [ 18] 0.00-10.00 sec 1.06 GBytes 914 Mbits/sec sender [ 18] 0.00-10.00 sec 1.06 GBytes 914 Mbits/sec receiver [ 20] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec sender [ 20] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec receiver [ 22] 0.00-10.00 sec 1.06 GBytes 913 Mbits/sec sender [ 22] 0.00-10.00 sec 1.06 GBytes 913 Mbits/sec receiver [SUM] 0.00-10.00 sec 10.7 GBytes 9.15 Gbits/sec sender [SUM] 0.00-10.00 sec 10.7 GBytes 9.15 Gbits/sec receiver iperf Done.
May 14, 20197 yr Community Expert Just now, noacess said: What kind of cache are you using? One uses a single NVMe device and the other 11 SSDs in RAID5
May 14, 20197 yr Community Expert 1 minute ago, noacess said: iperf leads me to believe the network side shouldn't be an issue. Do you get 10GbE speeds or close with a single thread? Multiple threads can give better results than single and one should be enough.
May 14, 20197 yr Author 2 minutes ago, johnnie.black said: Do you get 10GbE speeds or close with a single thread? Multiple threads can give better results than single and one should be enough. Single thread yields about half the speed. Maybe this is my problem (although I don't know how to fix it)? Two threads yields similar results to 10 threads like above. Thanks! C:\temp\iperf-3.1.3-win64>iperf3.exe -P 1 -c 192.168.1.15 Connecting to host 192.168.1.15, port 5201 [ 4] local 192.168.1.8 port 9056 connected to 192.168.1.15 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 639 MBytes 5.36 Gbits/sec [ 4] 1.00-2.00 sec 640 MBytes 5.37 Gbits/sec [ 4] 2.00-3.00 sec 662 MBytes 5.55 Gbits/sec [ 4] 3.00-4.00 sec 655 MBytes 5.49 Gbits/sec [ 4] 4.00-5.00 sec 640 MBytes 5.37 Gbits/sec [ 4] 5.00-6.00 sec 636 MBytes 5.34 Gbits/sec [ 4] 6.00-7.00 sec 778 MBytes 6.52 Gbits/sec [ 4] 7.00-8.00 sec 632 MBytes 5.30 Gbits/sec [ 4] 8.00-9.00 sec 643 MBytes 5.39 Gbits/sec [ 4] 9.00-10.00 sec 682 MBytes 5.72 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 6.45 GBytes 5.54 Gbits/sec sender [ 4] 0.00-10.00 sec 6.45 GBytes 5.54 Gbits/sec receiver iperf Done.
May 14, 20197 yr Community Expert Just now, noacess said: Single thread yields about half the speed. Maybe this is my problem I think so, I get get close to max with a single thread, and a copy works like a single thread. Jumbo frames might help with that, if not it's likely the NIC hardware/cables.
May 14, 20197 yr Author 4 minutes ago, johnnie.black said: I think so, I get get close to max with a single thread, and a copy works like a single thread. Jumbo frames might help with that, if not it's likely the NIC hardware/cables. I'll do some science tonight with jumbo frames. I've also go another NIC I can try if that doesn't yield and results. Wish me luck! Thanks for the help.
Archived
This topic is now archived and is closed to further replies.