noacess Posted May 14, 2019 Share Posted May 14, 2019 Hey guys, What kind of speeds are folks getting on their cache drives when copying to them over a 10GB network via SMB (no jumbo frames)? I've got 2 Samsung PM953 nvme drives setup in brtfs raid 0 and can only get it to write at 400 - 500MB/s from my Windows desktop. I have a Windows 2019 server on the same 10GB network that I'm able copy to at 1GB/s to a pair of SATA3 SSDs in raid 0. Does anyone have any tuning advice? Diags attached. Thanks! tower-diagnostics-20190514-0022.zip Link to comment
JorgeB Posted May 14, 2019 Share Posted May 14, 2019 I only use jumbo frames and Direct IO (Settings -> Global Share Settings) and get practically line speed. Link to comment
noacess Posted May 14, 2019 Author Share Posted May 14, 2019 4 hours ago, johnnie.black said: I only use jumbo frames and Direct IO (Settings -> Global Share Settings) and get practically line speed. What kind of cache are you using? I have Direct IO enabled. iperf leads me to believe the network side shouldn't be an issue. Thanks for the response! C:\temp\iperf-3.1.3-win64>iperf3.exe -P 10 -c 192.168.1.15 Connecting to host 192.168.1.15, port 5201 [ 4] local 192.168.1.8 port 8667 connected to 192.168.1.15 port 5201 [ 6] local 192.168.1.8 port 8668 connected to 192.168.1.15 port 5201 [ 8] local 192.168.1.8 port 8669 connected to 192.168.1.15 port 5201 [ 10] local 192.168.1.8 port 8670 connected to 192.168.1.15 port 5201 [ 12] local 192.168.1.8 port 8671 connected to 192.168.1.15 port 5201 [ 14] local 192.168.1.8 port 8672 connected to 192.168.1.15 port 5201 [ 16] local 192.168.1.8 port 8673 connected to 192.168.1.15 port 5201 [ 18] local 192.168.1.8 port 8674 connected to 192.168.1.15 port 5201 [ 20] local 192.168.1.8 port 8675 connected to 192.168.1.15 port 5201 [ 22] local 192.168.1.8 port 8676 connected to 192.168.1.15 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 93.2 MBytes 782 Mbits/sec [ 6] 0.00-1.00 sec 92.8 MBytes 778 Mbits/sec [ 8] 0.00-1.00 sec 93.0 MBytes 780 Mbits/sec [ 10] 0.00-1.00 sec 93.5 MBytes 784 Mbits/sec [ 12] 0.00-1.00 sec 93.4 MBytes 783 Mbits/sec [ 14] 0.00-1.00 sec 94.2 MBytes 790 Mbits/sec [ 16] 0.00-1.00 sec 95.5 MBytes 801 Mbits/sec [ 18] 0.00-1.00 sec 93.1 MBytes 781 Mbits/sec [ 20] 0.00-1.00 sec 94.6 MBytes 793 Mbits/sec [ 22] 0.00-1.00 sec 93.1 MBytes 781 Mbits/sec [SUM] 0.00-1.00 sec 936 MBytes 7.85 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 6] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 8] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 10] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 12] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 14] 1.00-2.00 sec 113 MBytes 945 Mbits/sec [ 16] 1.00-2.00 sec 113 MBytes 946 Mbits/sec [ 18] 1.00-2.00 sec 113 MBytes 945 Mbits/sec [ 20] 1.00-2.00 sec 113 MBytes 945 Mbits/sec [ 22] 1.00-2.00 sec 112 MBytes 944 Mbits/sec [SUM] 1.00-2.00 sec 1.10 GBytes 9.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 6] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 8] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 10] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 12] 2.00-3.00 sec 112 MBytes 937 Mbits/sec [ 14] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 16] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 18] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 20] 2.00-3.00 sec 112 MBytes 936 Mbits/sec [ 22] 2.00-3.00 sec 111 MBytes 935 Mbits/sec [SUM] 2.00-3.00 sec 1.09 GBytes 9.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-4.00 sec 113 MBytes 945 Mbits/sec [ 6] 3.00-4.00 sec 113 MBytes 945 Mbits/sec [ 8] 3.00-4.00 sec 113 MBytes 947 Mbits/sec [ 10] 3.00-4.00 sec 113 MBytes 947 Mbits/sec [ 12] 3.00-4.00 sec 113 MBytes 947 Mbits/sec [ 14] 3.00-4.00 sec 113 MBytes 945 Mbits/sec [ 16] 3.00-4.00 sec 113 MBytes 947 Mbits/sec [ 18] 3.00-4.00 sec 113 MBytes 945 Mbits/sec [ 20] 3.00-4.00 sec 113 MBytes 944 Mbits/sec [ 22] 3.00-4.00 sec 113 MBytes 944 Mbits/sec [SUM] 3.00-4.00 sec 1.10 GBytes 9.46 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 4.00-5.00 sec 113 MBytes 946 Mbits/sec [ 6] 4.00-5.00 sec 113 MBytes 946 Mbits/sec [ 8] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 10] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 12] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 14] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 16] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 18] 4.00-5.00 sec 113 MBytes 945 Mbits/sec [ 20] 4.00-5.00 sec 113 MBytes 944 Mbits/sec [ 22] 4.00-5.00 sec 113 MBytes 944 Mbits/sec [SUM] 4.00-5.00 sec 1.10 GBytes 9.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 5.00-6.00 sec 113 MBytes 946 Mbits/sec [ 6] 5.00-6.00 sec 113 MBytes 946 Mbits/sec [ 8] 5.00-6.00 sec 113 MBytes 946 Mbits/sec [ 10] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 12] 5.00-6.00 sec 113 MBytes 946 Mbits/sec [ 14] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 16] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 18] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 20] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [ 22] 5.00-6.00 sec 113 MBytes 945 Mbits/sec [SUM] 5.00-6.00 sec 1.10 GBytes 9.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 6] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 8] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 10] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 12] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 14] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 16] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 18] 6.00-7.00 sec 113 MBytes 946 Mbits/sec [ 20] 6.00-7.00 sec 113 MBytes 945 Mbits/sec [ 22] 6.00-7.00 sec 112 MBytes 944 Mbits/sec [SUM] 6.00-7.00 sec 1.10 GBytes 9.46 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 6] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 8] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 10] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 12] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 14] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 16] 7.00-8.00 sec 113 MBytes 946 Mbits/sec [ 18] 7.00-8.00 sec 113 MBytes 945 Mbits/sec [ 20] 7.00-8.00 sec 113 MBytes 945 Mbits/sec [ 22] 7.00-8.00 sec 113 MBytes 945 Mbits/sec [SUM] 7.00-8.00 sec 1.10 GBytes 9.46 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 6] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 8] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 10] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 12] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 14] 8.00-9.00 sec 113 MBytes 945 Mbits/sec [ 16] 8.00-9.00 sec 113 MBytes 946 Mbits/sec [ 18] 8.00-9.00 sec 113 MBytes 945 Mbits/sec [ 20] 8.00-9.00 sec 113 MBytes 945 Mbits/sec [ 22] 8.00-9.00 sec 113 MBytes 945 Mbits/sec [SUM] 8.00-9.00 sec 1.10 GBytes 9.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 9.00-10.00 sec 96.4 MBytes 808 Mbits/sec [ 6] 9.00-10.00 sec 97.4 MBytes 817 Mbits/sec [ 8] 9.00-10.00 sec 96.1 MBytes 806 Mbits/sec [ 10] 9.00-10.00 sec 96.5 MBytes 809 Mbits/sec [ 12] 9.00-10.00 sec 96.1 MBytes 806 Mbits/sec [ 14] 9.00-10.00 sec 97.1 MBytes 815 Mbits/sec [ 16] 9.00-10.00 sec 98.5 MBytes 826 Mbits/sec [ 18] 9.00-10.00 sec 96.2 MBytes 807 Mbits/sec [ 20] 9.00-10.00 sec 97.6 MBytes 819 Mbits/sec [ 22] 9.00-10.00 sec 96.2 MBytes 807 Mbits/sec [SUM] 9.00-10.00 sec 968 MBytes 8.12 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec sender [ 4] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec receiver [ 6] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec sender [ 6] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec receiver [ 8] 0.00-10.00 sec 1.06 GBytes 914 Mbits/sec sender [ 8] 0.00-10.00 sec 1.06 GBytes 914 Mbits/sec receiver [ 10] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec sender [ 10] 0.00-10.00 sec 1.07 GBytes 915 Mbits/sec receiver [ 12] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec sender [ 12] 0.00-10.00 sec 1.06 GBytes 915 Mbits/sec receiver [ 14] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec sender [ 14] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec receiver [ 16] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec sender [ 16] 0.00-10.00 sec 1.07 GBytes 918 Mbits/sec receiver [ 18] 0.00-10.00 sec 1.06 GBytes 914 Mbits/sec sender [ 18] 0.00-10.00 sec 1.06 GBytes 914 Mbits/sec receiver [ 20] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec sender [ 20] 0.00-10.00 sec 1.07 GBytes 916 Mbits/sec receiver [ 22] 0.00-10.00 sec 1.06 GBytes 913 Mbits/sec sender [ 22] 0.00-10.00 sec 1.06 GBytes 913 Mbits/sec receiver [SUM] 0.00-10.00 sec 10.7 GBytes 9.15 Gbits/sec sender [SUM] 0.00-10.00 sec 10.7 GBytes 9.15 Gbits/sec receiver iperf Done. Link to comment
JorgeB Posted May 14, 2019 Share Posted May 14, 2019 Just now, noacess said: What kind of cache are you using? One uses a single NVMe device and the other 11 SSDs in RAID5 Link to comment
JorgeB Posted May 14, 2019 Share Posted May 14, 2019 1 minute ago, noacess said: iperf leads me to believe the network side shouldn't be an issue. Do you get 10GbE speeds or close with a single thread? Multiple threads can give better results than single and one should be enough. Link to comment
noacess Posted May 14, 2019 Author Share Posted May 14, 2019 2 minutes ago, johnnie.black said: Do you get 10GbE speeds or close with a single thread? Multiple threads can give better results than single and one should be enough. Single thread yields about half the speed. Maybe this is my problem (although I don't know how to fix it)? Two threads yields similar results to 10 threads like above. Thanks! C:\temp\iperf-3.1.3-win64>iperf3.exe -P 1 -c 192.168.1.15 Connecting to host 192.168.1.15, port 5201 [ 4] local 192.168.1.8 port 9056 connected to 192.168.1.15 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 639 MBytes 5.36 Gbits/sec [ 4] 1.00-2.00 sec 640 MBytes 5.37 Gbits/sec [ 4] 2.00-3.00 sec 662 MBytes 5.55 Gbits/sec [ 4] 3.00-4.00 sec 655 MBytes 5.49 Gbits/sec [ 4] 4.00-5.00 sec 640 MBytes 5.37 Gbits/sec [ 4] 5.00-6.00 sec 636 MBytes 5.34 Gbits/sec [ 4] 6.00-7.00 sec 778 MBytes 6.52 Gbits/sec [ 4] 7.00-8.00 sec 632 MBytes 5.30 Gbits/sec [ 4] 8.00-9.00 sec 643 MBytes 5.39 Gbits/sec [ 4] 9.00-10.00 sec 682 MBytes 5.72 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 6.45 GBytes 5.54 Gbits/sec sender [ 4] 0.00-10.00 sec 6.45 GBytes 5.54 Gbits/sec receiver iperf Done. Link to comment
JorgeB Posted May 14, 2019 Share Posted May 14, 2019 Just now, noacess said: Single thread yields about half the speed. Maybe this is my problem I think so, I get get close to max with a single thread, and a copy works like a single thread. Jumbo frames might help with that, if not it's likely the NIC hardware/cables. Link to comment
noacess Posted May 14, 2019 Author Share Posted May 14, 2019 4 minutes ago, johnnie.black said: I think so, I get get close to max with a single thread, and a copy works like a single thread. Jumbo frames might help with that, if not it's likely the NIC hardware/cables. I'll do some science tonight with jumbo frames. I've also go another NIC I can try if that doesn't yield and results. Wish me luck! Thanks for the help. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.