iap128 Posted June 20, 2020 Share Posted June 20, 2020 I'm sure this comes up a lot, but I've tried various things and nothing seems to help with my issues. I recently setup a new Unraid server and I'm not getting the read/write speeds expected. Writing to the server gets around 275mb/s where reading is up to 450mb/s. Here is a list of my hardware: Dell PowerEdge R710 SAS 6/IR 2 2TB Seagate Barracuda drives 500GB Samsung 970 Evo Plus in a PCIe 2.0 x8 slot (set as cache) 10GB SFP+ card Netgear XS708E switch Sonnet 10GB Thunderbolt 2 external NIC I've done a few performance tweaks to the settings with little to no difference: Jumbo frames on server set to 9000 Direct IO set to "yes" Reconstruct write enabled I've tested the Sonnet adapter on both Mac and Windows. I changed the jumbo frames on there as well to 9000. My Netgear switch is managed, but there's no option for setting the jumbo frames values. From what I can tell, it just passes whatever through. I'm not sure if this is a limitation of my hardware or not, but I'd appreciate any help you could provide. Quote Link to comment
JorgeB Posted June 20, 2020 Share Posted June 20, 2020 Start by running a single stream iperf test to check if the LAN is performing as expected. Quote Link to comment
iap128 Posted June 20, 2020 Author Share Posted June 20, 2020 Sure, here is what I got: Quote Link to comment
JorgeB Posted June 21, 2020 Share Posted June 21, 2020 That suggest a problem with the LAN itself, you should get close to line speed with iperf when all is working as it should be. Quote Link to comment
Vr2Io Posted June 21, 2020 Share Posted June 21, 2020 (edited) 17 hours ago, iap128 said: Sonnet 10GB Thunderbolt 2 external NIC From some other forums user report, TB to Ethernet always perform not well. 17 hours ago, iap128 said: server gets around 275mb/s where reading is up to 450mb/s. So 4.5Gbps seems be network ceiling, and ~3Gbps also common in Unraid's write performance. Your 970evo run in PCIe 2.0 also affect a little bit. Edited June 21, 2020 by Benson Quote Link to comment
jms2321 Posted June 23, 2020 Share Posted June 23, 2020 (edited) On 6/21/2020 at 4:50 AM, johnnie.black said: That suggest a problem with the LAN itself, you should get close to line speed with iperf when all is working as it should be. It would suggest that except I have a 10Gbe fiber dual connections (one point to point and the other via 10Gbe switch an Aruba S2500) between Unraid and Proxmox. The 192.168.1.x has the switch, the 192.168.2.x is direct connection Going from Unraid to Proxmox I get the following when running iperf3. root@Tower:~# iperf3 -c 192.168.2.11 Connecting to host 192.168.2.11, port 5201 [ 4] local 192.168.2.10 port 37210 connected to 192.168.2.11 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.09 GBytes 9.35 Gbits/sec 0 345 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.34 Gbits/sec 0 337 KBytes [ 4] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 0 345 KBytes [ 4] 3.00-4.00 sec 1.09 GBytes 9.41 Gbits/sec 0 337 KBytes [ 4] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 0 342 KBytes [ 4] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 337 KBytes [ 4] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 337 KBytes [ 4] 7.00-8.00 sec 1.09 GBytes 9.34 Gbits/sec 0 339 KBytes [ 4] 8.00-9.00 sec 1.07 GBytes 9.18 Gbits/sec 0 339 KBytes [ 4] 9.00-10.00 sec 1.07 GBytes 9.18 Gbits/sec 0 342 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 10.9 GBytes 9.34 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 10.9 GBytes 9.34 Gbits/sec receiver iperf Done. root@Tower:~# iperf3 -c 192.168.1.101 Connecting to host 192.168.1.101, port 5201 [ 4] local 192.168.1.109 port 52582 connected to 192.168.1.101 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 1.07 GBytes 9.22 Gbits/sec 0 393 KBytes [ 4] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0 399 KBytes [ 4] 2.00-3.00 sec 1.09 GBytes 9.40 Gbits/sec 0 396 KBytes [ 4] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 396 KBytes [ 4] 4.00-5.00 sec 1.08 GBytes 9.26 Gbits/sec 0 370 KBytes [ 4] 5.00-6.00 sec 1.05 GBytes 9.05 Gbits/sec 0 362 KBytes [ 4] 6.00-7.00 sec 1.06 GBytes 9.07 Gbits/sec 0 373 KBytes [ 4] 7.00-8.00 sec 1.06 GBytes 9.12 Gbits/sec 0 376 KBytes [ 4] 8.00-9.00 sec 1.06 GBytes 9.15 Gbits/sec 0 370 KBytes [ 4] 9.00-10.00 sec 1.05 GBytes 9.05 Gbits/sec 0 376 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 10.7 GBytes 9.21 Gbits/sec 0 sender [ 4] 0.00-10.00 sec 10.7 GBytes 9.21 Gbits/sec receiver iperf Done. This looks great, But here what I get when I run from the Proxmox to Unraid, again only 10GBe ports one via the switch 192.168.1.x and one direct connect 192.168.2.x root@pve:~# iperf3 -c 192.168.1.109 -f M Connecting to host 192.168.1.109, port 5201 [ 5] local 192.168.1.101 port 47336 connected to 192.168.1.109 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 346 MBytes 346 MBytes/sec 3861 150 KBytes [ 5] 1.00-2.00 sec 352 MBytes 352 MBytes/sec 5142 209 KBytes [ 5] 2.00-3.00 sec 335 MBytes 335 MBytes/sec 4307 156 KBytes [ 5] 3.00-4.00 sec 367 MBytes 367 MBytes/sec 3999 236 KBytes [ 5] 4.00-5.00 sec 320 MBytes 320 MBytes/sec 3519 188 KBytes [ 5] 5.00-6.00 sec 336 MBytes 336 MBytes/sec 4205 168 KBytes [ 5] 6.00-7.00 sec 338 MBytes 338 MBytes/sec 3999 136 KBytes [ 5] 7.00-8.00 sec 315 MBytes 315 MBytes/sec 4095 153 KBytes [ 5] 8.00-9.00 sec 333 MBytes 333 MBytes/sec 3397 141 KBytes [ 5] 9.00-10.00 sec 354 MBytes 354 MBytes/sec 3925 146 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 3.32 GBytes 340 MBytes/sec 40449 sender [ 5] 0.00-10.00 sec 3.31 GBytes 339 MBytes/sec receiver iperf Done. root@pve:~# iperf3 -c 192.168.2.10 -f M Connecting to host 192.168.2.10, port 5201 [ 5] local 192.168.2.11 port 56706 connected to 192.168.2.10 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 980 MBytes 979 MBytes/sec 178 762 KBytes [ 5] 1.00-2.00 sec 999 MBytes 999 MBytes/sec 123 860 KBytes [ 5] 2.00-3.00 sec 1004 MBytes 1004 MBytes/sec 116 611 KBytes [ 5] 3.00-4.00 sec 1006 MBytes 1006 MBytes/sec 519 614 KBytes [ 5] 4.00-5.00 sec 992 MBytes 992 MBytes/sec 171 803 KBytes [ 5] 5.00-6.00 sec 986 MBytes 987 MBytes/sec 242 626 KBytes [ 5] 6.00-7.00 sec 1014 MBytes 1014 MBytes/sec 204 819 KBytes [ 5] 7.00-8.00 sec 991 MBytes 991 MBytes/sec 208 662 KBytes [ 5] 8.00-9.00 sec 1012 MBytes 1012 MBytes/sec 602 802 KBytes [ 5] 9.00-10.00 sec 1004 MBytes 1004 MBytes/sec 190 824 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 9.75 GBytes 999 MBytes/sec 2553 sender [ 5] 0.00-10.00 sec 9.75 GBytes 999 MBytes/sec receiver iperf Done. Checking the switch ports and I see both the ports are configured as 10G/full duplex From the Aruba Switch CLI (ArubaS2500-48T) (config) #show interface gigabitethernet 1/1/0 GE1/1/0 is administratively Up, Link is Up, Line protocol is Up Hardware is Gigabit Ethernet, Interface is GE1/1/0, Address is 00:0b:86:8b:fe:b2 Encapsulation ARPA, Loopback not set Configured: duplex (Full), Speed (10 Gbps), FC (Off), Autoneg (Off) Interface index: 273 MTU 1514 bytes Link flaps: 3 (ArubaS2500-48T) (config) #show interface gigabitethernet 1/1/1 GE1/1/1 is administratively Up, Link is Up, Line protocol is Up Hardware is Gigabit Ethernet, Interface is GE1/1/1, Address is 00:0b:86:8b:fe:b3 Encapsulation ARPA, Loopback not set Configured: duplex (Full), Speed (10 Gbps), FC (Off), Autoneg (Off) Interface index: 274 MTU 1514 bytes Link flaps: 12 This problem is worse if I enable the 1G port, it drops to 100MBytes/sec on the 10G connection thru the switch, but only going to the Unraid. Also note the number of retries when going from proxmox to unraid, but no retries from Unraid to Proxmox. Edited June 23, 2020 by jms2321 added detail Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.