Ford Prefect Posted September 14, 2021 Share Posted September 14, 2021 (edited) 16 minutes ago, KptnKMan said: I just finished getting everything installed and setting up the links, and looks like the core CRS309+CSS610 have booted up ok and pushing data. For the CSS610 make sure you update it to SwOS-2.14lite...you should be able to see it from its update menu. For the CRS309, when on SwOS, also do an update....when in RouterOS upgrade to the latest "long-term" until you hit a flaw/problem listed to be fixed in "stable". 16 minutes ago, KptnKMan said: Before I get into setting them up, I seem to have run into a strange issue with the MACs of the interfaces. The bonding configuration seems to think there are duplicate MAC adresses: Ugh! OK, when you want to start networking from scratch, delete these two files. root@alarm:/boot/config# ls -l network* -rw------- 1 root root 383 Sep 9 22:37 network-rules.cfg -rw------- 1 root root 1052 Sep 5 01:45 network.cfg ...and do not connect the 1G onboard NIC...this should start the 10G nic as eth0 (do you have a DHCP-Server active on that Link (otherwise a local Monitor to enter/boot in GUI mode? Do de-activate bonding mode, only bridge-mode with only eth0 as part if the bridge....you can enhance from there later. Also, check the Firmware of the cards. @ich777 does have a tool lying around somewhere. Edit: a tool for flashing/upgrading a mellanox card, I mean. Edited September 14, 2021 by Ford Prefect Quote Link to comment
Ford Prefect Posted September 14, 2021 Share Posted September 14, 2021 13 minutes ago, KptnKMan said: I think I may need to open a new thread for this issue. Try this in addition to what I wrote in my other post: 1 Quote Link to comment
KptnKMan Posted September 14, 2021 Author Share Posted September 14, 2021 8 minutes ago, Ford Prefect said: Ugh! OK, when you want to start networking from scratch, delete these two files. Actually, yeah I think I fixed it. I took a look in those files after I posted and just deleted the listing for eth3. Seemed to work: 9 minutes ago, Ford Prefect said: Try this in addition to what I wrote in my other post: I think I might have fixed it, by deleting the eth3 entry from the network-rules.cfg I have the Mellanox as primary now, need to setup the second server and do some tests. 12 minutes ago, Ford Prefect said: ...and do not connect the 1G onboard NIC...this should start the 10G nic as eth0 (do you have a DHCP-Server active on that Link (otherwise a local Monitor to enter/boot in GUI mode? Do de-activate bonding mode, only bridge-mode with only eth0 as part if the bridge....you can enhance from there later. Oh I see, I was thinking I could use the MAC of my onboard as the bond0 MAC. Maybe that's a bad idea then, and I should just configure the Mellanox as the only card. I should probably do that, just reconfigure the router DHCP rather than be lazy. 😄 17 minutes ago, Ford Prefect said: Also, check the Firmware of the cards. @ich777 does have a tool lying around somewhere. Edit: a tool for flashing/upgrading a mellanox card, I mean. That sounds useful. 29 minutes ago, Ford Prefect said: Try this in addition to what I wrote in my other post: I'll definitely read through this, thanks. 1 Quote Link to comment
KptnKMan Posted September 14, 2021 Author Share Posted September 14, 2021 Alright... so it looks like I got it working. Reconfigured the DHCP and the active-backup bond0 seems to be up without trouble. some new iperf3 results between servers: Quote root@unraid1:~# iperf3 -c 192.168.178.112 Connecting to host 192.168.178.112, port 5201 [ 5] local 192.168.178.111 port 53912 connected to 192.168.178.112 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 931 MBytes 7.81 Gbits/sec 0 362 KBytes [ 5] 1.00-2.00 sec 706 MBytes 5.92 Gbits/sec 0 311 KBytes [ 5] 2.00-3.00 sec 771 MBytes 6.47 Gbits/sec 0 277 KBytes [ 5] 3.00-4.00 sec 868 MBytes 7.28 Gbits/sec 0 291 KBytes [ 5] 4.00-5.00 sec 929 MBytes 7.79 Gbits/sec 0 291 KBytes [ 5] 5.00-6.00 sec 889 MBytes 7.46 Gbits/sec 0 297 KBytes [ 5] 6.00-7.00 sec 1001 MBytes 8.40 Gbits/sec 0 300 KBytes [ 5] 7.00-8.00 sec 888 MBytes 7.45 Gbits/sec 0 288 KBytes [ 5] 8.00-9.00 sec 786 MBytes 6.60 Gbits/sec 0 288 KBytes [ 5] 9.00-10.00 sec 815 MBytes 6.84 Gbits/sec 0 305 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.38 GBytes 7.20 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.38 GBytes 7.20 Gbits/sec receiver root@unraid2:~# iperf3 -c 192.168.178.111 Connecting to host 192.168.178.111, port 5201 [ 5] local 192.168.178.112 port 60536 connected to 192.168.178.111 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 703 MBytes 5.90 Gbits/sec 0 297 KBytes [ 5] 1.00-2.00 sec 1021 MBytes 8.57 Gbits/sec 0 272 KBytes [ 5] 2.00-3.00 sec 1.06 GBytes 9.13 Gbits/sec 0 274 KBytes [ 5] 3.00-4.00 sec 869 MBytes 7.29 Gbits/sec 0 269 KBytes [ 5] 4.00-5.00 sec 1.06 GBytes 9.11 Gbits/sec 0 269 KBytes [ 5] 5.00-6.00 sec 1.06 GBytes 9.14 Gbits/sec 0 272 KBytes [ 5] 6.00-7.00 sec 1.07 GBytes 9.20 Gbits/sec 0 272 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.12 Gbits/sec 0 266 KBytes [ 5] 8.00-9.00 sec 1.06 GBytes 9.14 Gbits/sec 0 260 KBytes [ 5] 9.00-10.00 sec 1.07 GBytes 9.17 Gbits/sec 0 308 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 9.99 GBytes 8.58 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 9.98 GBytes 8.57 Gbits/sec receiver Not quite full 10Gb, especially as they are in physical x16 ports... but pretty good. 😁 1 Quote Link to comment
Ford Prefect Posted September 14, 2021 Share Posted September 14, 2021 14 minutes ago, KptnKMan said: Alright... so it looks like I got it working. Reconfigured the DHCP and the active-backup bond0 seems to be up without trouble. ...cool thing. 14 minutes ago, KptnKMan said: some new iperf3 results between servers: Not quite full 10Gb, especially as they are in physical x16 ports... but pretty good. 😁 Hmmm...is this with a direct link between servers or are both connected to the CRS309? Quote Link to comment
KptnKMan Posted September 14, 2021 Author Share Posted September 14, 2021 1 hour ago, Ford Prefect said: For the CSS610 make sure you update it to SwOS-2.14lite...you should be able to see it from its update menu. For the CRS309, when on SwOS, also do an update....when in RouterOS upgrade to the latest "long-term" until you hit a flaw/problem listed to be fixed in "stable". Thanks, I'm definitely going to be updating and investigating more into these in the coming days. In recent weeks I've already been reading and watching setup guides and configuration instructions. I haven't used any Mikrotik switches before, so it's going to be lots to learn. 😁 3 minutes ago, Ford Prefect said: Hmmm...is this with a direct link between servers or are both connected to the CRS309? This is via the CRS309, using the DAC cables I got included with the Mellanox cards. Bearing in mind I haven't adjusted anything, its all just plugged in and pushing data right now. The test show they don't seem to be running at 100% full speed, but I think I'll watch overnight and see how the throughput goes when backups run between servers. The current core setup, in its current position, due to be moved at some point soon: The unraid servers are hooked into SFP+ ports 3 and 4. The other CSS610 on the lab/desk will be setup and uplinked in the next days. Quote Link to comment
ich777 Posted September 14, 2021 Share Posted September 14, 2021 1 hour ago, KptnKMan said: That sounds useful. There you go (since 6.9.0-beta23 in the CA App): On the plugin page is a Guide to flash the cards, crossflash, remove the boot ROM and you also can change the mode if you are running Mellanox Infniband Cards to Ethernet if they support it, but please keep in mind that these are the open source Mellanox Firmware tools and they all start with mst... and not only with flint like the most tutorial mention, but this is basically the same. 2 Quote Link to comment
Ford Prefect Posted September 14, 2021 Share Posted September 14, 2021 (edited) 1 hour ago, KptnKMan said: Thanks, I'm definitely going to be updating and investigating more into these in the coming days. In recent weeks I've already been reading and watching setup guides and configuration instructions. I haven't used any Mikrotik switches before, so it's going to be lots to learn. 😁 ...just let me know if you need some more insights. 1 hour ago, KptnKMan said: This is via the CRS309, using the DAC cables I got included with the Mellanox cards. Bearing in mind I haven't adjusted anything, its all just plugged in and pushing data right now. The test show they don't seem to be running at 100% full speed, but I think I'll watch overnight and see how the throughput goes when backups run between servers. ...agreed, that is not full speed....should gibe 9.5Mbps easily. 1 hour ago, KptnKMan said: The unraid servers are hooked into SFP+ ports 3 and 4. Some thoughts: - OK, so the bond is *active* backup....active means, that there should be some "management" between them, which cannot take place when ports are across switches. The CRS on RouterOS definitely has a separate config for this, where you should declare the Master-port as well. Edit2: forget this, that setting is for a bond setup in the main switch, faning out to others. - I'd gather you did not check the default config of the CRS, when in RouterOS mode (should be the default OS booting, for a CRS). Maybe the bridge setup is not fully configured. As said, RouterOS is a bit of a learning curve. Edit: OK, thinking of it, the bond0 - meaning the 10G+1G has a single same MAC (that of the first NIC, master = eth0 = 10G). I thing your loosing performance on SRTP between the switches, each seeing the same MAC (CRS for 10G and CSS for 1G). The fact that this is not as bad, is that there is a 10G link between CRS309 and CSS610. But I think some packets might be flowing in cicles See Edit2 above....this should not impose problem...however, the performance is a bit too slow. You should try and disable the bond0 for a first start. Then test again. Edited September 14, 2021 by Ford Prefect deleted wrongly defined usage of active backup bond Quote Link to comment
KptnKMan Posted September 19, 2021 Author Share Posted September 19, 2021 (edited) On 9/14/2021 at 10:40 PM, Ford Prefect said: ...just let me know if you need some more insights. I do have a few questions, maybe there's some things I'm missing. On 9/14/2021 at 10:40 PM, Ford Prefect said: ...agreed, that is not full speed....should gibe 9.5Mbps easily. So I monitored this over the last few days, and the speed has been fairly consistent, but still not at the 9.5 that I figure it should be capable of. There are backups that runs every night between servers (for VMS, docker appdata, personal files, etc) and I have a particular VM that has a primary vdisk of 200GB. That particular backup used to consistently take about 31 minutes, 19 seconds. Now the same backup runs at about 3 minutes, 44 seconds. I've not been seeing any errors of packet loss in any of the logs as well, but I've rerun a few tests over the last few days between servers. I'm pretty happy with it, but it definitely should be doing a touch better given the specs in both systems, so there should be no shortage of bandwidth or bottlenecks that I'm aware of: Quote ### POST-UPGRADE SPEED TESTS - unraid1 root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 44118 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.07 GBytes 9.20 Gbits/sec 0 303 KBytes [ 5] 1.00-2.00 sec 1.06 GBytes 9.14 Gbits/sec 0 291 KBytes [ 5] 2.00-3.00 sec 1.07 GBytes 9.20 Gbits/sec 0 277 KBytes [ 5] 3.00-4.00 sec 1.06 GBytes 9.12 Gbits/sec 0 283 KBytes [ 5] 4.00-5.00 sec 1.06 GBytes 9.10 Gbits/sec 0 320 KBytes [ 5] 5.00-6.00 sec 1.07 GBytes 9.18 Gbits/sec 0 274 KBytes [ 5] 6.00-7.00 sec 1.06 GBytes 9.13 Gbits/sec 0 294 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.14 Gbits/sec 0 297 KBytes [ 5] 8.00-9.00 sec 1.06 GBytes 9.14 Gbits/sec 0 288 KBytes [ 5] 9.00-10.00 sec 1.07 GBytes 9.16 Gbits/sec 0 297 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.7 GBytes 9.15 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.6 GBytes 9.15 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 47008 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.01 GBytes 8.68 Gbits/sec 0 334 KBytes [ 5] 1.00-2.00 sec 1.03 GBytes 8.82 Gbits/sec 0 351 KBytes [ 5] 2.00-3.00 sec 1.06 GBytes 9.07 Gbits/sec 0 286 KBytes [ 5] 3.00-4.00 sec 1.06 GBytes 9.13 Gbits/sec 0 311 KBytes [ 5] 4.00-5.00 sec 1.06 GBytes 9.09 Gbits/sec 0 274 KBytes [ 5] 5.00-6.00 sec 1.06 GBytes 9.09 Gbits/sec 0 320 KBytes [ 5] 6.00-7.00 sec 1.06 GBytes 9.13 Gbits/sec 0 291 KBytes [ 5] 7.00-8.00 sec 1.07 GBytes 9.15 Gbits/sec 0 294 KBytes [ 5] 8.00-9.00 sec 1.06 GBytes 9.10 Gbits/sec 0 286 KBytes [ 5] 9.00-10.00 sec 1.06 GBytes 9.12 Gbits/sec 0 325 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.5 GBytes 9.04 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.5 GBytes 9.03 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 47024 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1015 MBytes 8.51 Gbits/sec 0 427 KBytes [ 5] 1.00-2.00 sec 985 MBytes 8.26 Gbits/sec 0 390 KBytes [ 5] 2.00-3.00 sec 1.01 GBytes 8.65 Gbits/sec 0 277 KBytes [ 5] 3.00-4.00 sec 1.06 GBytes 9.13 Gbits/sec 0 283 KBytes [ 5] 4.00-5.00 sec 1.07 GBytes 9.18 Gbits/sec 0 322 KBytes [ 5] 5.00-6.00 sec 1.07 GBytes 9.16 Gbits/sec 0 308 KBytes [ 5] 6.00-7.00 sec 1.08 GBytes 9.24 Gbits/sec 0 294 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.07 Gbits/sec 0 291 KBytes [ 5] 8.00-9.00 sec 1.06 GBytes 9.14 Gbits/sec 0 294 KBytes [ 5] 9.00-10.00 sec 1.07 GBytes 9.17 Gbits/sec 0 345 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.4 GBytes 8.95 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.4 GBytes 8.95 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 47052 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1013 MBytes 8.50 Gbits/sec 0 416 KBytes [ 5] 1.00-2.00 sec 1004 MBytes 8.42 Gbits/sec 0 424 KBytes [ 5] 2.00-3.00 sec 1.01 GBytes 8.70 Gbits/sec 0 288 KBytes [ 5] 3.00-4.00 sec 1.06 GBytes 9.12 Gbits/sec 0 286 KBytes [ 5] 4.00-5.00 sec 1.06 GBytes 9.14 Gbits/sec 0 305 KBytes [ 5] 5.00-6.00 sec 1.06 GBytes 9.11 Gbits/sec 0 308 KBytes [ 5] 6.00-7.00 sec 1.06 GBytes 9.12 Gbits/sec 0 283 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.12 Gbits/sec 0 277 KBytes [ 5] 8.00-9.00 sec 1.07 GBytes 9.22 Gbits/sec 0 303 KBytes [ 5] 9.00-10.00 sec 1.07 GBytes 9.17 Gbits/sec 0 354 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.4 GBytes 8.96 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.4 GBytes 8.96 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 47716 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 989 MBytes 8.30 Gbits/sec 0 396 KBytes [ 5] 1.00-2.00 sec 974 MBytes 8.17 Gbits/sec 0 311 KBytes [ 5] 2.00-3.00 sec 1.08 GBytes 9.24 Gbits/sec 0 305 KBytes [ 5] 3.00-4.00 sec 1.06 GBytes 9.12 Gbits/sec 0 314 KBytes [ 5] 4.00-5.00 sec 1.05 GBytes 9.04 Gbits/sec 0 288 KBytes [ 5] 5.00-6.00 sec 1.06 GBytes 9.07 Gbits/sec 0 300 KBytes [ 5] 6.00-7.00 sec 1.06 GBytes 9.09 Gbits/sec 0 288 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.08 Gbits/sec 0 379 KBytes [ 5] 8.00-9.00 sec 1.06 GBytes 9.11 Gbits/sec 0 303 KBytes [ 5] 9.00-10.00 sec 1.06 GBytes 9.09 Gbits/sec 0 551 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.4 GBytes 8.93 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.4 GBytes 8.93 Gbits/sec receiver ### POST-UPGRADE SPEED TESTS - unraid2 root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 60682 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 692 MBytes 5.81 Gbits/sec 0 328 KBytes [ 5] 1.00-2.00 sec 1.01 GBytes 8.70 Gbits/sec 0 291 KBytes [ 5] 2.00-3.00 sec 631 MBytes 5.30 Gbits/sec 0 243 KBytes [ 5] 3.00-4.00 sec 1.03 GBytes 8.86 Gbits/sec 0 294 KBytes [ 5] 4.00-5.00 sec 760 MBytes 6.38 Gbits/sec 0 283 KBytes [ 5] 5.00-6.00 sec 1.03 GBytes 8.82 Gbits/sec 0 303 KBytes [ 5] 6.00-7.00 sec 1.03 GBytes 8.86 Gbits/sec 0 294 KBytes [ 5] 7.00-8.00 sec 832 MBytes 6.98 Gbits/sec 0 283 KBytes [ 5] 8.00-9.00 sec 1.02 GBytes 8.78 Gbits/sec 0 288 KBytes [ 5] 9.00-10.00 sec 1.02 GBytes 8.76 Gbits/sec 0 286 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.99 GBytes 7.72 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.99 GBytes 7.72 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 60686 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 745 MBytes 6.25 Gbits/sec 0 339 KBytes [ 5] 1.00-2.00 sec 481 MBytes 4.04 Gbits/sec 0 320 KBytes [ 5] 2.00-3.00 sec 858 MBytes 7.19 Gbits/sec 0 274 KBytes [ 5] 3.00-4.00 sec 750 MBytes 6.29 Gbits/sec 0 283 KBytes [ 5] 4.00-5.00 sec 786 MBytes 6.60 Gbits/sec 0 283 KBytes [ 5] 5.00-6.00 sec 811 MBytes 6.81 Gbits/sec 0 277 KBytes [ 5] 6.00-7.00 sec 1.01 GBytes 8.64 Gbits/sec 0 280 KBytes [ 5] 7.00-8.00 sec 719 MBytes 6.03 Gbits/sec 0 303 KBytes [ 5] 8.00-9.00 sec 1.02 GBytes 8.77 Gbits/sec 0 297 KBytes [ 5] 9.00-10.00 sec 1.02 GBytes 8.76 Gbits/sec 0 274 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.08 GBytes 6.94 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.07 GBytes 6.93 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 60690 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.01 GBytes 8.67 Gbits/sec 0 410 KBytes [ 5] 1.00-2.00 sec 1.06 GBytes 9.12 Gbits/sec 0 387 KBytes [ 5] 2.00-3.00 sec 1.01 GBytes 8.70 Gbits/sec 0 331 KBytes [ 5] 3.00-4.00 sec 795 MBytes 6.67 Gbits/sec 0 396 KBytes [ 5] 4.00-5.00 sec 1.08 GBytes 9.28 Gbits/sec 0 419 KBytes [ 5] 5.00-6.00 sec 1.08 GBytes 9.24 Gbits/sec 0 393 KBytes [ 5] 6.00-7.00 sec 1.07 GBytes 9.15 Gbits/sec 0 407 KBytes [ 5] 7.00-8.00 sec 1.07 GBytes 9.17 Gbits/sec 0 399 KBytes [ 5] 8.00-9.00 sec 1.05 GBytes 9.02 Gbits/sec 0 382 KBytes [ 5] 9.00-10.00 sec 825 MBytes 6.92 Gbits/sec 0 274 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.0 GBytes 8.60 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.0 GBytes 8.59 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 60694 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.07 GBytes 9.16 Gbits/sec 0 376 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.37 Gbits/sec 0 390 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.37 Gbits/sec 0 385 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 393 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.38 Gbits/sec 0 399 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.38 Gbits/sec 0 390 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.38 Gbits/sec 0 396 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 402 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.36 Gbits/sec 0 387 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.37 Gbits/sec 0 385 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.36 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.35 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 60698 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 668 MBytes 5.60 Gbits/sec 0 294 KBytes [ 5] 1.00-2.00 sec 905 MBytes 7.59 Gbits/sec 0 272 KBytes [ 5] 2.00-3.00 sec 836 MBytes 7.01 Gbits/sec 0 283 KBytes [ 5] 3.00-4.00 sec 1.01 GBytes 8.71 Gbits/sec 0 277 KBytes [ 5] 4.00-5.00 sec 1018 MBytes 8.54 Gbits/sec 0 283 KBytes [ 5] 5.00-6.00 sec 1.02 GBytes 8.77 Gbits/sec 0 269 KBytes [ 5] 6.00-7.00 sec 1.05 GBytes 9.03 Gbits/sec 0 269 KBytes [ 5] 7.00-8.00 sec 1.01 GBytes 8.71 Gbits/sec 0 277 KBytes [ 5] 8.00-9.00 sec 1.01 GBytes 8.71 Gbits/sec 0 269 KBytes [ 5] 9.00-10.00 sec 1.05 GBytes 9.01 Gbits/sec 0 274 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 9.51 GBytes 8.17 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 9.51 GBytes 8.16 Gbits/sec receiver iperf Done. On 9/14/2021 at 10:40 PM, Ford Prefect said: Edit2: forget this, that setting is for a bond setup in the main switch, faning out to others. - I'd gather you did not check the default config of the CRS, when in RouterOS mode (should be the default OS booting, for a CRS). Maybe the bridge setup is not fully configured. As said, RouterOS is a bit of a learning curve. Yeah it's setup as an active-backup configuration. I've still got the CRS running in RouterOS bridge mode, as I haven't yet moved my WAN router and that does my WAN routing pretty well at this point. I can't seem to setup the active-backup bond though, as the bridge mode throws an error: Something else I'm having trouble with, I went and purchased a couple Ubiquity 1Gb RJ45 transceivers but they seem to not be recognised at all by the CRS309. I was trying to test bond by having everything on the same switch. I may have goofed. Are these not compatible: On 9/14/2021 at 10:40 PM, Ford Prefect said: You should try and disable the bond0 for a first start. Then test again. This is my next move, I haven't had much chance to bring everything down and play around until today. Will post back those results. Edited September 19, 2021 by KptnKMan Quote Link to comment
KptnKMan Posted September 19, 2021 Author Share Posted September 19, 2021 On 9/14/2021 at 10:35 PM, ich777 said: There you go (since 6.9.0-beta23 in the CA App): On the plugin page is a Guide to flash the cards, crossflash, remove the boot ROM and you also can change the mode if you are running Mellanox Infniband Cards to Ethernet if they support it, but please keep in mind that these are the open source Mellanox Firmware tools and they all start with mst... and not only with flint like the most tutorial mention, but this is basically the same. Oh something I forgot to mention above. I installed the Mellanox Firmware Tools and got the details of my cards: unraid1: unraid2: The cards should be identical as well, but it seems they have different firmwares and also versions. Thing is, I've been to the Mellanox ConnectX-3 firmware download page, and I'm a little confused which firmware to download: Would the different firmwares be responsible for the performance behaviour I've been experiencing? Quote Link to comment
Ford Prefect Posted September 19, 2021 Share Posted September 19, 2021 @KptnKMan ...sorry, i am on the road, and the post are hard to manage on my Phone. RE bonding Mode....check the bridge setting. All interface, that are to be Part of a bond will need to be removed from the bridge first. Check bridge- bridge Ports. Then under Interfaces create the bond.After that, add the newly created bonding interface to the bridge.MT is said to be very tolerant regarding modules and brands, but you never know. One cannot test all variants out there, can't they?You should see the parameters and status of the module under interfaces. If it is not shown at all, you are goofed...try an another brand.But upgrade ROS to the latest stable first, as a last resort. If it is shown there, but does not sync, disable auto- negotiation and force/enable a 1G link only.Gesendet von meinem SM-G780G mit Tapatalk 1 Quote Link to comment
ich777 Posted September 19, 2021 Share Posted September 19, 2021 2 hours ago, KptnKMan said: Thing is, I've been to the Mellanox ConnectX-3 firmware download page Oh, these are IBM cards, please read the cross flashing guide on how to flash them with the stock Mellanox firmware and also research what firmware do you need for those cards. A quick google search for the PSID number should do the trick. I also recommend removing the BIOS from the cards after flashing them to speed up boot times. 1 Quote Link to comment
KptnKMan Posted September 20, 2021 Author Share Posted September 20, 2021 Thanks for the advice, I managed to test the cards today, but I need to do some more testing of the Ubiquity 1Gb transceivers. So far though, I couldn't get them to be recognised at all in the CRS309, they don't seem to show up from what I can see. I'm going to try them in one of the CSS610s and see if they are recognised there at all. On 9/14/2021 at 10:40 PM, Ford Prefect said: You should try and disable the bond0 for a first start. Then test again. I disabled the bridge and bond0, and did another set of speed tests between servers. Looks like with bond and bridging disabled (This test was done with the array unmounted and nothing running) I can get around 9.39/9.4Gb between systems fairly reliably: Quote ### IPERF3 SPEED TEST, BONDING DISABLED, BRIDGING DISABLED - UNRAID1 root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 46030 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 356 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 0 356 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.38 Gbits/sec 0 356 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.38 Gbits/sec 0 354 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.38 Gbits/sec 0 359 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 362 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.37 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 46034 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 354 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 0 356 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.41 Gbits/sec 0 348 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec 0 351 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.39 Gbits/sec 0 356 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec 0 356 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 0 354 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.37 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 46038 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 351 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.41 Gbits/sec 0 356 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 362 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 362 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 362 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 46042 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 354 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 0 351 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.39 Gbits/sec 0 351 KBytes [ 5] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec 0 354 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.37 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.39 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 46046 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.41 Gbits/sec 0 356 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.41 Gbits/sec 0 356 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 0 356 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 359 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 354 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 365 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 0 354 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver iperf Done. ### IPERF3 SPEED TEST, BONDING DISABLED, BRIDGING DISABLED - UNRAID2 root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 47920 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.43 Gbits/sec 0 356 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0 359 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.41 Gbits/sec 0 359 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 359 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.41 Gbits/sec 0 356 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 359 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 368 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.41 Gbits/sec 0 354 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.37 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 47924 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 373 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 0 376 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.41 Gbits/sec 0 373 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 368 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 0 368 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.41 Gbits/sec 0 368 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec 0 373 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 373 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.41 Gbits/sec 0 373 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.40 Gbits/sec 0 368 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 47928 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.43 Gbits/sec 0 354 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 0 359 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.40 Gbits/sec 0 370 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 359 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.41 Gbits/sec 0 356 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.41 Gbits/sec 0 354 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 356 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.38 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 47932 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 393 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.40 Gbits/sec 0 385 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.41 Gbits/sec 0 385 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 387 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.41 Gbits/sec 0 387 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 387 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 390 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.40 Gbits/sec 0 396 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 390 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 47936 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.42 Gbits/sec 0 376 KBytes [ 5] 1.00-2.00 sec 1.09 GBytes 9.41 Gbits/sec 0 373 KBytes [ 5] 2.00-3.00 sec 1.09 GBytes 9.41 Gbits/sec 0 368 KBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.40 Gbits/sec 0 370 KBytes [ 5] 4.00-5.00 sec 1.09 GBytes 9.41 Gbits/sec 0 376 KBytes [ 5] 5.00-6.00 sec 1.09 GBytes 9.40 Gbits/sec 0 365 KBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.40 Gbits/sec 0 376 KBytes [ 5] 7.00-8.00 sec 1.09 GBytes 9.41 Gbits/sec 0 370 KBytes [ 5] 8.00-9.00 sec 1.09 GBytes 9.40 Gbits/sec 0 368 KBytes [ 5] 9.00-10.00 sec 1.09 GBytes 9.41 Gbits/sec 0 368 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver iperf Done. After the speed test, I enabled bridge, but the networking became quite unreliable and wouldn't function correctly on either server. The onboard 1Gb cards are recognised as eth0 and will not allow me to reliably use the 10Gb cards without disabling the onboard adaptors. After some frustration, it looks like I may need to On 9/19/2021 at 8:16 PM, ich777 said: Oh, these are IBM cards, please read the cross flashing guide on how to flash them with the stock Mellanox firmware and also research what firmware do you need for those cards. Yeah thanks, I upgraded the firmware of both servers cards to the latest 2.42.5000, so that I could test some more with latest firmware, as the cards had different old versions installed. Both cards now show identical details: After installing the firmware, I re-enabled the bond0+bridging and tested again. It seems that I'm getting slower speeds than before, which is interesting: Quote ### IPERF3 SPEED TEST, BONDING ENABLED, BRIDGING ENABLED, LATEST FIRMWARE - UNRAID1 root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 56120 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 729 MBytes 6.11 Gbits/sec 0 339 KBytes [ 5] 1.00-2.00 sec 877 MBytes 7.35 Gbits/sec 0 370 KBytes [ 5] 2.00-3.00 sec 885 MBytes 7.42 Gbits/sec 0 376 KBytes [ 5] 3.00-4.00 sec 901 MBytes 7.56 Gbits/sec 0 368 KBytes [ 5] 4.00-5.00 sec 871 MBytes 7.31 Gbits/sec 0 370 KBytes [ 5] 5.00-6.00 sec 929 MBytes 7.79 Gbits/sec 0 373 KBytes [ 5] 6.00-7.00 sec 925 MBytes 7.76 Gbits/sec 0 365 KBytes [ 5] 7.00-8.00 sec 905 MBytes 7.59 Gbits/sec 0 376 KBytes [ 5] 8.00-9.00 sec 844 MBytes 7.08 Gbits/sec 0 365 KBytes [ 5] 9.00-10.00 sec 876 MBytes 7.35 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.54 GBytes 7.33 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.53 GBytes 7.33 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 56124 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 858 MBytes 7.19 Gbits/sec 0 368 KBytes [ 5] 1.00-2.00 sec 874 MBytes 7.33 Gbits/sec 0 365 KBytes [ 5] 2.00-3.00 sec 795 MBytes 6.67 Gbits/sec 0 368 KBytes [ 5] 3.00-4.00 sec 649 MBytes 5.44 Gbits/sec 0 382 KBytes [ 5] 4.00-5.00 sec 860 MBytes 7.21 Gbits/sec 0 390 KBytes [ 5] 5.00-6.00 sec 869 MBytes 7.29 Gbits/sec 0 382 KBytes [ 5] 6.00-7.00 sec 806 MBytes 6.76 Gbits/sec 0 376 KBytes [ 5] 7.00-8.00 sec 814 MBytes 6.83 Gbits/sec 0 373 KBytes [ 5] 8.00-9.00 sec 771 MBytes 6.47 Gbits/sec 0 368 KBytes [ 5] 9.00-10.00 sec 781 MBytes 6.55 Gbits/sec 0 365 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 7.89 GBytes 6.77 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 7.88 GBytes 6.77 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 56128 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 908 MBytes 7.61 Gbits/sec 0 356 KBytes [ 5] 1.00-2.00 sec 875 MBytes 7.34 Gbits/sec 0 376 KBytes [ 5] 2.00-3.00 sec 905 MBytes 7.59 Gbits/sec 0 370 KBytes [ 5] 3.00-4.00 sec 886 MBytes 7.43 Gbits/sec 0 373 KBytes [ 5] 4.00-5.00 sec 895 MBytes 7.51 Gbits/sec 0 373 KBytes [ 5] 5.00-6.00 sec 884 MBytes 7.41 Gbits/sec 0 359 KBytes [ 5] 6.00-7.00 sec 865 MBytes 7.26 Gbits/sec 0 359 KBytes [ 5] 7.00-8.00 sec 882 MBytes 7.40 Gbits/sec 0 416 KBytes [ 5] 8.00-9.00 sec 864 MBytes 7.25 Gbits/sec 0 356 KBytes [ 5] 9.00-10.00 sec 908 MBytes 7.61 Gbits/sec 0 359 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.66 GBytes 7.44 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.66 GBytes 7.44 Gbits/sec receiver iperf Done. ### IPERF3 SPEED TEST, BONDING ENABLED, BRIDGING ENABLED, LATEST FIRMWARE - UNRAID1 root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 42818 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 884 MBytes 7.42 Gbits/sec 0 317 KBytes [ 5] 1.00-2.00 sec 819 MBytes 6.87 Gbits/sec 0 314 KBytes [ 5] 2.00-3.00 sec 580 MBytes 4.87 Gbits/sec 0 328 KBytes [ 5] 3.00-4.00 sec 685 MBytes 5.75 Gbits/sec 0 322 KBytes [ 5] 4.00-5.00 sec 701 MBytes 5.88 Gbits/sec 0 277 KBytes [ 5] 5.00-6.00 sec 734 MBytes 6.16 Gbits/sec 0 322 KBytes [ 5] 6.00-7.00 sec 670 MBytes 5.62 Gbits/sec 0 280 KBytes [ 5] 7.00-8.00 sec 515 MBytes 4.32 Gbits/sec 0 274 KBytes [ 5] 8.00-9.00 sec 859 MBytes 7.20 Gbits/sec 0 322 KBytes [ 5] 9.00-10.00 sec 815 MBytes 6.84 Gbits/sec 0 317 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 7.09 GBytes 6.09 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 7.09 GBytes 6.09 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 42822 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1023 MBytes 8.58 Gbits/sec 0 283 KBytes [ 5] 1.00-2.00 sec 1.00 GBytes 8.62 Gbits/sec 0 280 KBytes [ 5] 2.00-3.00 sec 1.00 GBytes 8.63 Gbits/sec 0 297 KBytes [ 5] 3.00-4.00 sec 1.01 GBytes 8.67 Gbits/sec 0 291 KBytes [ 5] 4.00-5.00 sec 1.01 GBytes 8.66 Gbits/sec 0 297 KBytes [ 5] 5.00-6.00 sec 1.00 GBytes 8.62 Gbits/sec 0 286 KBytes [ 5] 6.00-7.00 sec 1.00 GBytes 8.62 Gbits/sec 0 294 KBytes [ 5] 7.00-8.00 sec 1.00 GBytes 8.62 Gbits/sec 0 288 KBytes [ 5] 8.00-9.00 sec 1022 MBytes 8.58 Gbits/sec 0 300 KBytes [ 5] 9.00-10.00 sec 1.00 GBytes 8.63 Gbits/sec 0 288 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.0 GBytes 8.62 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 10.0 GBytes 8.62 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 42826 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 736 MBytes 6.17 Gbits/sec 0 288 KBytes [ 5] 1.00-2.00 sec 679 MBytes 5.70 Gbits/sec 0 322 KBytes [ 5] 2.00-3.00 sec 720 MBytes 6.04 Gbits/sec 0 328 KBytes [ 5] 3.00-4.00 sec 635 MBytes 5.32 Gbits/sec 0 297 KBytes [ 5] 4.00-5.00 sec 694 MBytes 5.82 Gbits/sec 0 311 KBytes [ 5] 5.00-6.00 sec 679 MBytes 5.69 Gbits/sec 0 325 KBytes [ 5] 6.00-7.00 sec 680 MBytes 5.71 Gbits/sec 0 322 KBytes [ 5] 7.00-8.00 sec 682 MBytes 5.72 Gbits/sec 0 354 KBytes [ 5] 8.00-9.00 sec 681 MBytes 5.71 Gbits/sec 0 331 KBytes [ 5] 9.00-10.00 sec 678 MBytes 5.69 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 6.70 GBytes 5.76 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 6.70 GBytes 5.75 Gbits/sec receiver iperf Done. I guess I may need to decide if I want to use this bonding setup, as it seems the speed is significantly compromised by having it enabled. I guess either I'll have to disable the active-backup configuration, and disable to onboard 1Gb, or try to fix it on the switch config side if that is the issue. The 1Gb transceivers not working is also an annoying issue that I had planned by using them would solve the port forwarding issue. Quote Link to comment
Ford Prefect Posted September 21, 2021 Share Posted September 21, 2021 20 hours ago, KptnKMan said: Thanks for the advice, I managed to test the cards today, but I need to do some more testing of the Ubiquity 1Gb transceivers. So far though, I couldn't get them to be recognised at all in the CRS309, they don't seem to show up from what I can see. I'm going to try them in one of the CSS610s and see if they are recognised there at all. ...that is unfortunate indeed. Should you replace them, go for ones that list MT or Cisco as compatible. The CRS309 also can boot into SwOS...the CSS610 has SwOS-Lite v2.14 and the CRS309 has SwOS2.13 (non lite) as latest version. You can test this as well...you'll never know what to expect. 20 hours ago, KptnKMan said: I disabled the bridge and bond0, and did another set of speed tests between servers. Looks like with bond and bridging disabled (This test was done with the array unmounted and nothing running) I can get around 9.39/9.4Gb between systems fairly reliably: OK... 20 hours ago, KptnKMan said: After the speed test, I enabled bridge, but the networking became quite unreliable Actually using a bridge is standard and recommended. Maybe this is somehow related to your AMD board and the CPU model? I notziced, that the PCIe Slots Specs change, based on that. 20 hours ago, KptnKMan said: and wouldn't function correctly on either server. The onboard 1Gb cards are recognised as eth0 and will not allow me to reliably use the 10Gb cards without disabling the onboard adaptors. After some frustration, it looks like I may need to You should re-order the NICs in a way, that the first, deplyed 10G is eth0, the second 10G port is eth1 and the onboard should go as eth2. Then enable the bridge with only eth0 as part in it. You can change the NIC ordering in the Network Settings of unraid. 20 hours ago, KptnKMan said: After installing the firmware, I re-enabled the bond0+bridging and tested again. It seems that I'm getting slower speeds than before, which is interesting: Hmpf...this is weird. 20 hours ago, KptnKMan said: I guess either I'll have to disable the active-backup configuration, and disable to onboard 1Gb, or try to fix it on the switch config side if that is the issue. The 1Gb transceivers not working is also an annoying issue that I had planned by using them would solve the port forwarding issue. See my comment for renumbering the NICs above... Quote Link to comment
Xaero Posted September 21, 2021 Share Posted September 21, 2021 (edited) IMHO instead of using a PCI-E 1x slot for a 10GbE adapter in any situation, you should select either an NVME to PCI-E x4 adapter, if you board has NVME slots, or a USB 3.0/3.1 10GbE adapter as it will result in higher usable bandwidth in almost every case. If thunderbolt is available, it is also a great choice. USB 3.0 is 4.8 GBit, or ~5x as fast as a PCI-e 3.0 x1 link. USB 3.1 is 10 GBit, and has very low protocol overhead so you can expect near 10GBit performance. Thunderbolt 3 is 40GBit and will obviously support full speeds for up to 3 10GBit ports, and near full speed for 4. Just my 2c. I only use my 1x slot for powering a port expander. Edited September 21, 2021 by Xaero Quote Link to comment
KptnKMan Posted September 22, 2021 Author Share Posted September 22, 2021 (edited) 16 hours ago, Ford Prefect said: ...that is unfortunate indeed. Should you replace them, go for ones that list MT or Cisco as compatible. The CRS309 also can boot into SwOS...the CSS610 has SwOS-Lite v2.14 and the CRS309 has SwOS2.13 (non lite) as latest version. You can test this as well...you'll never know what to expect. Thanks, I appreciate the advice. I think I might just go ahead and replace these with 2x of the 10Gb ethernet transceivers you indicated before, as I know I'll use those before too long. The Mikrotik 10Gb ethernet transceivers are available also on eurodk, do you think these would be comparable to the fs.com modules? I assume they would be preferable for Mikrotik hardware. 16 hours ago, Ford Prefect said: Actually using a bridge is standard and recommended. Maybe this is somehow related to your AMD board and the CPU model? I notziced, that the PCIe Slots Specs change, based on that. Ok, well I think I would need these for using docker (without custom config) and my VMs anyway. I'm not sure what you meant by the PCIe slots specs change? I've not had many issues using this AMD setup, I expected a lot more bugginess tbh. Currently the cards in both systems are in the second PCIe x16 slot, especially as I want to get a working baseline for some time before eventually migrating them to an x1 slot. I know now that raw they get close to the 9.5Gb I should expect. 16 hours ago, Ford Prefect said: You should re-order the NICs in a way, that the first, deplyed 10G is eth0, the second 10G port is eth1 and the onboard should go as eth2. Then enable the bridge with only eth0 as part in it. You can change the NIC ordering in the Network Settings of unraid. Do you mean in the "interface rules" section of the network settings? I already set that order, I think I posted the screenshot before. Is there another part? Is this correct: I currently HAD them set like this (below), and I set active-backup bond0 members to eth0 and eth1. They should be functionally the same, as 10Gb port2 is unplugged, but I've set them back to the first configuration (above), prioritising both the 10Gb ports, and unplugged the onboard: 16 hours ago, Ford Prefect said: Hmpf...this is weird. Yeah, I'm not sure what to do about it. Nothing else changed, all same hardware and bonding config. I'm about to finish the reconfigure of both systems to use only eth0 and reboot, so I'll test again and edit this with the results (As I'm posting from within one of the VMs now). 👍 Edit: Well I tested again with only eth0 in bond0, and everything else working as expected. The speed seems slow again, not sure why, as everything is going through only the CRS309. I'll have to do some research into why the active-backup bonding configuration slows everything down. Test results: Quote ### IPERF3 SPEED TEST, BONDING ENABLED, BRIDGING ENABLED, LATEST FIRMWARE, BOND ETH0 ONLY - UNRAID1 root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 36310 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 947 MBytes 7.94 Gbits/sec 0 269 KBytes [ 5] 1.00-2.00 sec 771 MBytes 6.47 Gbits/sec 0 283 KBytes [ 5] 2.00-3.00 sec 858 MBytes 7.19 Gbits/sec 0 221 KBytes [ 5] 3.00-4.00 sec 1.00 GBytes 8.62 Gbits/sec 0 280 KBytes [ 5] 4.00-5.00 sec 1.00 GBytes 8.62 Gbits/sec 0 277 KBytes [ 5] 5.00-6.00 sec 745 MBytes 6.25 Gbits/sec 0 283 KBytes [ 5] 6.00-7.00 sec 659 MBytes 5.53 Gbits/sec 0 218 KBytes [ 5] 7.00-8.00 sec 949 MBytes 7.96 Gbits/sec 0 269 KBytes [ 5] 8.00-9.00 sec 685 MBytes 5.75 Gbits/sec 0 283 KBytes [ 5] 9.00-10.00 sec 636 MBytes 5.34 Gbits/sec 0 277 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.11 GBytes 6.97 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.11 GBytes 6.96 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 36314 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 699 MBytes 5.86 Gbits/sec 0 339 KBytes [ 5] 1.00-2.00 sec 485 MBytes 4.07 Gbits/sec 0 277 KBytes [ 5] 2.00-3.00 sec 663 MBytes 5.57 Gbits/sec 0 291 KBytes [ 5] 3.00-4.00 sec 443 MBytes 3.72 Gbits/sec 0 291 KBytes [ 5] 4.00-5.00 sec 802 MBytes 6.73 Gbits/sec 0 368 KBytes [ 5] 5.00-6.00 sec 661 MBytes 5.55 Gbits/sec 0 320 KBytes [ 5] 6.00-7.00 sec 712 MBytes 5.98 Gbits/sec 0 308 KBytes [ 5] 7.00-8.00 sec 639 MBytes 5.36 Gbits/sec 0 317 KBytes [ 5] 8.00-9.00 sec 648 MBytes 5.43 Gbits/sec 0 317 KBytes [ 5] 9.00-10.00 sec 600 MBytes 5.03 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 6.20 GBytes 5.33 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 6.20 GBytes 5.33 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 36318 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 858 MBytes 7.20 Gbits/sec 0 362 KBytes [ 5] 1.00-2.00 sec 492 MBytes 4.13 Gbits/sec 0 300 KBytes [ 5] 2.00-3.00 sec 666 MBytes 5.59 Gbits/sec 0 283 KBytes [ 5] 3.00-4.00 sec 532 MBytes 4.47 Gbits/sec 0 320 KBytes [ 5] 4.00-5.00 sec 420 MBytes 3.52 Gbits/sec 0 280 KBytes [ 5] 5.00-6.00 sec 711 MBytes 5.97 Gbits/sec 0 294 KBytes [ 5] 6.00-7.00 sec 614 MBytes 5.15 Gbits/sec 0 308 KBytes [ 5] 7.00-8.00 sec 588 MBytes 4.93 Gbits/sec 0 311 KBytes [ 5] 8.00-9.00 sec 430 MBytes 3.61 Gbits/sec 0 269 KBytes [ 5] 9.00-10.00 sec 438 MBytes 3.67 Gbits/sec 0 266 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 5.61 GBytes 4.82 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 5.61 GBytes 4.82 Gbits/sec receiver iperf Done. ### IPERF3 SPEED TEST, BONDING ENABLED, BRIDGING ENABLED, LATEST FIRMWARE, BOND ETH0 ONLY - UNRAID2 root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 59456 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 635 MBytes 5.32 Gbits/sec 0 322 KBytes [ 5] 1.00-2.00 sec 582 MBytes 4.88 Gbits/sec 0 303 KBytes [ 5] 2.00-3.00 sec 549 MBytes 4.60 Gbits/sec 0 317 KBytes [ 5] 3.00-4.00 sec 585 MBytes 4.90 Gbits/sec 0 300 KBytes [ 5] 4.00-5.00 sec 693 MBytes 5.81 Gbits/sec 0 322 KBytes [ 5] 5.00-6.00 sec 575 MBytes 4.82 Gbits/sec 0 337 KBytes [ 5] 6.00-7.00 sec 520 MBytes 4.36 Gbits/sec 0 317 KBytes [ 5] 7.00-8.00 sec 698 MBytes 5.85 Gbits/sec 0 322 KBytes [ 5] 8.00-9.00 sec 695 MBytes 5.83 Gbits/sec 0 370 KBytes [ 5] 9.00-10.00 sec 554 MBytes 4.64 Gbits/sec 0 277 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 5.94 GBytes 5.10 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 5.94 GBytes 5.10 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 59460 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 691 MBytes 5.80 Gbits/sec 0 283 KBytes [ 5] 1.00-2.00 sec 741 MBytes 6.22 Gbits/sec 0 294 KBytes [ 5] 2.00-3.00 sec 719 MBytes 6.03 Gbits/sec 0 288 KBytes [ 5] 3.00-4.00 sec 712 MBytes 5.98 Gbits/sec 0 291 KBytes [ 5] 4.00-5.00 sec 718 MBytes 6.02 Gbits/sec 0 328 KBytes [ 5] 5.00-6.00 sec 565 MBytes 4.74 Gbits/sec 0 345 KBytes [ 5] 6.00-7.00 sec 655 MBytes 5.49 Gbits/sec 0 314 KBytes [ 5] 7.00-8.00 sec 710 MBytes 5.96 Gbits/sec 0 314 KBytes [ 5] 8.00-9.00 sec 642 MBytes 5.39 Gbits/sec 0 320 KBytes [ 5] 9.00-10.00 sec 670 MBytes 5.62 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 6.66 GBytes 5.72 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 6.66 GBytes 5.72 Gbits/sec receiver iperf Done. root@unraid2:~# iperf3 -c 192.168.178.11 Connecting to host 192.168.178.11, port 5201 [ 5] local 192.168.178.12 port 59464 connected to 192.168.178.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 651 MBytes 5.46 Gbits/sec 0 291 KBytes [ 5] 1.00-2.00 sec 639 MBytes 5.36 Gbits/sec 0 291 KBytes [ 5] 2.00-3.00 sec 604 MBytes 5.07 Gbits/sec 0 291 KBytes [ 5] 3.00-4.00 sec 625 MBytes 5.24 Gbits/sec 0 291 KBytes [ 5] 4.00-5.00 sec 640 MBytes 5.37 Gbits/sec 0 300 KBytes [ 5] 5.00-6.00 sec 647 MBytes 5.42 Gbits/sec 0 314 KBytes [ 5] 6.00-7.00 sec 641 MBytes 5.37 Gbits/sec 0 305 KBytes [ 5] 7.00-8.00 sec 658 MBytes 5.52 Gbits/sec 0 334 KBytes [ 5] 8.00-9.00 sec 642 MBytes 5.38 Gbits/sec 0 334 KBytes [ 5] 9.00-10.00 sec 621 MBytes 5.21 Gbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 6.22 GBytes 5.34 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 6.22 GBytes 5.34 Gbits/sec receiver iperf Done. Edited September 22, 2021 by KptnKMan Quote Link to comment
KptnKMan Posted September 22, 2021 Author Share Posted September 22, 2021 (edited) 12 hours ago, Xaero said: IMHO instead of using a PCI-E 1x slot for a 10GbE adapter in any situation, you should select either an NVME to PCI-E x4 adapter, if you board has NVME slots, or a USB 3.0/3.1 10GbE adapter as it will result in higher usable bandwidth in almost every case. If thunderbolt is available, it is also a great choice. A x4 slot will definitely have more lanes and bandwidth than a x1 slot yeah, but for the performance I think I'd get (And as others have posted in this thread, it seems reasonable to get around 6/7Gb from a PCIe 3.0 x1 slot. I don't have thunderbolt on either of these boards (in my signature) as well. Edit: Also, both the NVME slots in both servers are in use for fast NVME storage, not really looking to replace them, they work pretty well. That would compromise actually being able to transfer ~10Gb over the network, as spinning drives won't cut it. 12 hours ago, Xaero said: USB 3.0 is 4.8 GBit, or ~5x as fast as a PCI-e 3.0 x1 link. USB 3.1 is 10 GBit, and has very low protocol overhead so you can expect near 10GBit performance. Thunderbolt 3 is 40GBit and will obviously support full speeds for up to 3 10GBit ports, and near full speed for 4. I did look into this a while ago, but I didn't find much reliable information on usable adaptors for this. Do you have any suggestions? I remember looking at a reddit thread discussing this (among a few other places) and they seemed to conclude that using USB3.1 for 10Gb networking isn't as good as it sounds. Have you had a different experience? Edited September 22, 2021 by KptnKMan Quote Link to comment
KptnKMan Posted September 22, 2021 Author Share Posted September 22, 2021 So I disabled all boding on both systems and left bridging enabled, as I need that for docker and VMs. Turns out, I'm still getting about 5.4Gb using iperf3, and I'm not sure why. The onboard 1Gb is still unplugged, and the Mellanox 10Gb is set to eth0, used as the only bridge interface. Maybe the bond isn't the problem at all, but something with the bridging? This is strange. Quote Link to comment
KptnKMan Posted September 22, 2021 Author Share Posted September 22, 2021 Ok, so I went back to the drawing board on this one... and I may have turned up some results. I did some more research, and was reminded that when optimising 10Gb, I should adjust the MTU settings of interfaces and switches, among some other tips and things. I think I may have lost the MTU config when changing to the Mellanox cards, and also forgot to review that. So I started with that, and adjusted the MTU to 9000 on both my unraid systems. This resulted in a disastrous set of performance results: Quote root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 56946 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 472 KBytes 3.86 Mbits/sec 3 8.74 KBytes [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 472 KBytes 387 Kbits/sec 6 sender [ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec receiver iperf Done. Yikes. 🤔 But at least something changed, which in my experience prompts closer inspection. I adjusted both systems back, and saw the performance go back to what I was seeing before. Ok... So I reconfigured both systems to higher MTU and investigated the CRS309, then adjusted the MTU of interfaces sfpplus3+4 to 9000 (Actual and L2): Then tested again: Quote root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 56950 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.13 GBytes 9.74 Gbits/sec 0 332 KBytes [ 5] 1.00-2.00 sec 1.13 GBytes 9.70 Gbits/sec 0 350 KBytes [ 5] 2.00-3.00 sec 1.13 GBytes 9.70 Gbits/sec 0 350 KBytes [ 5] 3.00-4.00 sec 1.13 GBytes 9.69 Gbits/sec 0 332 KBytes [ 5] 4.00-5.00 sec 1.13 GBytes 9.67 Gbits/sec 0 350 KBytes [ 5] 5.00-6.00 sec 1.13 GBytes 9.70 Gbits/sec 0 332 KBytes [ 5] 6.00-7.00 sec 1.13 GBytes 9.68 Gbits/sec 0 350 KBytes [ 5] 7.00-8.00 sec 1.13 GBytes 9.70 Gbits/sec 0 332 KBytes [ 5] 8.00-9.00 sec 1.13 GBytes 9.69 Gbits/sec 0 332 KBytes [ 5] 9.00-10.00 sec 1.13 GBytes 9.69 Gbits/sec 0 35.0 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.3 GBytes 9.70 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.3 GBytes 9.69 Gbits/sec receiver iperf Done. root@unraid1:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 56954 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.15 GBytes 9.87 Gbits/sec 0 332 KBytes [ 5] 1.00-2.00 sec 1.14 GBytes 9.83 Gbits/sec 0 350 KBytes [ 5] 2.00-3.00 sec 1.14 GBytes 9.83 Gbits/sec 0 350 KBytes [ 5] 3.00-4.00 sec 1.14 GBytes 9.79 Gbits/sec 0 350 KBytes [ 5] 4.00-5.00 sec 1.14 GBytes 9.75 Gbits/sec 0 332 KBytes [ 5] 5.00-6.00 sec 1.14 GBytes 9.81 Gbits/sec 0 332 KBytes [ 5] 6.00-7.00 sec 1.13 GBytes 9.75 Gbits/sec 0 367 KBytes [ 5] 7.00-8.00 sec 1.14 GBytes 9.78 Gbits/sec 0 332 KBytes [ 5] 8.00-9.00 sec 1.14 GBytes 9.78 Gbits/sec 0 350 KBytes [ 5] 9.00-10.00 sec 1.14 GBytes 9.80 Gbits/sec 0 35.0 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.4 GBytes 9.80 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.4 GBytes 9.80 Gbits/sec receiver iperf Done. Now that's more like it! 😄 Now, as far as I understand in my experience with this, and what I've read is that there can be a performance hit (Due to translation) if other parts of the network are set to other MTUs. With that in mind, and that I'm testing directly between 2 systems via the CRS309 switch, I'm going to leave this for now and do more testing from other devices to my unraid systems (Like from my WAN router). I'll maybe set the MTU between all 10Gb links to the same to prevent translation performance hits as well. I should still be getting the full 1Gb from that I think, but I'll see what it turns up. I also have bonding disabled, still at this point, so I may be able to see better bonding performance with the higher MTU. Currently all interfaces are set to 9000, but only port 1 of the Mellanox cards are plugged in and up. Gotta go for now, but will report back later. For now, further tests show fairly consistently I'm getting 9.85Gb to 9.88Gb, at least between my unraids. 😄 Hopefully, someone will benefit from reading this journey as well. Quote Link to comment
Ford Prefect Posted September 22, 2021 Share Posted September 22, 2021 @KptnKMan ...I am still traveling, with only my Phone at Hand, so doing sophisticated quotes is beyond me atm.RE 10G-T transceivers. Yes, the MT one's you linked will definitely work, also can fallback to 1, 2.5 and 5G, even on cat5e cabling, but only for length up to 30m. The one I linked from fs.com is a newer model, for up to 80m but I don't know if it will fallback to slower speeds as well. While I do have the MT ones myself, I cannot vouch for that model from fs.com. but I think someone in the hardwareluxx forum mentioned, that it was running successfully in a CRS switch model. However, the MT one is a safer bet.EDIT: found it, but I don't know if user purchased the 80m transceiver in Cisco or Generic coding: see https://www.hardwareluxx.de/community/threads/10gbit-homenetzwerk.807277/post-28657307RE MTU, yes, the MTU must be the same for all ports along the logical IP connection.But using MTU and L2-MTU with the same number ist wrong. MTU is at a higher layer, so in order to transmit a packet of MTU size, L2-MTU which is used in the lower transport layer, must be higher than MTU size. There is a link for that in the MT-Wiki, somewhere. Also, my own numbers and the ones from @ich777 were with MTU 1500, standard but still above 9.4Gbps, weren't they.Gesendet von meinem SM-G780G mit Tapatalk Quote Link to comment
ich777 Posted September 22, 2021 Share Posted September 22, 2021 9 minutes ago, Ford Prefect said: Also, my own numbers and the ones from @ich777 were with MTU 1500, standard but still above 9.4Gbps, weren't they. Exactly I also have the default MTU from 1500 and get 10Gbit/s speeds. Quote Link to comment
KptnKMan Posted September 24, 2021 Author Share Posted September 24, 2021 On 9/22/2021 at 10:03 PM, Ford Prefect said: RE 10G-T transceivers. Yes, the MT one's you linked will definitely work, also can fallback to 1, 2.5 and 5G, even on cat5e cabling, but only for length up to 30m. The one I linked from fs.com is a newer model, for up to 80m but I don't know if it will fallback to slower speeds as well. While I do have the MT ones myself, I cannot vouch for that model from fs.com. but I think someone in the hardwareluxx forum mentioned, that it was running successfully in a CRS switch model. However, the MT one is a safer bet. EDIT: found it, but I don't know if user purchased the 80m transceiver in Cisco or Generic coding: see https://www.hardwareluxx.de/community/threads/10gbit-homenetzwerk.807277/post-28657307 Thanks, good to know the MT transceivers would work as intended upto 30m. With that in mind, this afternoon I decided to not be lazy and finally rerun that CAT7 cable to where my WAN router is, I mentioned this earlier, and this would affect the requirement of how long I would need a capable transceiver. So with a bit of blood and sweat, I managed to run 2x CAT7A S/FTP and 2x CAT6 U/UTP across the house. After some reworking, I got the run down to 21m. I also installed and tested 10Gb rated keystones on the CAT7A cables (I'll get around to the CAT6 cables tomorrow) and I'll be able to 1) move my WAN router to the server room, 2) buy an additional CSS610 for 10Gb uplink to that part of the house and 3) go with the MT transceivers as I guess it would be safely under the 30m limit: On 9/22/2021 at 10:13 PM, ich777 said: On 9/22/2021 at 10:03 PM, Ford Prefect said: Also, my own numbers and the ones from @ich777 were with MTU 1500, standard but still above 9.4Gbps, weren't they. Exactly I also have the default MTU from 1500 and get 10Gbit/s speeds. On 9/22/2021 at 10:03 PM, Ford Prefect said: RE MTU, yes, the MTU must be the same for all ports along the logical IP connection. But using MTU and L2-MTU with the same number ist wrong. MTU is at a higher layer, so in order to transmit a packet of MTU size, L2-MTU which is used in the lower transport layer, must be higher than MTU size. There is a link for that in the MT-Wiki, somewhere. Also, my own numbers and the ones from @ich777 were with MTU 1500, standard but still above 9.4Gbps, weren't they. Now this has led me to an interesting issue, in that I've left everything as-is with MTU set to 9000 for the last few days, and I haven't seen any strange or adverse affects. I've been doing a lot of reading about this, and various recommendations point to setting the MTU to 9000, but I must say I've not seen mention of discriminating between MTU and L2-MTU. Also, the detriment I've seen mentioned elsewhere is the performance hit of translation between different MTU specifications on the same network. Interestingly also, I've seen no noteworthy difference in backup times between servers, last night was about 3 minutes and 46 seconds. I also can't find errors or performance penalties in any of the router, switch or system logs. The only real difference I've seen is that iperf3 now consistently results in ~9.80Gb-9.85Gb throughput, with no retries. I downloaded iperf3 to my windows laptop (An old cygwin-compiled version v3.1.3 from 2016) and that returns a solid 1Gb speed (~931 to 939 Mbit/sec), as expected. I'm also not seeing any Rx/Tx errors or Rx/Tx drops on the sfpplus interfaces for my unraids. Do you have any idea where I can look, or should look to figure out if something bad is happening? I've been reading through the MTU docs on the MT-Wiki as advised, but I'm struggling to see an advice. The Wiki does state "Make sure that MPLS MTU is smaller or equal to L2MTU", but I don't see where "MPLS MTU" applies, and there is no specific mention of it in RouterOS, only "MTU" and "L2 MTU". The MT-Wiki also mentions re "L3 MTU": "If router receives IP packet of size 1500, but MTU for outgoing interface is set to 1400, router will either fragment the packet (if "Don't Fragment" bit is not set in IP header) or drop the packet and send ICMP "Need Fragmentation" error back to originator (this is essential for Path MTU Discovery to work). Sometimes it can be bad idea to change IP MTU from its default 1500 bytes on router interfaces if complete path end-to-end is not in administrators control. Although IP fragmentation and end-to-end Path MTU Discovery is intended to handle this situation, if ICMP Need Fragmentation errors are filtered somewhere along the path, Path MTU Discovery will not work. There are several features in MikroTik RouterOS that can benefit from possibility to exceed standard MTU" Although, I'm not seeing any drops and it's only these 2 sfpplus ports on this CRS309 device alone set to 9000. Would that break things? Also, I'm a little confused by your advice, do you mean I should reduce "MTU" back to 1500, but leave "L2-MTU" at 9000? That would make sense with the diagram from the MT-Wiki, indicating that "IP-MTU" can be less than "L2-MTU": With all this in mind, what should I do? Quote Link to comment
Ford Prefect Posted September 25, 2021 Share Posted September 25, 2021 With all this in mind, what should I do?The MT transceivers should work fine, especially when used in pairs and not in a mixed config.Just remember, that you should/can not use two directly side by side in the css610 or crs309 as both are not actively cooled. So use a fiber or DAC module or leave the adjacent port undeployed.RE MTU, the diagram already explains it, I think.when commonly referred ro MRU size, this is L3-MTU....which you will use, when connecting client and server via IPv4.Hence, a packet on L3, ultimately transported on L2/Ethernet layer must fit in. Therefore L2-MTU must hold address info PLUS all from L3 address-info & data and settings needs to be a larger number than L3-MTU.My references to my tests with 1500MTU being better than yours at 9000, did not imply that you should change back. Merely indicating that using the larger MTU and getting better results near ours wasn't the root cause for your improvement. You can safely keep 9000MTU on the switches and server NICs. Just make L2-MTU an appropriate higher value Gesendet von meinem SM-G780G mit Tapatalk Quote Link to comment
KptnKMan Posted September 25, 2021 Author Share Posted September 25, 2021 11 minutes ago, Ford Prefect said: The MT transceivers should work fine, especially when used in pairs and not in a mixed config. Just remember, that you should/can not use two directly side by side in the css610 or crs309 as both are not actively cooled. So use a fiber or DAC module or leave the adjacent port undeployed. Thanks, that's great to know. I'm not planning to mix the units, so that should be good. Hopefully, using the MT-branded transceivers will ensure proper compatibility and performance. I've also read elsewhere that when using SFP+ RJ45 transceivers, that they should be spaced out as well. I've been planning to do this, so ideally shouldn't be an issue, if I can just sandwich them between the existing DAC modules. 14 minutes ago, Ford Prefect said: RE MTU, the diagram already explains it, I think. when commonly referred ro MRU size, this is L3-MTU....which you will use, when connecting client and server via IPv4. Hence, a packet on L3, ultimately transported on L2/Ethernet layer must fit in. Therefore L2-MTU must hold address info PLUS all from L3 address-info & data and settings needs to be a larger number than L3-MTU. My references to my tests with 1500MTU being better than yours at 9000, did not imply that you should change back. Merely indicating that using the larger MTU and getting better results near ours wasn't the root cause for your improvement. You can safely keep 9000MTU on the switches and server NICs. Just make L2-MTU an appropriate higher value See, I'm genuinely trying to understand and I spent some hours yesterday diving into this to get a definitive answer of why, but I'm still not finding a clear understanding. You mentioned a few posts ago, that L3-MTU and L2-MTU should not be set to the same value, but I'm struggling to understand why because the default setting for both MTU and L2-MTU are both 1500. Is that not technically the same problem, and yet the majority of networks work with that default setting? From what I have been observing on networks, and reading in places like the MT-Wiki (And online), the MTU and L2-MTU can be "smaller or equal". This would make sense, from what I'm seeing, the "equal" part being interesting for me. According to the OSI model, wouldn't this mean that an L3-MTU/MTU and L2-MTU of the same value would be functional, and would encapsulate all required data and overhead, as long as I have "complete path end-to-end" control? From my understanding, my CRS309 would have that control and would translate/fragment/defragment packets from my unraids to other ports with other MTU settings? Am I right in thinking that if I setup MTU 9000 links/ports to (for example) my other switches, then I would need to make sure the MTU/MRU of those switch ports would be set on that end as well to avoid issues? From what I understand, it gets out of hand when there are other out-of-control entities (Like my WAN link to my ISP) transmitting on different MTU sizes from mine, causing packets to be dropped and return ICMP "fragmentation needed" requests. But surely, that is not a concern for me, seeing as my ISP link (To my understanding as I've never reconfigured it) would be at default MTU 1500? Thinking about this I just checked, its default MTU 1500. Again, I'm not challenging anyone's advice or knowledge here, I'm trying to understand. I'm just trying to understand, what would be (In my case) an appropriate higher value for L2-MTU? Is there an known/typical overhead I should calculate in (Like how most PPPOE connections add 8 bytes to headers, reducing payload to 1492), and how can I find information about this? I'm sure there's a rabbit hole somewhere about packet encapsulation that I have yet to misunderstand, but I'm just trying to get a grip on the broad strokes and how this applies to what I'm doing. Quote Link to comment
Ford Prefect Posted September 25, 2021 Share Posted September 25, 2021 You mentioned a few posts ago, that L3-MTU and L2-MTU should not be set to the same value, but I'm struggling to understand why.I agree, that there is some Kind of confusion.Maybe caused by the Statement that the L3-MTU, although sitting on a higher layer needs to be smaller as L2-MTU, which sits on a lower layer.What you need to understand, at least that is the way I understood it, is that a communication on L3 will not travel on L3 physically, only logically. Hence MTU on the respective layer must be the same for each communication device (in oder to avoid fragmentation).Physically the communication travels down on the respective local device stack until it reaches the lowest layer and only then the packet gets actually transported to the next hop.A data packet on L3 with 1500MTU can have 1480bytes of net data without risking fragmentation. L3 envelope is an additional 20bytes. As depicted oin the diagram you linked earlier, from MT, L2 adds another envelope. This is what interfaces on my CRS326 look like:notice, that L2 MTU is shown with higher numbers (where L3 MTU is 9000 and 1500 respectively).Gesendet von meinem SM-G780G mit Tapatalk Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.