KptnKMan

You mean beside the netboot bios? I haven't seen any delays other than that with the cards coming up, they seem to be recognised and active when Unraid boots on both servers. Yep, I'm using the dual-port SFP+ cards, and only using a single SFP+ on each card, just as you say. I'm not sure why more "ghost" ports would be registered, as I can see all 3 (Including the onboard 1Gb) listed when the system comes online. Is a ghost port something that has been observed in other uses of the Mellanox X-3 cards? Is this known behaviour with dual-port cards? The BIOS on these cards is the latest installed from the Mellanox site, installed using the Mellanox Tools for Unraid, exactly as instructed for the tools (As detailed in the GUI). Also, I have this issue on both servers where there is seemingly a skipped interface. I noticed in the network.cfg that initially there is the line for 'BONDNICS[0]'. Originally, this read as 'BONDNICS[0]="eth0 eth1 eth2 eth3"' But now it reads as 'BONDNICS[0]="eth0 eth1 eth3"' on Unraid1, and 'BONDNICS[0]="eth0 eth2 eth3"' on Unraid2. Unraid1 is where I'm having the issue where it seems to not be able to decide which interface id is being used, and lists 4 in the network-rules.cfg file. Unraid2 has not shown this particular issue. Also, this bootup error seems to have disappeared after I jumped through hoops to remove-then-re-add the br0 interface, that I mentioned above. I suspected that was related to the bridge getting messed up somehow, and that seems to have reset it. I understand that the bridge itself is also listed as a device, as basically everything is in linux. So with it being broken the startup scripts were probably either looking for it or assuming it was there? to setup the bridge and/or bond? Interestingly, this s***show began after I simply set the Interface Description. Changed nothing else. Before that I had wiped the network.cfg and network-rules.cfg files, all settings were running default, and I noted that the network.cfg file was not recreated until I set the Interface Description. As soon as I did that, everything seemed to get weird. Happened on both Unraid systems. ¯\_(ツ)_/¯

So I got the br0 network interface back to a "working" state by essentially: - in settings, disabling Docker - in settings, disabling VMManager - network settings, disable bonding and bridging - reboot - fix "interface rules" duplicates manually again - confirm that everything is in correct order - reboot - fix "interface rules" again, and confirm - network settings, enable bonding and bridging - reboot - confirm settings stuck, fix "interface rules" again - confirm br0 is present and working - enable docker - enable VMManager - verify VMs work again Really not sure why that got broken. Also on Unraid1, the physical interfaces now seem to show up as eth0, eth1 and eth3. I still need to manually modify the network-rules.cfg file after each bootup, otherwise 4 interfaces show up in the "interface rules" GUI. I just encountered the same br0 missing issue on Unraid2, and resolved it following the same process above, but the interfaces are labelled the same afterward. Would really appreciate if anyone has any idea what might be happening?

I started a new thread to investigate some issues I encountered today regarding more networking trouble. I can troubleshoot networking issues well enough, but Unraid configuration I'm not an expert at. Something strange is going on, and I'm really not sure why. I'd really just like to get the bond networking working stable and well so I can not have to mess with it.

Hi all, I've been getting very strange behaviour on both my Unraid systems since upgrading my NICs to 10Gb Mellanox ConnectX-3 cards. I've chronicled my journey to using 10Gb in a separate thread over here. Started this thread to try and figure out if there is a way to resolve the issues I'm encountering. My intention is to have the bonded network (active backup mode 1) working with the first port of the Mellanox 10Gb card as primary, and the onboard 1Gb as secondary failover. My understanding of how this should work is that the "primary" interface should be prioritised and fail-over to the secondary, then further interfaces (if configured) if primary interface is interrupted. I understand the bond should also fail-over back to the primary interface, should it reconnect. That doesn't seem to be happening, or I'm misunderstood. So far I've not been able to make that work as expected, and I'm struggling to determine why. When either system boots up, the onboard 1Gb is set as primary bonding interface until I manually unplug the 1Gb and it fails-over to the 10Gb. It does not seem to fail-over back to the 1Gb, unless I unplug the 10Gb. I'm quite sure its not the cards themselves that are the issue, as they seem to operate within expectations, but some software configuration or settings in Unraid. At this point I'm running everything pretty standard, and still experiencing issues. Is there some way to manually set the bonding I'm also seeing some other strange behaviours that I can't determine the reason for, and I continue to troubleshoot: On both servers, I have (and are using) the 10Gb Mellanox card, and the 1Gb onboard for the respective motherboard. On both servers, I have tried wiping the network.cfg and network-rules.cfg but the same issues return. On both servers, bonding (active-backup mode 1) and bridging is enabled by default. If anyone has any advice or can help troubleshoot I'd be thankful. A few issues are present, being: - network interface assignments changed. On Unraid1 I have eth0, eth1, eth2 and also eth3(?) yet I only should have 3 physical interfaces. On Unraid2 I'm getting eth0, eth2, and eth3. I'm not sure where eth1 has been skipped. - duplicate entries in the network-rules.cfg file. On Unraid1, I'm seeing 4 entries in config for the rules (eth0 to eth3) but still show 3 physical interfaces. This causes an apparently known issue where I am unable to edit the "interface rules" in the network setting GUI. I have to go back to the network-rules.cfg file, delete one of the entries and back to the GUI. This reoccurs after each reboot. On Unraid2, this occured once but has not seemed to be an issue since, the system has apparently settled on eth0, eth2 and eth3. - br0 has disappeared. On Unraid1, as of today, the br0 interface apparently seems to have disappeared. I'm not sure why this is, I haven't modified anything in networking as yet, apart from modifying the "interface rules" order and setting an interface description. I can access the webgui but I now get an error when trying to start any VMs: - boot up log errors (I think) related to interface assignments. On Unraid1, I have started seeing errors in the bootup log stating: 'Not enough information "dev argument is required. Error: Argument "down" is wrong: Device does not exist' There doesn't seem to show an IP on the main bootup screen either, not sure if this is related:

Yeah, this is what I'm wondering about as well. I'm going to have to do some tests and see if there is a timeout I'm missing. I remember seeing in the CRS309 that there were a few timeout settings, so I assumed there would be something similar for unraid. That's something I could do. I imagine the passive side would just be regular links? I guess that would that mean disabling the bond0, then configuring the separate switch interfaces as a bond interface as before? Would that mess with the DHCP or do I need to spoof both nics as the same MAC? 🤔 I need to read the mikrotik wiki to find out their recommendations, or try their forum as you advised. 🤔 I need to go find some answers, I appreciate you taking the time to run through this with me.

Ah gotcha. Thanks, that makes total sense. Absolutely, this seems to match what I read and understand. I'm not sure how to set the priority in unraid apart from the NIC order in the "interface rules". Is there a config I can set where I can assign some kind of weight to the ethX interfaces? I made sure to rearrange the interfaces to prioritise both the 10Gb links (1 being disconnected of course, but ok) and then the 1Gb link. This way I figured it would just go down the totem until it reaches the 1Gb link, and refer back to the 10Gb when it returns. Doesn't seem to follow that behaviour, unless there is a timeout I'm not aware of? I figured that 30 seconds was the norm, even though it failed-over immediately, didn't seem to skip a beat. Hmm, I assumed this was the order in the network-rules.cfg file or the interface order in the "bonding members of bond0". Interestingly, there is no network.cfg file since I deleted it to reset the config. I assumed this is because I have not modified anything, and the active-backup bond0 was configured by default. If I set an interface description or something, I expect it to return. I initially set the "interface rules" in the order I want, but I should look further into configuring linux UDEV rules, if there is something else I can set for weight or timeout. I think there's something I'm missing, but I can't see anything else obvious.

Ok, so I pulled the 1Gb cable on unraid2 (the unraid end) and it also failed over to the 10Gb. I can also see it on the CRS309: iperf3 confirms the (inconsistent as before with MTU@1500) 10Gb link: root@blaster:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 32828 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 822 MBytes 6.89 Gbits/sec 0 291 KBytes [ 5] 1.00-2.00 sec 894 MBytes 7.50 Gbits/sec 0 297 KBytes [ 5] 2.00-3.00 sec 876 MBytes 7.35 Gbits/sec 0 300 KBytes [ 5] 3.00-4.00 sec 930 MBytes 7.80 Gbits/sec 0 294 KBytes [ 5] 4.00-5.00 sec 721 MBytes 6.05 Gbits/sec 0 300 KBytes [ 5] 5.00-6.00 sec 964 MBytes 8.08 Gbits/sec 0 288 KBytes [ 5] 6.00-7.00 sec 876 MBytes 7.35 Gbits/sec 0 303 KBytes [ 5] 7.00-8.00 sec 916 MBytes 7.69 Gbits/sec 0 291 KBytes [ 5] 8.00-9.00 sec 775 MBytes 6.50 Gbits/sec 0 280 KBytes [ 5] 9.00-10.00 sec 880 MBytes 7.38 Gbits/sec 0 283 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 8.45 GBytes 7.26 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 8.45 GBytes 7.26 Gbits/sec receiver iperf Done. Honestly, I'm less concerned about the speed and MTU settings at this point, it seems to work so that's good enough. I'd rather like to figure out how to set a priority/preference on the eth0 10Gb link.

Pulled the 1Gb cable (the unraid end) on unraid1, and looks like it failed over to the 10Gb. After plugging it back in, it stayed on the 10Gb as primary: I need to figure out how to configure unraid to prioritise the 10Gb always, unless its disconnected. Any idea on that?

iperf3 confirms the 1Gb is primary: root@blaster:~# iperf3 -c 192.168.178.12 Connecting to host 192.168.178.12, port 5201 [ 5] local 192.168.178.11 port 52652 connected to 192.168.178.12 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 106 MBytes 889 Mbits/sec 0 235 KBytes [ 5] 1.00-2.00 sec 105 MBytes 883 Mbits/sec 0 235 KBytes [ 5] 2.00-3.00 sec 105 MBytes 880 Mbits/sec 0 235 KBytes [ 5] 3.00-4.00 sec 107 MBytes 898 Mbits/sec 0 238 KBytes [ 5] 4.00-5.00 sec 106 MBytes 893 Mbits/sec 0 238 KBytes [ 5] 5.00-6.00 sec 105 MBytes 877 Mbits/sec 0 235 KBytes [ 5] 6.00-7.00 sec 107 MBytes 894 Mbits/sec 0 235 KBytes [ 5] 7.00-8.00 sec 105 MBytes 882 Mbits/sec 0 235 KBytes [ 5] 8.00-9.00 sec 106 MBytes 891 Mbits/sec 0 235 KBytes [ 5] 9.00-10.00 sec 106 MBytes 891 Mbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.03 GBytes 888 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.03 GBytes 886 Mbits/sec receiver iperf Done.

That makes sense, and I should probably do some more reading on the subject. I will say that the active-backup bond worked just fine without the CRS309 being setup with a special bonding config... So maybe that could be it. I've just nuked the CRS309, and both unraids, all came up without issues first time, with the active-backup bond setup by default (Only with the onboard primary). I've just switched them around and rebooted. Yeah, no worries. I appreciate all you've done to help me out. I'm not really sure what to make of this, but I was also on the long-term until yesterday. Well, I'm going ahead right now with some testing. Everything seems to be behaving, reboots come up without fuss. Also took this opportunity to move the unraid SFP+ DACs along, to give that S+RJ10 a little space. In the new config, it looks like the 1Gb is working as primary, so I will pull some cables later and see if it fails over to the 10Gb. From my observations, the standby always has no Rx transmissions, and the Tx is only ever a few kbps. The CRS309 reset config (only set interface names): unraid with default config (only reordered the nics):

Yeah, I did some reading and came to the same conclusion before I posted that the 10Gb ones NEED to be separated, but not so much for the 1Gb ones. I'll keep an eye on them regardless, but the 1Gb transceivers are barely warm to the touch. Well, I'm quite a daring guy so that sounds harmless. When I have time I'll do some research and see if anyone has done a teardown of the CRS309, and if not then I'll take a look. Its not reached 90c but its been in the mid 80s at load, so I'll investigate the cooling possibilities just the same. Yeah super weird. Pulled it at the CRS309 side, should that make a difference? This whole goose chase has kinda confused me, as it doesn't seem to have a good reason for it. I might just wipe the network config by deleting the network.cfg and network-rules.cfg files and start again, not sure what else I can do on the unraid side to be honest. Yeah sorry this is my bad, I miscommunicated here. I didn't add an extra bridge, I know that would be a bad idea. There's a default configured bridge already present, and that's what I used. I meant to say: "I started by removing the interfaces from the default bridge (As that's what the error I experienced earlier about this referred to), and configuring a new bonding interface (aka bonding-unraid2), as you advised in your earlier post. Then I added that bonding-unraid2 interface to the bridge." Reading your instructions, that's exactly what I did so that's good to know at least. From everything I've read and videos I've seen, setting up an active-backup bond should be simple and easy. Story of my life. 🙄 In the unraid interface it even states: "Mode 1 (active-backup) is the recommended setting. Other modes allow you to set up a specific environment, but may require proper switch support. Choosing a unsupported mode can result in a disrupted communication." I can set the link L2-MTU and L3-MTU but on the bond interface, I'm not sure why, I cant set the L3-MTU (Labelled 'MTU'). You can see in the 2nd screenshot I posted earlier, the L2-MTU is greyed out, and inaccessible: Yeah I thought that the active-backup bond would be simple enough, especially as its "default" bond in unraid, and would be nice to fail back to the onboard controller... But its not proven so simple. Maybe I should have just gotten a 2nd DAC for each server and tried bond across the dual 10Gb instead. Then I could have just used more simple 802.3ad link-aggregation, balance-rr or another method that requires same-speed links, like I did before when I had 1Gb cards. I guess you live and learn, still looking to nuke the network config on both systems and start again, maybe should do on the CRS309 as well, in case I (more than likely) messed something up. Yeah, thanks I appreciate that. Honestly I'm hesitant to start from the beginning seeking help in yet another forum just yet, so I'm gonna go with the scorched earth approach first, just nuke all the configs, start again and see if that might fix things. I think I might just go step by step and see where it all falls apart.

Ah, yeah, a couple things there. I bought 2x S+RJ10 10Gb tranceivers and 2x S-RJ01 1Gb (I mistakenly stated them as S-RJ10), which are rated for much lower temperature. Currently they are next to each other, as in the diagram, but as of now they are very cool and not even nearly as lava hot as the single S+RJ10 hooked up to the media room CSS610. I'm planning to relocate the S-RJ01s soon, just haven't yet as they seem good right now. Yeah, I combed over the specs of the S+RJ10 and S-RJ01 to make sure I understood their tolerances as well. That 10Gb transceiver is very hot, burns to the touch. I might have to put a small fan near it or something. Not yet, I temporarily relocated cards I don't need for a while out and they are both running in the 2nd full PCIEx16 slot of each system. I'd like to get a good idea of full performance before I eventually move them, if at all. At some point I will have to, I think. Proper VLANs are something I have my network pre-segmented for already, so that I can do that. My IOT stuff all uses separate WLANs from my domestic household stuff, and the Home Automation cabinet is separated for this exact reason as well. I played with VLANs at home a while ago, but now I have all widespread VLAN-capable hardware so the time is getting close... very close. I try to be take things "one at a time" so I know things are stable before moving onto the next radical change. I guess thats the Ops Engineer in me. Haha, I should probably be spending less money not more but this 10Gbit setup has been a pipe dream of mine for more than 8 years. I'm finally doing it, and very happy its working as well as it has. Saying that... something weird is happening... Yeah, the bonds were working well... or so I thought... but I've highlighted something very frustratingly confusing by trying to properly configure the active-backup bond for both systems. First of all, I did some testing by unplugging/disabling the 10Gb link for unraid2, which seemed to fail the link over as expected, but then it refused to fail back. It refused like at all, causing the server to become unavailable, until I eventually rebooted the system and it reverted to the 10Gb as primary (Without touching/changing anything else). This doesn't seem correct, as I shouldn't have to reboot the server to fix a failover. I could see the link was up, and registering on the CRS309 but I could only see a couple Kb of Tx and nothing Rx. So I decided that I should properly configure the active-backup on the CRS309, especially as I can now do it on a single box. I did some reading, and watched some YouTube vids about it, then dived in. Simple, I thought... I started by removing the interfaces from the default bridge (As that's what the error I experienced earlier about this referred to), and configuring a new bridge, as you advised in your earlier post. Then I added that "bonding-unraid2" interface to the bridge. Now there's a few things I have run into that are frustrating me, I've been round and round it for hours now and I'm not sure what to do. I've setup both unraid systems identically the same now and have had varying results. 1) If I so much as touch the bonding configuration of either bond on the CRS309, they stop working, and the entire server goes dark. I can't reach it, I have to gracefully shut it down or reboot it at the box and upon reboot, it comes up available again. Even so much as looking at the interface and clicking Ok or Apply (Without changing anything) causes it to freak out and give up. 2) I cannot set the MTU of the bonding-unraidX interface, as much as I've tried. It refuses, and throws an error that the "MTU could be set", then the server goes dark again until reboot. So at this point I've set everything back to 1500 on both unraid systems, just to try and have consistency to test and get working. 3) My second system unraid2, like refuses to play ball 90% of the time, and boots up with a link-local address of 169.254.129.35, indicating that it cannot find the network. I double/triple/quadruple checked the unraid network config and rebooted more than a dozen times to figure out what's going on. I found that a cold boot works more often and picks up an Ip of 192.168.178.12, but a reboot usually stalls for a bit right before login, and then uses 169.254.129.35. I use DHCP on my network, from my WAN router, never had any DHCP issues. I think something on the CRS309 is very wrong, and I'm missing something. I've tried using "arp" link monitoring with an IP, as well as Mii, both seem to have the same behaviour. At this point I've shut down unraid2, as the alternative would be to delete the bonding interface and go back to what was "working". 4) My primary system unraid1, doesn't seem to be too bothered about any of this, and seems to be working with the bonding-unraid1 configuration on the CRS309. I haven't tested the bond failover by pulling the cable yet, as I've been busy trying to figure out why unraid2 is not picking up an IP with an identical config. 5) I updated the CRS309 to latest RouterOS stable 6.49, and nothing seems to have helped. Same issues. I saw some mentions of link stability in the changelogs and thought it might be worth a shot. At this point I'm not sure what to do. Some config screenshots (both systems networking configured identically) if that helps:

So I'm back with an update. TL;DR: 1) I haven't returned the Ubiquity transceivers yet. 2) I finally made a quick and dirty network diagram. 3) Finally moved my main WAN router over to the server room, centralising everything except the ISP router. 4) Ordered, received, and setup the new 3rd CSS610 for the media setup. 5) Connected the new CSS610 across the house with new Mikrotik S+RJ10 transceivers and CAT7A+ S/FTP. 6) Connected the bond0 active-backup secondary 1Gb to the CRS309 using a couple new Mikrotik S-RJ10 transceivers. Detailed: 1) I've been doing some testing on the Ubiquity transceivers I purchased before, I'm planning to return them but they seem to register now but still no link. Gonna keep playing with them a bit more. 2) I've made better, but I finally got around to making a very quick diagram: 3) Yep, I finally got the CAT7 cables run over to the server room and moved the WAN router. All working well and centralised in the same location. The ISP router runs in passthrough, and is direct connected to coaxial, so I can't move that unfortunately. Pic of the core setup: 4+5) So with the CAT7A+ run complete with keystones, and some short CAT7 cables acquired for each end, I hooked everything up. Those S+RJ10 transceivers do get toasty though, as advised. Measured temps of 83c, highest I've seen so far under load, which seems kinda high but I think its understood that these get hot? Got the 3rd CSS610 hooked up and it is behaving normally, link up at 10Gbit, no issues encountered. 6) Purchased a couple S-RJ10 transceivers so that I could hook those directly into the CRS309, and that seems to work well. WIthout needing to bridge over to the CSS610 it seems more stable also, I haven't seen any dropouts. Also looks like I've almost filled the CRS309 SFP+ ports as well, which I didn't expect when starting this. So all in all, I'm pretty pleased with how everything is turned out. Gonna keep tweaking as I go, and I heard on the Uncast YouTube video on 10Gb that there is apparently some tips for running 10Gb Unraid somewhere. Gonna try and find that.

Yeah, I don't think your statements were confusing for me in that way exactly, because I think I get the L2-L3 relationship (I'm not new to networking, but with this particular use case I've been trying to be "correct".) but I guess there is no correct answer. I was mainly looking for a recommendation of a way forward to make sense of. You indicated what I shouldn't do, so I was really trying to understand what a recommendation of what I should do would be. Anyhow, thinking of a 20 byte IP header + payload combination makes enough sense that I can go on with that. I also found some more examples like this article that support that understanding is a good way to go forward. As much as this makes sense, the MTUs in the image raise new questions for me, like how are the odd numbered MTUs related to 1500 and 9000 respectively. But as it is, I'm not going to worry too much about that right now. As it is now, I've set the MTU to 9000, and L2-MTU to 9800 (Max L2 apparently being 10218), and the sky doesn't seem to have fallen yet. 😅 Everybody seems to be behaving, and the network appears to be performing as desired. I've also re-enabled the active-backup/10Gb-1Gb bond0, and that seems to be working without issues. 👍 I'll be monitoring the bond setup and seeing what happens. Next I guess will be to return the 1Gb transceivers, get 10Gb RJ45 transceivers, and get an additional CSS610 for across the house. 🤔

Thanks, that's great to know. I'm not planning to mix the units, so that should be good. Hopefully, using the MT-branded transceivers will ensure proper compatibility and performance. I've also read elsewhere that when using SFP+ RJ45 transceivers, that they should be spaced out as well. I've been planning to do this, so ideally shouldn't be an issue, if I can just sandwich them between the existing DAC modules. See, I'm genuinely trying to understand and I spent some hours yesterday diving into this to get a definitive answer of why, but I'm still not finding a clear understanding. You mentioned a few posts ago, that L3-MTU and L2-MTU should not be set to the same value, but I'm struggling to understand why because the default setting for both MTU and L2-MTU are both 1500. Is that not technically the same problem, and yet the majority of networks work with that default setting? From what I have been observing on networks, and reading in places like the MT-Wiki (And online), the MTU and L2-MTU can be "smaller or equal". This would make sense, from what I'm seeing, the "equal" part being interesting for me. According to the OSI model, wouldn't this mean that an L3-MTU/MTU and L2-MTU of the same value would be functional, and would encapsulate all required data and overhead, as long as I have "complete path end-to-end" control? From my understanding, my CRS309 would have that control and would translate/fragment/defragment packets from my unraids to other ports with other MTU settings? Am I right in thinking that if I setup MTU 9000 links/ports to (for example) my other switches, then I would need to make sure the MTU/MRU of those switch ports would be set on that end as well to avoid issues? From what I understand, it gets out of hand when there are other out-of-control entities (Like my WAN link to my ISP) transmitting on different MTU sizes from mine, causing packets to be dropped and return ICMP "fragmentation needed" requests. But surely, that is not a concern for me, seeing as my ISP link (To my understanding as I've never reconfigured it) would be at default MTU 1500? Thinking about this I just checked, its default MTU 1500. Again, I'm not challenging anyone's advice or knowledge here, I'm trying to understand. I'm just trying to understand, what would be (In my case) an appropriate higher value for L2-MTU? Is there an known/typical overhead I should calculate in (Like how most PPPOE connections add 8 bytes to headers, reducing payload to 1492), and how can I find information about this? I'm sure there's a rabbit hole somewhere about packet encapsulation that I have yet to misunderstand, but I'm just trying to get a grip on the broad strokes and how this applies to what I'm doing.

Thanks, good to know the MT transceivers would work as intended upto 30m. With that in mind, this afternoon I decided to not be lazy and finally rerun that CAT7 cable to where my WAN router is, I mentioned this earlier, and this would affect the requirement of how long I would need a capable transceiver. So with a bit of blood and sweat, I managed to run 2x CAT7A S/FTP and 2x CAT6 U/UTP across the house. After some reworking, I got the run down to 21m. I also installed and tested 10Gb rated keystones on the CAT7A cables (I'll get around to the CAT6 cables tomorrow) and I'll be able to 1) move my WAN router to the server room, 2) buy an additional CSS610 for 10Gb uplink to that part of the house and 3) go with the MT transceivers as I guess it would be safely under the 30m limit: Exactly I also have the default MTU from 1500 and get 10Gbit/s speeds. Now this has led me to an interesting issue, in that I've left everything as-is with MTU set to 9000 for the last few days, and I haven't seen any strange or adverse affects. I've been doing a lot of reading about this, and various recommendations point to setting the MTU to 9000, but I must say I've not seen mention of discriminating between MTU and L2-MTU. Also, the detriment I've seen mentioned elsewhere is the performance hit of translation between different MTU specifications on the same network. Interestingly also, I've seen no noteworthy difference in backup times between servers, last night was about 3 minutes and 46 seconds. I also can't find errors or performance penalties in any of the router, switch or system logs. The only real difference I've seen is that iperf3 now consistently results in ~9.80Gb-9.85Gb throughput, with no retries. I downloaded iperf3 to my windows laptop (An old cygwin-compiled version v3.1.3 from 2016) and that returns a solid 1Gb speed (~931 to 939 Mbit/sec), as expected. I'm also not seeing any Rx/Tx errors or Rx/Tx drops on the sfpplus interfaces for my unraids. Do you have any idea where I can look, or should look to figure out if something bad is happening? I've been reading through the MTU docs on the MT-Wiki as advised, but I'm struggling to see an advice. The Wiki does state "Make sure that MPLS MTU is smaller or equal to L2MTU", but I don't see where "MPLS MTU" applies, and there is no specific mention of it in RouterOS, only "MTU" and "L2 MTU". The MT-Wiki also mentions re "L3 MTU": "If router receives IP packet of size 1500, but MTU for outgoing interface is set to 1400, router will either fragment the packet (if "Don't Fragment" bit is not set in IP header) or drop the packet and send ICMP "Need Fragmentation" error back to originator (this is essential for Path MTU Discovery to work). Sometimes it can be bad idea to change IP MTU from its default 1500 bytes on router interfaces if complete path end-to-end is not in administrators control. Although IP fragmentation and end-to-end Path MTU Discovery is intended to handle this situation, if ICMP Need Fragmentation errors are filtered somewhere along the path, Path MTU Discovery will not work. There are several features in MikroTik RouterOS that can benefit from possibility to exceed standard MTU" Although, I'm not seeing any drops and it's only these 2 sfpplus ports on this CRS309 device alone set to 9000. Would that break things? Also, I'm a little confused by your advice, do you mean I should reduce "MTU" back to 1500, but leave "L2-MTU" at 9000? That would make sense with the diagram from the MT-Wiki, indicating that "IP-MTU" can be less than "L2-MTU": With all this in mind, what should I do?

Ok, so I went back to the drawing board on this one... and I may have turned up some results. I did some more research, and was reminded that when optimising 10Gb, I should adjust the MTU settings of interfaces and switches, among some other tips and things. I think I may have lost the MTU config when changing to the Mellanox cards, and also forgot to review that. So I started with that, and adjusted the MTU to 9000 on both my unraid systems. This resulted in a disastrous set of performance results: Yikes. 🤔 But at least something changed, which in my experience prompts closer inspection. I adjusted both systems back, and saw the performance go back to what I was seeing before. Ok... So I reconfigured both systems to higher MTU and investigated the CRS309, then adjusted the MTU of interfaces sfpplus3+4 to 9000 (Actual and L2): Then tested again: Now that's more like it! 😄 Now, as far as I understand in my experience with this, and what I've read is that there can be a performance hit (Due to translation) if other parts of the network are set to other MTUs. With that in mind, and that I'm testing directly between 2 systems via the CRS309 switch, I'm going to leave this for now and do more testing from other devices to my unraid systems (Like from my WAN router). I'll maybe set the MTU between all 10Gb links to the same to prevent translation performance hits as well. I should still be getting the full 1Gb from that I think, but I'll see what it turns up. I also have bonding disabled, still at this point, so I may be able to see better bonding performance with the higher MTU. Currently all interfaces are set to 9000, but only port 1 of the Mellanox cards are plugged in and up. Gotta go for now, but will report back later. For now, further tests show fairly consistently I'm getting 9.85Gb to 9.88Gb, at least between my unraids. 😄 Hopefully, someone will benefit from reading this journey as well.

So I disabled all boding on both systems and left bridging enabled, as I need that for docker and VMs. Turns out, I'm still getting about 5.4Gb using iperf3, and I'm not sure why. The onboard 1Gb is still unplugged, and the Mellanox 10Gb is set to eth0, used as the only bridge interface. Maybe the bond isn't the problem at all, but something with the bridging? This is strange.

A x4 slot will definitely have more lanes and bandwidth than a x1 slot yeah, but for the performance I think I'd get (And as others have posted in this thread, it seems reasonable to get around 6/7Gb from a PCIe 3.0 x1 slot. I don't have thunderbolt on either of these boards (in my signature) as well. Edit: Also, both the NVME slots in both servers are in use for fast NVME storage, not really looking to replace them, they work pretty well. That would compromise actually being able to transfer ~10Gb over the network, as spinning drives won't cut it. I did look into this a while ago, but I didn't find much reliable information on usable adaptors for this. Do you have any suggestions? I remember looking at a reddit thread discussing this (among a few other places) and they seemed to conclude that using USB3.1 for 10Gb networking isn't as good as it sounds. Have you had a different experience?

Thanks, I appreciate the advice. I think I might just go ahead and replace these with 2x of the 10Gb ethernet transceivers you indicated before, as I know I'll use those before too long. The Mikrotik 10Gb ethernet transceivers are available also on eurodk, do you think these would be comparable to the fs.com modules? I assume they would be preferable for Mikrotik hardware. Ok, well I think I would need these for using docker (without custom config) and my VMs anyway. I'm not sure what you meant by the PCIe slots specs change? I've not had many issues using this AMD setup, I expected a lot more bugginess tbh. Currently the cards in both systems are in the second PCIe x16 slot, especially as I want to get a working baseline for some time before eventually migrating them to an x1 slot. I know now that raw they get close to the 9.5Gb I should expect. Do you mean in the "interface rules" section of the network settings? I already set that order, I think I posted the screenshot before. Is there another part? Is this correct: I currently HAD them set like this (below), and I set active-backup bond0 members to eth0 and eth1. They should be functionally the same, as 10Gb port2 is unplugged, but I've set them back to the first configuration (above), prioritising both the 10Gb ports, and unplugged the onboard: Yeah, I'm not sure what to do about it. Nothing else changed, all same hardware and bonding config. I'm about to finish the reconfigure of both systems to use only eth0 and reboot, so I'll test again and edit this with the results (As I'm posting from within one of the VMs now). 👍 Edit: Well I tested again with only eth0 in bond0, and everything else working as expected. The speed seems slow again, not sure why, as everything is going through only the CRS309. I'll have to do some research into why the active-backup bonding configuration slows everything down. Test results:

Thanks for the advice, I managed to test the cards today, but I need to do some more testing of the Ubiquity 1Gb transceivers. So far though, I couldn't get them to be recognised at all in the CRS309, they don't seem to show up from what I can see. I'm going to try them in one of the CSS610s and see if they are recognised there at all. I disabled the bridge and bond0, and did another set of speed tests between servers. Looks like with bond and bridging disabled (This test was done with the array unmounted and nothing running) I can get around 9.39/9.4Gb between systems fairly reliably: After the speed test, I enabled bridge, but the networking became quite unreliable and wouldn't function correctly on either server. The onboard 1Gb cards are recognised as eth0 and will not allow me to reliably use the 10Gb cards without disabling the onboard adaptors. After some frustration, it looks like I may need to Yeah thanks, I upgraded the firmware of both servers cards to the latest 2.42.5000, so that I could test some more with latest firmware, as the cards had different old versions installed. Both cards now show identical details: After installing the firmware, I re-enabled the bond0+bridging and tested again. It seems that I'm getting slower speeds than before, which is interesting: I guess I may need to decide if I want to use this bonding setup, as it seems the speed is significantly compromised by having it enabled. I guess either I'll have to disable the active-backup configuration, and disable to onboard 1Gb, or try to fix it on the switch config side if that is the issue. The 1Gb transceivers not working is also an annoying issue that I had planned by using them would solve the port forwarding issue.

Oh something I forgot to mention above. I installed the Mellanox Firmware Tools and got the details of my cards: unraid1: unraid2: The cards should be identical as well, but it seems they have different firmwares and also versions. Thing is, I've been to the Mellanox ConnectX-3 firmware download page, and I'm a little confused which firmware to download: Would the different firmwares be responsible for the performance behaviour I've been experiencing?

I do have a few questions, maybe there's some things I'm missing. So I monitored this over the last few days, and the speed has been fairly consistent, but still not at the 9.5 that I figure it should be capable of. There are backups that runs every night between servers (for VMS, docker appdata, personal files, etc) and I have a particular VM that has a primary vdisk of 200GB. That particular backup used to consistently take about 31 minutes, 19 seconds. Now the same backup runs at about 3 minutes, 44 seconds. I've not been seeing any errors of packet loss in any of the logs as well, but I've rerun a few tests over the last few days between servers. I'm pretty happy with it, but it definitely should be doing a touch better given the specs in both systems, so there should be no shortage of bandwidth or bottlenecks that I'm aware of: Yeah it's setup as an active-backup configuration. I've still got the CRS running in RouterOS bridge mode, as I haven't yet moved my WAN router and that does my WAN routing pretty well at this point. I can't seem to setup the active-backup bond though, as the bridge mode throws an error: Something else I'm having trouble with, I went and purchased a couple Ubiquity 1Gb RJ45 transceivers but they seem to not be recognised at all by the CRS309. I was trying to test bond by having everything on the same switch. I may have goofed. Are these not compatible: This is my next move, I haven't had much chance to bring everything down and play around until today. Will post back those results.

Thanks, I'm definitely going to be updating and investigating more into these in the coming days. In recent weeks I've already been reading and watching setup guides and configuration instructions. I haven't used any Mikrotik switches before, so it's going to be lots to learn. 😁 This is via the CRS309, using the DAC cables I got included with the Mellanox cards. Bearing in mind I haven't adjusted anything, its all just plugged in and pushing data right now. The test show they don't seem to be running at 100% full speed, but I think I'll watch overnight and see how the throughput goes when backups run between servers. The current core setup, in its current position, due to be moved at some point soon: The unraid servers are hooked into SFP+ ports 3 and 4. The other CSS610 on the lab/desk will be setup and uplinked in the next days.

Alright... so it looks like I got it working. Reconfigured the DHCP and the active-backup bond0 seems to be up without trouble. some new iperf3 results between servers: Not quite full 10Gb, especially as they are in physical x16 ports... but pretty good. 😁