KptnKMan

Members
  • Posts

    267
  • Joined

  • Last visited

Everything posted by KptnKMan

  1. Thanks, that's great to know. I'm not planning to mix the units, so that should be good. Hopefully, using the MT-branded transceivers will ensure proper compatibility and performance. I've also read elsewhere that when using SFP+ RJ45 transceivers, that they should be spaced out as well. I've been planning to do this, so ideally shouldn't be an issue, if I can just sandwich them between the existing DAC modules. See, I'm genuinely trying to understand and I spent some hours yesterday diving into this to get a definitive answer of why, but I'm still not finding a clear understanding. You mentioned a few posts ago, that L3-MTU and L2-MTU should not be set to the same value, but I'm struggling to understand why because the default setting for both MTU and L2-MTU are both 1500. Is that not technically the same problem, and yet the majority of networks work with that default setting? From what I have been observing on networks, and reading in places like the MT-Wiki (And online), the MTU and L2-MTU can be "smaller or equal". This would make sense, from what I'm seeing, the "equal" part being interesting for me. According to the OSI model, wouldn't this mean that an L3-MTU/MTU and L2-MTU of the same value would be functional, and would encapsulate all required data and overhead, as long as I have "complete path end-to-end" control? From my understanding, my CRS309 would have that control and would translate/fragment/defragment packets from my unraids to other ports with other MTU settings? Am I right in thinking that if I setup MTU 9000 links/ports to (for example) my other switches, then I would need to make sure the MTU/MRU of those switch ports would be set on that end as well to avoid issues? From what I understand, it gets out of hand when there are other out-of-control entities (Like my WAN link to my ISP) transmitting on different MTU sizes from mine, causing packets to be dropped and return ICMP "fragmentation needed" requests. But surely, that is not a concern for me, seeing as my ISP link (To my understanding as I've never reconfigured it) would be at default MTU 1500? Thinking about this I just checked, its default MTU 1500. Again, I'm not challenging anyone's advice or knowledge here, I'm trying to understand. I'm just trying to understand, what would be (In my case) an appropriate higher value for L2-MTU? Is there an known/typical overhead I should calculate in (Like how most PPPOE connections add 8 bytes to headers, reducing payload to 1492), and how can I find information about this? I'm sure there's a rabbit hole somewhere about packet encapsulation that I have yet to misunderstand, but I'm just trying to get a grip on the broad strokes and how this applies to what I'm doing.
  2. Thanks, good to know the MT transceivers would work as intended upto 30m. With that in mind, this afternoon I decided to not be lazy and finally rerun that CAT7 cable to where my WAN router is, I mentioned this earlier, and this would affect the requirement of how long I would need a capable transceiver. So with a bit of blood and sweat, I managed to run 2x CAT7A S/FTP and 2x CAT6 U/UTP across the house. After some reworking, I got the run down to 21m. I also installed and tested 10Gb rated keystones on the CAT7A cables (I'll get around to the CAT6 cables tomorrow) and I'll be able to 1) move my WAN router to the server room, 2) buy an additional CSS610 for 10Gb uplink to that part of the house and 3) go with the MT transceivers as I guess it would be safely under the 30m limit: Exactly I also have the default MTU from 1500 and get 10Gbit/s speeds. Now this has led me to an interesting issue, in that I've left everything as-is with MTU set to 9000 for the last few days, and I haven't seen any strange or adverse affects. I've been doing a lot of reading about this, and various recommendations point to setting the MTU to 9000, but I must say I've not seen mention of discriminating between MTU and L2-MTU. Also, the detriment I've seen mentioned elsewhere is the performance hit of translation between different MTU specifications on the same network. Interestingly also, I've seen no noteworthy difference in backup times between servers, last night was about 3 minutes and 46 seconds. I also can't find errors or performance penalties in any of the router, switch or system logs. The only real difference I've seen is that iperf3 now consistently results in ~9.80Gb-9.85Gb throughput, with no retries. I downloaded iperf3 to my windows laptop (An old cygwin-compiled version v3.1.3 from 2016) and that returns a solid 1Gb speed (~931 to 939 Mbit/sec), as expected. I'm also not seeing any Rx/Tx errors or Rx/Tx drops on the sfpplus interfaces for my unraids. Do you have any idea where I can look, or should look to figure out if something bad is happening? I've been reading through the MTU docs on the MT-Wiki as advised, but I'm struggling to see an advice. The Wiki does state "Make sure that MPLS MTU is smaller or equal to L2MTU", but I don't see where "MPLS MTU" applies, and there is no specific mention of it in RouterOS, only "MTU" and "L2 MTU". The MT-Wiki also mentions re "L3 MTU": "If router receives IP packet of size 1500, but MTU for outgoing interface is set to 1400, router will either fragment the packet (if "Don't Fragment" bit is not set in IP header) or drop the packet and send ICMP "Need Fragmentation" error back to originator (this is essential for Path MTU Discovery to work). Sometimes it can be bad idea to change IP MTU from its default 1500 bytes on router interfaces if complete path end-to-end is not in administrators control. Although IP fragmentation and end-to-end Path MTU Discovery is intended to handle this situation, if ICMP Need Fragmentation errors are filtered somewhere along the path, Path MTU Discovery will not work. There are several features in MikroTik RouterOS that can benefit from possibility to exceed standard MTU" Although, I'm not seeing any drops and it's only these 2 sfpplus ports on this CRS309 device alone set to 9000. Would that break things? Also, I'm a little confused by your advice, do you mean I should reduce "MTU" back to 1500, but leave "L2-MTU" at 9000? That would make sense with the diagram from the MT-Wiki, indicating that "IP-MTU" can be less than "L2-MTU": With all this in mind, what should I do?
  3. Ok, so I went back to the drawing board on this one... and I may have turned up some results. I did some more research, and was reminded that when optimising 10Gb, I should adjust the MTU settings of interfaces and switches, among some other tips and things. I think I may have lost the MTU config when changing to the Mellanox cards, and also forgot to review that. So I started with that, and adjusted the MTU to 9000 on both my unraid systems. This resulted in a disastrous set of performance results: Yikes. ๐Ÿค” But at least something changed, which in my experience prompts closer inspection. I adjusted both systems back, and saw the performance go back to what I was seeing before. Ok... So I reconfigured both systems to higher MTU and investigated the CRS309, then adjusted the MTU of interfaces sfpplus3+4 to 9000 (Actual and L2): Then tested again: Now that's more like it! ๐Ÿ˜„ Now, as far as I understand in my experience with this, and what I've read is that there can be a performance hit (Due to translation) if other parts of the network are set to other MTUs. With that in mind, and that I'm testing directly between 2 systems via the CRS309 switch, I'm going to leave this for now and do more testing from other devices to my unraid systems (Like from my WAN router). I'll maybe set the MTU between all 10Gb links to the same to prevent translation performance hits as well. I should still be getting the full 1Gb from that I think, but I'll see what it turns up. I also have bonding disabled, still at this point, so I may be able to see better bonding performance with the higher MTU. Currently all interfaces are set to 9000, but only port 1 of the Mellanox cards are plugged in and up. Gotta go for now, but will report back later. For now, further tests show fairly consistently I'm getting 9.85Gb to 9.88Gb, at least between my unraids. ๐Ÿ˜„ Hopefully, someone will benefit from reading this journey as well.
  4. So I disabled all boding on both systems and left bridging enabled, as I need that for docker and VMs. Turns out, I'm still getting about 5.4Gb using iperf3, and I'm not sure why. The onboard 1Gb is still unplugged, and the Mellanox 10Gb is set to eth0, used as the only bridge interface. Maybe the bond isn't the problem at all, but something with the bridging? This is strange.
  5. A x4 slot will definitely have more lanes and bandwidth than a x1 slot yeah, but for the performance I think I'd get (And as others have posted in this thread, it seems reasonable to get around 6/7Gb from a PCIe 3.0 x1 slot. I don't have thunderbolt on either of these boards (in my signature) as well. Edit: Also, both the NVME slots in both servers are in use for fast NVME storage, not really looking to replace them, they work pretty well. That would compromise actually being able to transfer ~10Gb over the network, as spinning drives won't cut it. I did look into this a while ago, but I didn't find much reliable information on usable adaptors for this. Do you have any suggestions? I remember looking at a reddit thread discussing this (among a few other places) and they seemed to conclude that using USB3.1 for 10Gb networking isn't as good as it sounds. Have you had a different experience?
  6. Thanks, I appreciate the advice. I think I might just go ahead and replace these with 2x of the 10Gb ethernet transceivers you indicated before, as I know I'll use those before too long. The Mikrotik 10Gb ethernet transceivers are available also on eurodk, do you think these would be comparable to the fs.com modules? I assume they would be preferable for Mikrotik hardware. Ok, well I think I would need these for using docker (without custom config) and my VMs anyway. I'm not sure what you meant by the PCIe slots specs change? I've not had many issues using this AMD setup, I expected a lot more bugginess tbh. Currently the cards in both systems are in the second PCIe x16 slot, especially as I want to get a working baseline for some time before eventually migrating them to an x1 slot. I know now that raw they get close to the 9.5Gb I should expect. Do you mean in the "interface rules" section of the network settings? I already set that order, I think I posted the screenshot before. Is there another part? Is this correct: I currently HAD them set like this (below), and I set active-backup bond0 members to eth0 and eth1. They should be functionally the same, as 10Gb port2 is unplugged, but I've set them back to the first configuration (above), prioritising both the 10Gb ports, and unplugged the onboard: Yeah, I'm not sure what to do about it. Nothing else changed, all same hardware and bonding config. I'm about to finish the reconfigure of both systems to use only eth0 and reboot, so I'll test again and edit this with the results (As I'm posting from within one of the VMs now). ๐Ÿ‘ Edit: Well I tested again with only eth0 in bond0, and everything else working as expected. The speed seems slow again, not sure why, as everything is going through only the CRS309. I'll have to do some research into why the active-backup bonding configuration slows everything down. Test results:
  7. Thanks for the advice, I managed to test the cards today, but I need to do some more testing of the Ubiquity 1Gb transceivers. So far though, I couldn't get them to be recognised at all in the CRS309, they don't seem to show up from what I can see. I'm going to try them in one of the CSS610s and see if they are recognised there at all. I disabled the bridge and bond0, and did another set of speed tests between servers. Looks like with bond and bridging disabled (This test was done with the array unmounted and nothing running) I can get around 9.39/9.4Gb between systems fairly reliably: After the speed test, I enabled bridge, but the networking became quite unreliable and wouldn't function correctly on either server. The onboard 1Gb cards are recognised as eth0 and will not allow me to reliably use the 10Gb cards without disabling the onboard adaptors. After some frustration, it looks like I may need to Yeah thanks, I upgraded the firmware of both servers cards to the latest 2.42.5000, so that I could test some more with latest firmware, as the cards had different old versions installed. Both cards now show identical details: After installing the firmware, I re-enabled the bond0+bridging and tested again. It seems that I'm getting slower speeds than before, which is interesting: I guess I may need to decide if I want to use this bonding setup, as it seems the speed is significantly compromised by having it enabled. I guess either I'll have to disable the active-backup configuration, and disable to onboard 1Gb, or try to fix it on the switch config side if that is the issue. The 1Gb transceivers not working is also an annoying issue that I had planned by using them would solve the port forwarding issue.
  8. Oh something I forgot to mention above. I installed the Mellanox Firmware Tools and got the details of my cards: unraid1: unraid2: The cards should be identical as well, but it seems they have different firmwares and also versions. Thing is, I've been to the Mellanox ConnectX-3 firmware download page, and I'm a little confused which firmware to download: Would the different firmwares be responsible for the performance behaviour I've been experiencing?
  9. I do have a few questions, maybe there's some things I'm missing. So I monitored this over the last few days, and the speed has been fairly consistent, but still not at the 9.5 that I figure it should be capable of. There are backups that runs every night between servers (for VMS, docker appdata, personal files, etc) and I have a particular VM that has a primary vdisk of 200GB. That particular backup used to consistently take about 31 minutes, 19 seconds. Now the same backup runs at about 3 minutes, 44 seconds. I've not been seeing any errors of packet loss in any of the logs as well, but I've rerun a few tests over the last few days between servers. I'm pretty happy with it, but it definitely should be doing a touch better given the specs in both systems, so there should be no shortage of bandwidth or bottlenecks that I'm aware of: Yeah it's setup as an active-backup configuration. I've still got the CRS running in RouterOS bridge mode, as I haven't yet moved my WAN router and that does my WAN routing pretty well at this point. I can't seem to setup the active-backup bond though, as the bridge mode throws an error: Something else I'm having trouble with, I went and purchased a couple Ubiquity 1Gb RJ45 transceivers but they seem to not be recognised at all by the CRS309. I was trying to test bond by having everything on the same switch. I may have goofed. Are these not compatible: This is my next move, I haven't had much chance to bring everything down and play around until today. Will post back those results.
  10. Thanks, I'm definitely going to be updating and investigating more into these in the coming days. In recent weeks I've already been reading and watching setup guides and configuration instructions. I haven't used any Mikrotik switches before, so it's going to be lots to learn. ๐Ÿ˜ This is via the CRS309, using the DAC cables I got included with the Mellanox cards. Bearing in mind I haven't adjusted anything, its all just plugged in and pushing data right now. The test show they don't seem to be running at 100% full speed, but I think I'll watch overnight and see how the throughput goes when backups run between servers. The current core setup, in its current position, due to be moved at some point soon: The unraid servers are hooked into SFP+ ports 3 and 4. The other CSS610 on the lab/desk will be setup and uplinked in the next days.
  11. Alright... so it looks like I got it working. Reconfigured the DHCP and the active-backup bond0 seems to be up without trouble. some new iperf3 results between servers: Not quite full 10Gb, especially as they are in physical x16 ports... but pretty good. ๐Ÿ˜
  12. Actually, yeah I think I fixed it. I took a look in those files after I posted and just deleted the listing for eth3. Seemed to work: I think I might have fixed it, by deleting the eth3 entry from the network-rules.cfg I have the Mellanox as primary now, need to setup the second server and do some tests. Oh I see, I was thinking I could use the MAC of my onboard as the bond0 MAC. Maybe that's a bad idea then, and I should just configure the Mellanox as the only card. I should probably do that, just reconfigure the router DHCP rather than be lazy. ๐Ÿ˜„ That sounds useful. I'll definitely read through this, thanks.
  13. I think I may need to open a new thread for this issue. I'm trying to setup acive-backup on bond0, so that the (eth1) Mellanox 10Gb port1 is primary, and the (eth0) onboard 1Gb is backup. The duplicate MAC error seems to be blocking it. From what I'm reading I can't seem to query the bond status.
  14. It was on the old unmanaged Netgear switch I was using until this upgrade. I just finished getting everything installed and setting up the links, and looks like the core CRS309+CSS610 have booted up ok and pushing data. Before I get into setting them up, I seem to have run into a strange issue with the MACs of the interfaces. The bonding configuration seems to think there are duplicate MAC adresses: eth0 is the oboard 1Gb, and eth1/2/3 seem to be the Mellanox dual-port card. I think I need to delete one of the interfaces, reading up on why this might have happened. If I try to change something in the GUI I get an error: ๐Ÿค”
  15. Thanks, I've got everything setup... I even managed to move some hardware around, and I have both the Mellanox NICs in the 2nd PCIe x16 slot. I'll need to move them at some point, but for now I'll ideally get full speed out of them, and glad I have the x1-to-x16 adaptors. ๐Ÿ˜„ This is from the old 1Gb cards, but one is the onboard 1Gb and an Intel 2x 1Gb card. I had them bonded but the performance was still lacklustre from what they should be. I should be getting about 3Gb/s out of the bonded link... but the 10Gb card should do better.
  16. Looks like I'm having fun today: And some current iperf3 results between servers:
  17. Hey, although I have an AMD build, what you've listed is essentially what I've been doing with my setup. I've had great results, everything is listed in my signature below.
  18. Thanks I really appreciate you helping me out with this, and of course everyone else in this thread. It's been a long time figuring this out, getting feedback, deciding the setup. ๐Ÿ˜… I just ordered the eurodk kit. Once I get confirmation, I'll get the amazon order done. Also, yeah everything shows in stock for me. Not sure why it's not showing up in the screenshot as such.
  19. Updated the shopping list: Saw that I left the RJ45 transceivers, not getting those. ๐Ÿ˜…
  20. @Ford Prefect Sorry, reading back, I think I may have fudged my explanation. My bad, I really appreciate you spending your time to respond to me about this. I'll try again. ๐Ÿ˜… So, I'm pretty set on having the CRS309 as my core switch, in the server room, which my servers would DAC directly into. I have 2x 3m DACs with the cards I got already, so I'm super happy about that. ๐Ÿ˜ For expansion or other devices (WAN router, WiFi AP, anything else) in the server room, I think could either get a couple 1Gb RJ45 transceivers or short DAC to a CSS610? That is sounding more like a better option, the more I think about it. For the office/desk I'm thinking of another CSS610 with a 10Gb uplink to the core CRS309, which would definitely satisfy that requirement. Currently I have an unmanaged netgear 8-port switch on my desk so it would be a major upgrade there also. I guess the question becomes how do I uplink from the office/desk to the server room. This is a very nice find, I could see getting a 7m DAC for 26euros being perfect for the uplink. ๐Ÿ˜ Yeah the 0.5m would be because they would be right next to each other. You've got a good point though, a 1m or 2m DAC would be a better idea in case I need to move them in the room (Likely). Yes, my idea is that the CRS309 would be replace and be my new core switch, and would spoke out to other places in the house. Doing a quick review now, I have a few things hooked into my current core server room switch: -both unraid servers (direct, currently RJ45) -office/desk (uplink via 1Gb RJ45) -home automation cabinet (uplink via 1Gb RJ45) -WiFi AP (direct 1Gb RJ45) -pfSense WAN router (uplink via 1Gb RJ45) I didn't mention before, but the WAN router is on the other side of the house, in the living room where the outside line comes in. That uplinks via an unmanaged switch, via a very long ~30m CAT5 ethernet cable that I consider "dirty" WAN. Xbox is on that switch also. I'm going to rerun at some point, as 2x CAT6/7 ethernet, so that the WAN router has a dedicated ethernet and I can move the WAN router to the server room. Or I could replace the unmanaged switch there with a CSS610 or something, and I can VLAN it. Not sure yet. Yeah, looking at my quick review above, I might well need the extra ports of the CSS610 after all. ๐Ÿค” The 7m DAC might be the best option for me here, for 26 euros it seems simple and ideal. a 24 euro jump to 10m though... not sure if I need the extra length. I use my primary unraid as my GPU passthrough desktop and have some 7m/5m cables run between the rooms already. Works pretty well. ๐Ÿ˜ I should really do that, draw it out. Gonna have a look at that today. As it is right now, I think the CRS309 core and 2x CSS601 uplink will work pretty well going forward. Just need a 1/2/3m + 7m DAC, and I should be good to go.
  21. Hi, sorry for the delay, had some stuff going on and getting back on track with this now. @Ford Prefect thanks for the link to eurodk.de, that's a great site. I made up a shopping list some time ago, and want to confirm if it looks about right: So something critical I'm wondering about, is that I have a distributed setup in my house. My servers are in one room with the core switch, and I have my desk/office in another room where my current uplink lab switch is. The room my servers are in, there's not much in there at the moment (basically by themselves apart from a couple 1Gb devices), but could I reasonably get away with a couple 1Gb SFP modules? I need at least 1x 1Gb for my WAN router and another for WiFi AP. After that I suppose its more cost-effective to have a CSS610? I know you said its better to get a CSS610, but is that really a better idea than getting a couple transceivers? I guess cost-wise it ends up not much more (extra ~40 euros instead of 2 transceivers). If I put the CSS610 on my office desk for lab work, and uplink to it via the multi-mode fiber, will a couple transceivers be better, or should I really just fess up the ~40e for a second CSS610 in the server room with the short DAC? Maybe I'm overthinking it, just get the second CSS610? I feel like I'm deciding the future of the entire infrastructure right now.
  22. Well its been an interesting few days ordering parts. I ordered the PCI-E x1 risers and dual-port 10Gb cards, and they have arrived. I also tried to acquire an CRS309-1G-8S+IN on ebay but the seller refused to ship to Netherlands, from Germany and would not respond to enquiries. I'm pretty disappointed in that actually, as it would have been perfect. So now I'm looking to purchase a CRS309-1G-8S+IN new with a couple transceivers for other connectivity (I have DACs for servers now). @Ford Prefect and @SimonF I'm wondering if I can use any SFP 1Gb RJ-45 transceiver or are specific models compatible? I'm looking at this listing and wondering if it would be compatible or if I should order from fs.com as suggested? As far as I know, either should be compatible?
  23. Oh Damn, sorry @SimonF wrong shoutout to @Ford Prefect my bad. Thanks to you both for the advice still. I would like to eventually use the second port for failover, and I don't see any issue with 8 SFP+ ports any time soon. Currently I'm using a 1Gb unmanaged switch, as I had an old HP Procurve 100Mb switch I used years ago with VLANs, but use the unmanaged now for the 1Gb bandwidth (And it was free and has lots of RJ45 ports). I've been looking at basically a new setup, which is why I haven't purchased anything yet, but just haven't got around to it until recently. I'm planning to use the MT CRS309-1G-8S+IN switch as a 10Gb core for my servers with a 1Gb port for my WAN router and a 1Gb/10Gb uplink to a 1Gb switch for the rest of the network, which I would also need to purchase. I was looking at the CRS326-24G-2S+IN as a good uplink switch with plenty of 1Gb RJ45 ports, but then maybe I should look at a single switch that can do everything... but that sounds more expensive. Alternatively I could purchase any other new/used 1Gb managed switch, doesn't have to be MT branded. Still looking at options there. Anyhow I think maybe having a separate switch would make placement easier for me, as I could leave the 10Gb close to my core servers and continue with the rest of the house hooked into the 1Gb switch. I think if I used something like the CRS317-1G-16S+RM, then I would also have to purchase 1Gb transceivers for everything, and it seems preferable to maintain my network direct to an ethernet switch. I also want to add more 10Gb later without compromising the 1Gb connectivity. In the end I think I can stretch to the dual-port Mellanox cards, even if I don't use the second port just yet (Especially as the card bandwidth is already limited to 6Gb).
  24. @Ford Prefect Thanks so much for the pictures, these are perfect for what I was wondering about. The riser looks like its doing a great job, glad I ordered 2 already. I didn't realise until seeing the pics that you're using a dual-port 10Gb card. Are you using those for Dynamic Link Aggregation, Backup Redundancy, Load Balancing or another use? Since the card bandwidth is limited to 6Gb, you'd get 3Gb out of each if used together? I'm checking ebay.de listings at the moment, and wondering if I should get a single-port or dual-port card? Listings like this and this have multiple single-port cards, but other listings like this and this have dual-port cards for about the same price as a bundle of single cards. I found another listing here for a pair of dual-port cards with a couple DAC cables included. Wondering if that last deal would be an ideal setup for a good start with the MT switch? The 2 included DACs would get me started, but I'd have to look at getting tranceivers if I'm going to expand. I guess that would give me the most flexibility at this time to expand how I want. Personally, I'm into redundancy as well so the dual-port cards would allow for that as well if I pick up a few more DACs, I guess?
  25. @SimonF I saw some pictures, but I would appreciate a pic if its easy for you. I saw another item, this one, that has the plate I think you're referring to. Is it much of an issue without it, I guess its secure with the usual bracket screw? Edit: Looks like the metal bracket wouldn't fit the X1-to-X16 riser anyway. I've ordered 2 just now, worth it.