Jump to content
misterwiggles

UPDATED: Network drops on 10g switch. Now unknown issue again.

50 posts in this topic Last Reply

Recommended Posts

39 minutes ago, Benson said:

Last update should be only new hardware revision could fix.

I was told on the UBNT forums that any hardware post revision 6 has been 'fixed' as to the port issues. My hw revision is 13 so I was assuming the problem was elsewhere and likely firmware related. But will try to get an answer out of UBNT today and swap the switch regardless in hopes of solving this. But would really like to know why I only see this issue with Unraid vs ESXi.

Share this post


Link to post

@Benson UBNT support asked me to try setting the RSTP priority values differently on my switches. My three switches all had the same priority value. They asked me to try setting my 'root bridge' switch - the US-24 in my case - to lower value than the XG and then to reboot both switches. I will update here once I've retested as to whether this resolve my issue. Again, posting all this here in case it some day helps someone else in same situation.

Share this post


Link to post

Thought I'd put this here if you were interested in using the 4 rj45 ports at some point.
 


Also the SFP + ports have been working fine for me, constant 10G to my other servers/PCs.

Edited by slimshizn

Share this post


Link to post

My XG-16 is a newer hardware revision so I'm not sure what's going on. Other than that I have no issues with the XG as it has the sfp+ ports I need and can use for expansion. I'm going to turn the LB6M into a brocade and probably use that as well, and contact UBNT for a hardware replacement. 

Edited by slimshizn

Share this post


Link to post

Thanks for that info. As an UPDATE: I ultimately not only changed the RSTP priorities but also put my XG as the Core root switch in my chain and moved the US-24 to hang off that. Also moved the USG connection to Port 16 on the XG. I was *sure* this would solve my issues. But it didn’t. I still got disconnects after a few hours on Port 13. Have been running Port 13 at 1g for the last 4 days and there have been no issues.

 

I’m frustrated with Ubiquiti support as I have an open case on this issue and they escalated for me to a higher support team. But I’ve heard nothing since then and have emailed them repeatedly to engage. Going to try again today via chat. I find it unacceptable to have a switch which is being sold as working only to find that one of it’s features just doesn’t work. While I can move my server to SFP+ connection, which I will do by putting in a new SFP+ NIC card this week, a couple other devices have 10Base-T built-in and are not upgradable to SFP+. So I need those RJ45 ports to work also.

Share this post


Link to post
3 hours ago, misterwiggles said:

Thanks for that info. As an UPDATE: I ultimately not only changed the RSTP priorities but also put my XG as the Core root switch in my chain and moved the US-24 to hang off that. Also moved the USG connection to Port 16 on the XG. I was *sure* this would solve my issues. But it didn’t. I still got disconnects after a few hours on Port 13. Have been running Port 13 at 1g for the last 4 days and there have been no issues.

 

I’m frustrated with Ubiquiti support as I have an open case on this issue and they escalated for me to a higher support team. But I’ve heard nothing since then and have emailed them repeatedly to engage. Going to try again today via chat. I find it unacceptable to have a switch which is being sold as working only to find that one of it’s features just doesn’t work. While I can move my server to SFP+ connection, which I will do by putting in a new SFP+ NIC card this week, a couple other devices have 10Base-T built-in and are not upgradable to SFP+. So I need those RJ45 ports to work also.

Yeah if those ports aren't working correctly(10Base-T) it should cut down the price a couple hundred dollars. This is a 300 dollar switch at the most. 

Share this post


Link to post

Sent my XG back, have been using the LB6M just fine for a while now. Going to order a copper 10G Base-T insert for an additional 10G to another PC. 

Share this post


Link to post

Update on my end: Using the LB6M with a 10G Base-T transceiver I stopped the array and tested. Seemed to be working fine so I started up the array. Next think you know the connection drops and there is no way to connect. So I tried a graceful reboot pressing the reset button which worked. Flaky connection after so I went back to aggregating two 1G ports instead. Maybe my intel ports aren't supported? Intel® X540-AT2 seems to be the NIC.

Share this post


Link to post

My update is that I ultimately installed an SFP+ card in my Unraid system and then connected that to the XG via a fiber cable. It has been rock solid since then. Not a surprise based on what I had expected and read about the flakey 10G-T ports on the XG. Disappointingly, UBNT has STILL not resolved my open support case on this despite promises that the escalation team would engage within a week. I think clearly those ports on the XG have issues with certain NICs as you suggest and it could come down to the ports not being powered properly so the connection getting dropped. I think UBNT is going to resolve this in subsequent hardware revisions and will probably do so silently. I’d like to have my support case result in a hardware upgrade when they fix this. I ultimately didn’t want to return the XG because my whole network is run off of the UBNT Controller.

Share this post


Link to post

Please let ubiquiti know that we at Unraid have been monitoring this thread and are willing to provide them a free registration key for testing if they are concerned it may be kernel / driver specific and they want to test with our OS. I can tell you that I have personally help set others up with multiple 10g solutions over copper and haven't really had any major issues.

My update is that I ultimately installed an SFP+ card in my Unraid system and then connected that to the XG via a fiber cable. It has been rock solid since then. Not a surprise based on what I had expected and read about the flakey 10G-T ports on the XG. Disappointingly, UBNT has STILL not resolved my open support case on this despite promises that the escalation team would engage within a week. I think clearly those ports on the XG have issues with certain NICs as you suggest and it could come down to the ports not being powered properly so the connection getting dropped. I think UBNT is going to resolve this in subsequent hardware revisions and will probably do so silently. I’d like to have my support case result in a hardware upgrade when they fix this. I ultimately didn’t want to return the XG because my whole network is run off of the UBNT Controller.


Sent from my Pixel 3 XL using Tapatalk

  • Upvote 1

Share this post


Link to post
2 hours ago, jonp said:

Please let ubiquiti know that we at Unraid have been monitoring this thread and are willing to provide them a free registration key for testing if they are concerned it may be kernel / driver specific and they want to test with our OS. I can tell you that I have personally help set others up with multiple 10g solutions over copper and haven't really had any major issues.

 


Sent from my Pixel 3 XL using Tapatalk
 

 

Thanks for looking into this a bit, is there any way you can let me know if the 10GbE NIC on the GIGABYTE - GA-7PESH2 is supported for 10G? According to the website it's the Intel® X540-AT2. Right now only getting stable connections via 1G on the same NIC.

Edited by slimshizn

Share this post


Link to post
1 hour ago, slimshizn said:

Thanks for looking into this a bit, is there any way you can let me know if the 10GbE NIC on the GIGABYTE - GA-7PESH2 is supported for 10G? According to the website it's the Intel® X540-AT2. Right now only getting stable connections via 1G on the same NIC.

So I'm not too sure on the -AT2 designation on the NIC.  I know that X540 NICs work fine because I've set up multiple for various clients over time and they all work wonderfully, but all of the NICs I've worked with weren't built-in on the motherboard.

 

One thing that is worth noting is your use of a Gigabyte motherboard.  Generally speaking, I cannot recommend that brand after having NUMEROUS issues with them over the years on both their motherboards and GPUs.  Their motherboards are janky (at best) and especially when it comes to utilizing a server OS.  In the future, I would recommend ASRock, Asus, and SuperMicro as solid brands to stick with for motherboards.

  • Upvote 1

Share this post


Link to post

No problem, in retrospect I would have looked at something else if I had known. For now though, I need(want) to find a way to have it work. Is there anywhere I can ask to have X540-AT2 10G NIC support added ( if it's not ) to Unraid?

Edited by slimshizn

Share this post


Link to post
13 hours ago, slimshizn said:

No problem, in retrospect I would have looked at something else if I had known. For now though, I need(want) to find a way to have it work. Is there anywhere I can ask to have X540-AT2 10G NIC support added ( if it's not ) to Unraid?

The reason I suspected this was an Unraid driver issue was because I had an identical Gigabyte server - same one as @slimshizn - and the onboard NIC 10GT connections dropped with the XG as well as failing to even pull an IP address during a reboot. However, I have a 2nd server with identical hardware running ESXi and that DID NOT DROP any 10G connections when connected to the UBNT XG. So @jonp I do think there is something going on with Unraid’s drivers and their interaction with the UBNT gear at 10G-T. I just couldn’t spend any more time trying to debug this and without UBNT actively engaging it was hard to make progress. I suspect UBNT is going to blame this on Unraid honestly and then there will be a blame game of pointing at the other vendor. So I needed a working solution and an SFP+ card in the Unraid box solved it for me. I will follow up with UBNT this week to not let them off the hook and will suggest what @jonp offered as a way for them to look into the issues. Is there some contact at Lime Tech that I should give them if they ask?

Share this post


Link to post
The reason I suspected this was an Unraid driver issue was because I had an identical Gigabyte server - same one as [mention=81437]slimshizn[/mention] - and the onboard NIC 10GT connections dropped with the XG as well as failing to even pull an IP address during a reboot. However, I have a 2nd server with identical hardware running ESXi and that DID NOT DROP any 10G connections when connected to the UBNT XG. So [mention=62528]jonp[/mention] I do think there is something going on with Unraid’s drivers and their interaction with the UBNT gear at 10G-T. I just couldn’t spend any more time trying to debug this and without UBNT actively engaging it was hard to make progress. I suspect UBNT is going to blame this on Unraid honestly and then there will be a blame game of pointing at the other vendor. So I needed a working solution and an SFP+ card in the Unraid box solved it for me. I will follow up with UBNT this week to not let them off the hook and will suggest what [mention=62528]jonp[/mention] offered as a way for them to look into the issues. Is there some contact at Lime Tech that I should give them if they ask?
Sure! Give them mine:

jonp@lime-technology.com

Sent from my Pixel 3 XL using Tapatalk

Share this post


Link to post
On ‎11‎/‎15‎/‎2018 at 12:37 AM, misterwiggles said:

I have a 2nd server with identical hardware running ESXi and that DID NOT DROP any 10G connections when connected to the UBNT XG.

Is it possible running Unraid on this hardware to prove Unraid relate or not ?

 

Or

 

Problem hardware running ESXi

Share this post


Link to post

I also have network drops since years now. Some days not a single time, sometimes multiple times in a minute...

 

I cant figure out where it comes from. Using cheap switches, Cat7 cables, i have 3 subnets, its happening to my PCs (other subnet then unraid).

 

Couldnt figure out whats wrong. This year i get an internet upgrade, so i can change one router, which might be the issue. I hope.

 

Did anyone find out what the problem was in their case? Just out of curisity?

Share this post


Link to post
5 hours ago, slimshizn said:

@misterwiggles Any luck with you on this setup? I could never get a stable connection with 10GbE, 1G however works fine still.

I gave up. I never got it to work and Ubiquiti just never followed up on the support case I opened. I expect the core issue DOES involve the interaction between Unraid using the on-board Intel NIC and Ubiquiti switch. I ended up buying a 10G SFP+ card and putting that in my Unraid server instead. That has been rock solid since.  I think those RJ45 ports on the XG do have issues with specific NICs and I've concluded that those ports, in some cases, don't provide enough power to keep the connection up. But it seems to DEFINITELY involve the Unraid driver in some weird way as I don't see the issue on the exact same hardware running ESXi to the same switch RJ45 ports. Unraid seems to handle the SFP+ 10G NIC just fine. I wish I had an answer as to why that config was flakey. Pretty disappointed in UBNT for not resolving the case.

Share this post


Link to post
2 hours ago, misterwiggles said:

I gave up. I never got it to work and Ubiquiti just never followed up on the support case I opened. I expect the core issue DOES involve the interaction between Unraid using the on-board Intel NIC and Ubiquiti switch. I ended up buying a 10G SFP+ card and putting that in my Unraid server instead. That has been rock solid since.  I think those RJ45 ports on the XG do have issues with specific NICs and I've concluded that those ports, in some cases, don't provide enough power to keep the connection up. But it seems to DEFINITELY involve the Unraid driver in some weird way as I don't see the issue on the exact same hardware running ESXi to the same switch RJ45 ports. Unraid seems to handle the SFP+ 10G NIC just fine. I wish I had an answer as to why that config was flakey. Pretty disappointed in UBNT for not resolving the case.

As much as I'd love to blame ubiquity, I can not. I bought the lb6m and a sfp+ rj45 10g adapter and had the same luck. I fully believe this has to do with unraid and this specific variant NIC. I'd love to know who to contact to get a better answer than to just blame the motherboard. Unraid should work with gigabyte not against them as they don't seem to be going away any time soon. Besides that I don't know where you can find a motherboard like the pesh2 with all of the extras it has. 

Share this post


Link to post

So, just to give feedback, my issue was resolved by a faulty 1m cable... 

Share this post


Link to post
6 hours ago, nuhll said:

So, just to give feedback, my issue was resolved by a faulty 1m cable... 

Nice I'm glad you got it worked out. I had tried four different brand new sets of cables for testing purposes. They all worked with 1G connections, but not 10G. It's not the cables on my end.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now