Unraid networking help

[rogueone]

Dear All,

 

I am currently running the latest unpaid version,

I have a 1 GB nic on the motherboard

and a PCIE Car with 2 nic of 10GB Intel NIC X710-T2L PCI-Express- x8 2x RJ45

 

I would like to connect to my 10gb SFP+ on unpaid the main connection,

Use a backup connection of 1GB incase it drops from an other router and inter provider and 

Connect my second 10GB to a synology for an extra disk link as I cannot put additional disk in my machine,

 

I have spent 3 days to make it work, I cannot seem to be able to,

 

Here is a screenshot of my actual config, your help would be highly appreciated,

 

Have a good day

 

Edited April 8, 2023 by rogueone

[Vr2Io]

You shouldn't bond them, config them in standalone and assign in different IP subnet.

Edited April 8, 2023 by Vr2Io

[rogueone]

Thank you, really appreciate it, 

 

Indeed but each time I do the un-bonding, I loose connectivity, something is odd, I cannot seem to figure it out

 

Edited April 8, 2023 by rogueone

[Vr2Io]

7 hours ago, rogueone said:

Thank you, really appreciate it, 

 

Indeed but each time I do the un-bonding, I loose connectivity, something is odd, I cannot seem to figure it out

 

My only diconnect was happen on 10G SFP+ with Mikrotik CSS326, after change to HP switch then no any drop, btw all 1G NIC never have disconnect issue, no matter which NIC or switch.

 

I never bond NIC.

Edited April 9, 2023 by Vr2Io

[rogueone]

Thank you,

 

yes that’s something I’ve experienced. 
 

I’m waiting for an USW-Aggregation 8 port switch from UniFi. Hoping that would stabilize the network and avoid disconnections. 

Can I see your config?

 

thanks 
 

 

[Vr2Io]

1 hour ago, rogueone said:

I’m waiting for an USW-Aggregation 8 port switch from UniFi. Hoping that would stabilize the network and avoid disconnections.

This is SFP+ only switch, how you connect with 1G port ? In most case, people should buy 10G + ( 1G / 2.5G ) switch.

I have UniFi XG-16 switch, this is my first 10G switch, those 4 10G-T port was connect several 1G switch(s) before, but now all retire which replace by two switch.

 

Currently I use two switch to connect two area, they are Mikrotik CSS326 (2 SFP+) and HP Aruba 1930 (4 SFP+), different 10G & 1G device connect either and one 1G link to connect both switch.

 

Edited April 9, 2023 by Vr2Io

[rogueone]

Thank you

 

i have bought currently 1 PCIE Card with 2 nic of 10GB Intel NIC X710-T2L PCI-Express- x8 2x RJ45 which I am going to replace with the same in sfp+. 
 

[Vr2Io]

I mean no other 1G ethernet device, i.e. router, cam, printer ... ?

 

You buy two SFP+ to RJ45 converter ?

Edited April 9, 2023 by Vr2Io

[rogueone]

Yes I have them, but I am replacing with an sfp+ card.

 

On this network I wanna put only the unraid on the USW Aggregation switch.

Do you know how to do that?

 

 

Edited April 9, 2023 by rogueone

[Vr2Io]

I would suggest connect Unraid to USW by 2 link and bond them in LACP (1+1), then connect Synology and existing 1G network to USW too. So use 4 SFP+ port.

 

To be honest, if NIC and Synology were 10G RJ45, then you should't buy USW-Aggregation 8 and use converter, those converter use lot of power and quite hot.

 

Pls also ref. below review for those converter

https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/

Edited April 9, 2023 by Vr2Io

[ljm42]

On 4/8/2023 at 10:10 AM, rogueone said:

Indeed but each time I do the un-bonding, I loose connectivity, something is odd, I cannot seem to figure it out

 

You need to break the bond in order to use the NICs independently.

 

The bond has its own MAC address, so when you break it you will likely get a different IP address from the router.

 

You want to configure the main NIC as eth0

 

I would get that much working before adding more complexity.

[rogueone]

Thank you for all of your responses,

 

So I now have receive the entire material,

 

I have installed a Broadcom network card SFP+ P210P PCI-Express- x8

with both ports connected with Ubiquiti Module SFP+ 10 Gbps, and Lightwin Cable  patch LC-LC, Multimode, OM4,

Both are connected to a USW-Aggregation switch with aggregation turned on on port 3&4 on the aggregation switch, with jumbo frames, flow control ON and spanning tree (STP) with  priority set at 28'672.

 

The unraid server is connected to both ports with:

- ETH0 and ETH1 on bond with 802.3ad turned on,

- Static IP and DNS: 1.1.1.1 and 1.0.0.1

- Jumbo frames on at 9000

- Vlan On as the traffic needs to be isolated from the rest of the network,

 

I still get connection on some of the dockers, Plex, sonarr and radarr, but deluge and the unraid interface are not working,

I don't understand, I do not know how to get diagnostics as I have the server running headless, I also cannot access the server on the local network, however I can ping it and it responds.

 

I am entirely lost, I have had this problem every time I am trying to make it work on 10GB, 1 gb connected to a router A7501 from zyxel that works perfectly fine.

 

Anyone with an idea would be welcomed

 

Thank you

 

EDIT: Managed to reboot the server via SSHing into it through unified USW debug console, I can access now, here is my current screenshot of my config, and managed to download the diagnostics

 

 

 

rogue-diagnostics-20230415-1446.zip

Edited April 15, 2023 by rogueone
Update on my server

[Vr2Io]

Where are the 1G ( Internet network ) connect ? to USW or elsewhere ?

Pls don't setting any VLAN in Unraid and USW, also don't use jumbo frame, you need simple and make all work first.

[rogueone]

Thank your for your response,

 

I am not sure I understand your question, however the USW-Aggregation is connected over fibre to the UDM PRO, 

 

I have setup the VLAN for an isolated network as it is on my office network.

 

Also I can connect, it drops from time to time.

I thought jumbo frames were a good thing.

 

[Vr2Io]

7 minutes ago, rogueone said:

I am not sure I understand your question, however the USW-Aggregation is connected over fibre to the UDM PRO

Just note you have UDM PRO, that's fine.

 

7 minutes ago, rogueone said:

I thought jumbo frames were a good thing.

I don't use jumbo frame, it always troublesome. 

 

I also apply VLAN to isolate different network / device / docker ....

Edited April 15, 2023 by Vr2Io

[rogueone]

When I was planing to use VLANS it was to isolate Unraid from the rest of the network (office network) on the screenshot I put, does those vlan option do that or its for the docker services? As this I don’t care for

[Vr2Io]

29 minutes ago, rogueone said:

When I was planing to use VLANS it was to isolate Unraid from the rest of the network (office network) on the screenshot I put, does those vlan option do that or its for the docker services? As this I don’t care for

Yes, those for docker / VM usually. You should simply connect Unraid to SW and simple untag the traffic, Unraid don't need to know you connect to which Vlan in your neteotk.

[rogueone]

Ah perfect. I did remove them and remove the mtu as well. 
here is a screenshot for the usw setup 

I hope I won’t get any disconnections again. 

[ljm42]

Jumbo frames are leftover tech from the early days of networking, IMHO they provide endless headaches for very little speed improvement on modern hardware. The only time they should be used is on a dedicated subnet that doesn't connect to the Internet, where every device on that subnet uses jumbo frames.

 

If you ever have any kind of networking problem, your first thought should be to disable jumbo frames and the second to disable the bond.  Both of these require proper configuration of other networking gear not related to Unraid. Supporting this requires knowledge of both Unraid and your networking gear and is beyond the scope of what most people can help with.

Fix Common Problems should be warning about both

 

[ljm42]

Having said that, if this is one of those "I want to try everything I possibly can" situations that can be a lot of fun : ) Just realize that is not the same as a "set it and forget it" situation; the more complex your network the harder it is to deal with issues.

 

I too really like the Ubiquity ecosystem. I have the UDM SE with a USW-EnterpriseXG-24 switch.  I have two Unraid servers with dual 10 GbE NICs and a 10 GbE connection for my laptop. Plus other 1 GbE, 2.5 GbE, and WiFi 6 devices.

 

eth0 on each server is one of the 10 GbE ports connecting to the main network.  This is a simple, easy to understand setup. But I do have those second 10 GbE ports on each server asking to be used : )  I can think of three options:

 

1) Setup a jumbo frames subnet. I did this by direct-connecting eth1 on the two servers and changing the mtu to 9000 on that connection. But it only added 5-10% to the speed and for me was not worth the hassle of remembering to use the alternate network when doing server to server transfers.

 

2) Bond the two ports. The thing is, this doesn't give you a 20 GbE connection between the servers. What it does is let each server talk to two clients at 10 GbE at the same time. That might be useful in a large business but in my home lab, not so much.

 

3) Use the second NIC for Docker containers. This is where I'm leaning: https://forums.unraid.net/topic/137048-guide-how-to-solve-macvlan-and-ipvlan-issues-with-containers-on-a-custom-network/ 

 

Anyway, that is my situation. Hopefully it gives you some ideas : )

 

[rogueone]

Thank you very much for your response,

 

Did you ever faced random disconnection?

What's your setup and config like if its not too much to ask?

[rogueone]

I am going to look into option 3

[ljm42]

For random disconnections I would get rid of any off-brand switches and replace them with Unify. After that, I'd suspect the cabling.

 

I threw out all of the random network cables I had collected over the years and replaced them with Monoprice Slimline Cat 6A cables https://www.amazon.com/Monoprice-SlimRun-Cat6A-Ethernet-Patch/dp/B01BGV2YS2/  I really like the slim form factor and they handle 10 GbE just as well as the Cat 7 cables I tried.

[rogueone]

So I am full fiber now. 
Brand new, all connected to the new USW aggregation switch.