Network settings 2x 10GbE aggregation and 1x 1GbE failover / active backup

[timomatrixer]

Dear Community,

 

I would like to know how I setup my network settings if I have the following in mind:

PCIe card with 2x 10GbE connected as aggregation pair to my Ubiquiti USW Aggregation Switch for main connectin with max possible bandwidth. (eth1 and eth2)

Motherboard 1x 1GbE connected as failover / active-backup to my Ubiquiti UDM Pro. Just when Switch, PCIe Card or cable is broken. (eth0)

 

In the screenshot you can see my current settings, which don't work. How do I set this up correctly?

 

I would be really happy for your help.

[MAM59]

This won't work ("You can't have it all").

 

Worst of all: your main connection will always be 1G only this way!

 

Forget about the backup, the aggragation alone is already a kind of backup.

 

You need to rearrange the cards so that the 10G ports are eth0 and eth1 (ok, only 0 is really important, but for better reading you should rename the 2nd port too I think).

Dont use the mainboard port at all, it will only give you routing problems.

 

(BTW: "aggragation" sounds good, don't be too disappointed seeing that in reality it does not really speed up anything, it sometimes even gives stress only to some devices)

I don't know this switch, so I can't say if it does the aggragation "clever" or "dumb". Try it out and prepare to deactivate it again if it turns out not helpful.

 

(the thing is "aggragation" itself only means that some ports are grouped together, but not in which way. That's why UNRAID offers so many different options for aggragations/bonding/trunking (pick your favorite naming scheme))

 

Edited October 2, 2023 by MAM59

[timomatrixer]

Thank you very much for reply.

 

I remapped the eth0 to SFP(Port1) und eth1 to SFP(Port2) but if I dismount the cable from motherboard eth, it isn't working at all.

ubiquiti USW Aggregation say's now aggregation between this two ports is working but with no linkspeed.

 

any suggestions?

[MAM59]

5 minutes ago, timomatrixer said:

say's now aggregation between this two ports is working but with no linkspeed.

 

As I said, there are many variations of "aggragations", you have currently picked the most likely 802.3ad. But this does not need to be true always.

So, try the other modes (beside backupmode, this does not need any switch support).

 

Also, you have put both cards into seperate bonds, they both need to be in bond0 to work aggregated, try this first. move eth1 to bond0 and pray.

[timomatrixer]

Ok, after I disconnected the cable from Motherboard ethernet port and restarted the USW Aggregation Switch, my 802.3ad connection worked. But in different bonds....

When I tried to see actual interface state with cmd "cat /sys/class/net/bond0/bonding/active_slave" nothing happen. Is there something wrong with my system?

 

I have to set both sfp+ Ports (802.3ad) in the same bond0 right?

Edited October 2, 2023 by timomatrixer

[MAM59]

8 minutes ago, timomatrixer said:

I have to set both sfp+ Ports (802.3ad) in the same bond0 right?

Yep, like I told above already

Bond0 is the only one that should exist. With the members eth0 AND eth1

 

[timomatrixer]

Ok, I understood. But how do I do that? The cat command does not work via console input. What am I doing wrong?

 

Do I have to do the following steps?

https://www.cyberciti.biz/faq/ubuntu-setup-a-bonding-device-and-enslave-two-real-ethernet-devices/

 

Or is there a simple unraid solution?

[MAM59]

8 minutes ago, timomatrixer said:

Or is there a simple unraid solution?

yeah, just use the gui. it is simple

Stop the array, go to network settings, deselect bonding from eth1 first, then

click here, checkmark eth1 and YOU ARE DONE

 

 

 

[timomatrixer]

Oh my g, I misunderstood the interface areas...

I thought "Interface eth0" und "Interface eth1" are physical ports of PCIe card and they have to be linked into an extra bond.

Thank you a lot @MAM59

[DanielPT]

On 10/2/2023 at 8:23 PM, MAM59 said:

Yep, like I told above already

Bond0 is the only one that should exist. With the members eth0 AND eth1

 

Hi @MAM59 You sound like you know what you are doing

 

Im into kind of the same place. I like to make a 10G SFP and a "Acitve-backup" with 1G. Is that even possible?

 

So if my fiber is broken (10G) it will switch over to the 1G cable

 

 

[MAM59]

59 minutes ago, DanielPT said:

I'm into kind of the same place. I like to make a 10G SFP and a "Acitve-backup" with 1G. Is that even possible?

Yes it is.

And your planned usage is also the only legal one for the "active backup" type.

But it only makes sense if you connect the two lines to two different switches too because it is more likely that a switch fails as the dog eats up the fiber cable.

So usually it does not help much, it even may cover an overseen error before somebody notices that things are running much slower as before.

 

I did not see your post before, sorry.

My guess is that you need to wait a little longer. Active Backup uses the MACs of the real cards, so if you change interfaces, ARP caches on the other devices become invalid and need to time out before the change can be recognized. This can take up to an hour. Be patient 🙂

(and yeah, this annoying feature renders the idea of active backups rather useless. It cannot be done without losing connection for a period of time)

Also all running connections are cut off too by the switch. So there will be many complains and so on. There is no smooth transition.

 

 

Edited June 28 by MAM59

[DanielPT]

On 6/28/2024 at 6:29 AM, MAM59 said:

Yes it is.

And your planned usage is also the only legal one for the "active backup" type.

But it only makes sense if you connect the two lines to two different switches too because it is more likely that a switch fails as the dog eats up the fiber cable.

So usually it does not help much, it even may cover an overseen error before somebody notices that things are running much slower as before.

 

I did not see your post before, sorry.

My guess is that you need to wait a little longer. Active Backup uses the MACs of the real cards, so if you change interfaces, ARP caches on the other devices become invalid and need to time out before the change can be recognized. This can take up to an hour. Be patient 🙂

(and yeah, this annoying feature renders the idea of active backups rather useless. It cannot be done without losing connection for a period of time)

Also all running connections are cut off too by the switch. So there will be many complains and so on. There is no smooth transition.

 

HI MAM. 

Meny thanks for the time and good answer !

 

My server is connectet to my UDM SE with 10G.

And i have 2 1G ethernet connectet to my 24p switch.

 

So if the fiber card in the server blows up or fiber cable is broken. I like the idea that a backup line can be configured.

 

Meaby i can get my Unify UDM to send me a notification if the 10G has gone down.

 

But too bad that it cant be done with "almost" no down time