802.3ad Bonding - Private macs being generated by Docker.

[DeekB]

Hi everyone,

 

New UnRaid user here currently going through the trial process to see if it is a suitable solution for me.

 

I am moving over from Hyper V where I was running a few VM's to run various services from my home for home automation, I am from a windows background and have limited experience with Linux, I was running a Linux server with some docker containers but I am no expert, I found unraid while looking for something to help me manage docker due to my lack of experience but happy to read and learn.

 

My plan is to  move away from Hyper V and consolidate everything into unraid.

I am running x2 HP Gn8 Micro servers and the spec consists of:

 

Intel® Xeon® CPU E31265L @ 2.40GHz (Plan to upgrade to E3 1240 v2 3.4GHZ)

16GB of ECC Ram

X1 240GB Cache drive

X3 2TB Seagate 7200RPM Drives (x2 data & 1 partity)

X1 1TB Crucial SSD for VM\Docker Storage

X2 Onboard Nic Ports (NetXtreme BCM5720 Gigabit Ethernet PCIe)

X4 PCi Nic Ports (82571EB/82571GB Gigabit Ethernet Controller)

 

The second server is currently hosting all my services using hyper V but that server has larger disks and the plan is to migrate everything from that to the server above so I can prepare that server which has slightly larger disk to also come over to Unraid.

 

Asides from above I seem to be having an issue when trying to configure the network sides of unraid, previously I had LACP set up on the microserver and it was all working fine prior to move which tells me the switch and hardware works ok.

 

I have tried bonding the 2 onboard nics using mode 4 but every time I do this I lose connectivity to the server and need to then connect manually via the ILO or keyboard and mouse to then revert the network.cfg from mode 4 back to mode 1 before the server will come back on the network.

 

So I then went to try bonding 2 other nics from my 4 port card which are not being used for the connection to the UnRaid OS and anytime I set 802.3ad I seem to get a 169 IP on that bond, I can however see on the switch end (A Netgear GS728TPv2) that the LACP link is up but as the IP is not correct I never get a connection, I then set a static IP but the server still stayed off the network and was not pinging, I have tried other modes like 0 & 6 and I also get a good IP from those its just LACP that doesnt seem to work.

 

I have read many forum posts with people having the same issue too but cannot seem to find where I am going wrong with this as the config works on hyper v.

Edited May 9, 2022 by DeekB

[DeekB]

Are there any UnRaid networking guru's around?

 

I managed to get this working after hours of messing around but for some reason I need to set a static IP

(no DHCP lease is ever attained by the server when using 802.3ad)

 

Above is a workable solution for me but I now have another networking related problem I need to overcome.

 

I use a Firewalla Gold as a home router, this has a feature that will block any new devices on the network until approved, I keep seeing a strange issue, I get requests to unblock the UnRaid Server on my network as the device is being registered using a private mac (Which seems to be changing) I think this may be expected behaviour of bonding the interface.

 

Was wondering if there is any way to statically set the mac address so that it never recycles or uses different from what has been set up?

Edited May 8, 2022 by DeekB

[DeekB]

After a bit more playing around I am starting to think these private mac addresses are not actually from the unraid O\S but potentially from docker containers.

 

when I was running docker in a linux server VM I managed to use Mac Vlan to map out the IP's and Mac addresses, does unraid have that same functionality?

[JonathanM]

6 hours ago, DeekB said:

I managed to use Mac Vlan to map out the IP's and Mac addresses, does unraid have that same functionality?

Yes, but for some reason it crashes hard on some people's hardware. The 6.10 rc versions have IP Vlan, which seems to solve the mac vlan crashes for those people.

 

You can always roll the dice and see if it works for you.

[DeekB]

7 hours ago, JonathanM said:

Yes, but for some reason it crashes hard on some people's hardware. The 6.10 rc versions have IP Vlan, which seems to solve the mac vlan crashes for those people.

 

You can always roll the dice and see if it works for you.

 

Thank you for the reply, I was looking for a way to upgrade to RC6 but I could not see it (currently running the trial version so may be why)

[ChatNoir]

15 minutes ago, DeekB said:

Thank you for the reply, I was looking for a way to upgrade to RC6 but I could not see it (currently running the trial version so may be why)

It should not be an issue.

Go to Tools, Update and set the branch to Next.

[DeekB]

15 minutes ago, ChatNoir said:

It should not be an issue.

Go to Tools, Update and set the branch to Next.

 

Thanks, I did try that yesterday but nothing was listed under the next branch for me.

 

Current branch:

 

When I select next branch it is showing blank.

 

Edited May 9, 2022 by DeekB

[DeekB]

Seems I was just impatient yesterday as it just takes a minute or 2 to load.

[DeekB]

Thanks everyone, now running the latest version!

[tjb_altf4]

On 5/8/2022 at 4:42 PM, DeekB said:

Was wondering if there is any way to statically set the mac address so that it never recycles or uses different from what has been set up?

docker mac address is generated based on IP, so if you fix the IP you fix the mac address

[DeekB]

5 minutes ago, tjb_altf4 said:

docker mac address is generated based on IP, so if you fix the IP you fix the mac address

Was not aware so thank you will look into that as an option.

[DeekB]

30 minutes ago, tjb_altf4 said:

docker mac address is generated based on IP, so if you fix the IP you fix the mac address

 

Quick question around this, does this mean every docker container needs to have its own IP?

 

Or is there a way to set docker to have its own IP, I have looked but don't see a way to set the docker IP, looks like its pulling its subnet details direct from the network bonds that are set up.

 

As I am new in the set up I only have 2 containers, Unifi controller which has its own IP, and Krusader which is sharing the IP on the host unraid server.

 

Just trying learn the system and to iron any any small issues\niggles before I start moving everything over and growing my set up so sorry for all the questions.

[DeekB]

I was using portainer previously where you can set IP's and macs like so

 

I see there is a portainer community app so just about to give it a try.

[tjb_altf4]

3 hours ago, DeekB said:

 

Quick question around this, does this mean every docker container needs to have its own IP?

 

Or is there a way to set docker to have its own IP, I have looked but don't see a way to set the docker IP, looks like its pulling its subnet details direct from the network bonds that are set up.

 

As I am new in the set up I only have 2 containers, Unifi controller which has its own IP, and Krusader which is sharing the IP on the host unraid server.

 

Just trying learn the system and to iron any any small issues\niggles before I start moving everything over and growing my set up so sorry for all the questions.

It depends on the network you assign your docker to, you can have a mix within a server.

 

To set an IP, the network needs to be a custom one, this might be one you created (docker network create), br0, or if you added vlans something like br0.10

Portainer in your screenshot looks like it is custom network named macvlan.network (of the type macvlan presumably).
The Fixed IP option only shows once a custom network is selected, if you don't add an IP, it will simply use DHCP for the next free IP.


Also note that the docker name can be used for name resolution within that network (when its on a custom network), which is quite handy if you don't want/need to have fixed IP.

Edited May 9, 2022 by tjb_altf4

[DeekB]

On 5/9/2022 at 2:40 PM, tjb_altf4 said:

It depends on the network you assign your docker to, you can have a mix within a server.

 

To set an IP, the network needs to be a custom one, this might be one you created (docker network create), br0, or if you added vlans something like br0.10

Portainer in your screenshot looks like it is custom network named macvlan.network (of the type macvlan presumably).
The Fixed IP option only shows once a custom network is selected, if you don't add an IP, it will simply use DHCP for the next free IP.


Also note that the docker name can be used for name resolution within that network (when its on a custom network), which is quite handy if you don't want/need to have fixed IP.

 Sorry for the delay in reply I spent a bit more time trying to understand what is going on here and I am pretty sure I have a good grasp of it now but not sure how to prevent it or change the behaviour.

 

The macvlan.network was configuration for one of my containers in portainer as part of the mac vlan driver, what I was trying to demonstrate with above is that using the mac vlan driver you can specify what IP address to give your container (Similar to what we have in unraid) this means any time you deploy your container it will always have the same static mac and IP that you have define as per my screenshot above (I guess the same functionality probably exists here in unraid as it is using the macvlan driver I guess) but its probably not opened to the UI for us to set that for each container but I dont see this as being a containe problem but an actual docker networking problem which I will try my best to explain below.

 

The problem I was and I am still having with the private mac addresses, if I start & enable docker it will bind to whatever networks you have told it to, when that happens it is then giving me a private mac, after each restart of docker this mac changes which is causing me lots of issue on my network as I have a fair bit of security set up there which I have had to disable to allow things to function which is not really ideal for me, not to mention every time I get a private mac it shows as a new device in my router as its tracking devices via macs so I end up with a new device on each of my networks that I am bridging into docker.

 

To make thing worse the bonding I had set up didnt seem to be working correctly which was resulting in the network connection being dropped every so often and when that was happening I would get a new private mac registered on the network which tells me its probably every time docker needs to reconnect it is somehow renewing with a different mac,  I removed the bonding so that I now have 1 network assigned to eth0 and another network assigned to eth2 (Both on different VLANS), removed all my docker containers and delete the docker v disk, I then started up docker to be faced with the same issue (A new device being registered on my network under the same hostname but with a new mac) so its down to docker for sure but not sure what can be done to prevent this, not sure why this is happening either as it never happened in my previous docker instance I was running on ubuntu server, I guess there must be a way to make this mac static, that would solve my problems but I cannot see an easy way to change that in the UI.

 

Hoping someone with a bit more unraid experience can help me out.

 

Here are some pictures of the issue I am facing in my router due to the private macs:

 

Edited May 10, 2022 by DeekB

[DeekB]

Here are some screenshots of my network and docker network config.

 

Vlan 1 NIC (No bonding)

 

Vlan 50 Nic (No Bonding)

 

routing table

 

Docker br settings:

 

I believe when docker starts and the network bridges are created is when the private macs are being generated, both ip's were static but in the screenshot vlan 50 is changed to automatic but this bears no relevance on what mac is being generated (As above it was mentioned that a static IP would indicate the mac should be static) but that is not what I am seeing.

[tjb_altf4]

Did you set the network + IP in the container(s) itself?

Those two MACs pictured, one is your server's NIC, the other looks like something custom set (from Portainer?), as that is not a valid/typical MAC.

[tjb_altf4]

Docker generated MACs start with 02:42:ac (or similar), with the rest derived from the IPv4 IP address.

[DeekB]

9 hours ago, tjb_altf4 said:

Docker generated MACs start with 02:42:ac (or similar), with the rest derived from the IPv4 IP address.

Hi after I replied last night I ended up flattening the server and going back to basics as I installs quite a few add ons and stuff and then my docker would not start and kept getting strange errors.

 

I am work today so need to continue with the fresh start later today when I get home and I will post fresh examples if it is still happening after the full reinstall (Will also include diags this time too)

Edited May 11, 2022 by DeekB

[oubeedoub]

Hi @DeekB,

 

I seemingly have/had the exact same issue.

I recently got rid of my ISP's modem/router/switch/access point box and upgraded to a pfsense box. 

I'm very much a novice and not the help you're looking for, but I've seemed to fix it? Temporarily perhaps? Hopefully my experience can help someone diagnose the actual problem. Hopefully, it is the same problem!?

 

For context my networking was setup as follows:

• Unraid eth0 and eth1 on bond network (eth1 had nothing physically plugged into it, I just never bothered turning off bonding);
• bridge network was turned on, which most of my docker containers participated on;
• proxy network for containers I wish to access outside my network (presumably no relevance to the problem);
• custom host network for pihole, which had it's own IP independent of the server (on the same network), I thought this might be the issue since I have 1 device, with two IP's associated with it? My networking skill isn't good enough to know if this would be and issue. I had my pfsense DHCP Server hand out this IP (pihole) as the DNS server, then pihole forwarded back to pfsense (to the DNS Resolver) which then forwarded to Google DNS servers;
• bridge network utilizes Google DNS Servers, as I had "Unraid can talk to host networks" turned off, as I understand this might cause issues, nor do I really need my server to run through pihole;
• physical network is: interwebs -> pfsense box -> network switch (smart one, but utilized as a dumb switch for the time being) -> server plugged into a switch, along with my ip cameras, access point, and a smattering of clients; and,
• only one LAN network in play, no VLAN's. 

I was having the same symptoms as yourself;

In addition to what you've noted, I noted the following with my setup:

1. When I ping my router from the server, I would have 90%-ish packet loss, so it was kind of connecting? Similarly, traceroute -I 8.8.8.8  would sometimes hit the router and out the door, other times returned nothing;
2. My pfsense log was throwing "pfsense kernel: arp: <Server IP> moved from <MAC Address 1> to <MAC Address 2> on igb1", this would constantly switch back and forth between the two MAC's. pfSense has documentation about this here.
3. The MAC Address 1 I recognized as that reported by ifconfig for eth0 and br0 (makes good sense, I think). MAC Address 2, no idea what this was, much confuse.

Actions I took to correct or accidentally correct the issue:

1. I had a custom docker network setup, I noticed this while faffing about trying to solve the problem. I was like "what's this garbage" and deleted it. My pihole proceeded to get booted from it's host network and onto the bridge network. "Ah, that's why I had that..." Well... okay, this should still work anyways. I changed the DHCP Server so it handed out my servers ip. Then cleared the DHCP Lease table. All of my client's seemed happy with the new arrangement. However, I was still getting the log message about the flopping back and forth of this MAC Address. Who is this imposter MAC? My server has no internet still.
2. I turned off the bond network on Unraid.
3. Next, I assigned a static ARP Table entry for my server. So, static IP and Static MAC (bizarre concept to me). So this imposter MAC could no longer hijack what I interpreted as the correct MAC Address. Flushed my DHCP leases again, and my ARP table (aside from the static assignment).
4. Rebooted my server. Voila! Everything is working now. I had rebooted before doing items 1, 2 and 3. So, I'm certain I didn't just need a reboot.

I'm guessing item 3 was the game changer.

 

Today in my pfsense logs I noticed that the imposter MAC got thrown onto another IP after attempting to alter a permanent ARP entry multiple times (haha, take that imposter MAC). I cannot track down if it is some other device on my network that was previously assigned my servers IP and refused to let go, or something coming from within Unraid itself. Log below (imposter MAC is in red):

 

May 10 22:46:54kernelarp: 192.168.2.112 moved from 34:2e:b6:f9:8f:94 to 74:d6:37:1a:eb:3b on igb1

May 10 22:46:50kernelarp: 192.168.2.109 moved from 64:16:66:90:31:17 to 60:a4:b7:b7:60:c2 on igb1

May 10 22:46:46kernelarp: 192.168.2.107 moved from 80:f3:ef:c0:e1:9b to 40:3f:8c:c6:02:f4 on igb1

May 10 22:46:42kernelarp: 192.168.2.106 moved from 74:d6:37:1a:eb:3b to 68:d7:9a:0b:5d:eb on igb1

May 10 22:46:07kernelarp: 68:d7:9a:0b:5d:eb attempts to modify permanent entry for <Server IP> on igb1

May 10 22:46:00kernelarp: 68:d7:9a:0b:5d:eb attempts to modify permanent entry for <Server IP> on igb1

...

 

Hopefully, this helps in some way or another. Someone let me know if they don't think I fixed a single thing and it's still broken. I feel as if a static ARP is a band-aid?

Let me know if you need more information or context or if we're even having the same problem, I'm far too smooth brain to know.

 

 

[DeekB]

Hi

@oubeedoub thanks for the very detailed post above it does help alot.

@tjb_altf4 Thanks for all your advice too I have learned a little here around how it should be working which is more then I knew before.

 

I dont use any of the ISP equipment apart from the modem\router, I am currently running firewalla gold as my router, this is attached to my cable modem\router which is set to modem only mode.

 

The Firewalla gold has LACP set up on 2 ports to a 28 port netgear switch, I have a few Vlans ect on my network for security as well as mac filtering which is how I discovered I had a problem in the first place (I have only been using UnRaid around 1 week so very inexperienced with it) I had to disable the mac filtering for starters previously but its set back up again and working great for the last couple of hours during my testing, I set up bonding from the get go last time and I think that is the route cause of the issues I was having.

 

As per above post I went back to basics, I have come home from work today with a fresh head, I removed the bonding in previous days as per above posts, Today I have set up the same containers and I no longer have the issue I was seeing, I get zero private macs, only macs for the containers where I have specific a IP and these macs do seem to stay static which is what I would expect.

 

The only difference is from the get go I have treated my individual nics as individual nics and I have not tried any bonding, The bonding is also creating private macs as that is how bonding is supposed to work but the mac for the bond should be static imo, what I cannot get my head round is  when I removed the bonding the other day before I flattened the unraid server I was still seeing private macs when I was stopping and starting docker, I dont know a lot about linux but it says to me that there was maybe some bad config on the network stack left over from the bonding but not sure how to prove if that was or was not the case. to mention I did also try nearly all of the bonding modes and suspect this could be what has caused the bad config in the first place and the other issue to mention is anytime I set up bonding DHCP fails to work and I need to go static too so maybe the bonding being done here by unraid doesnt like my hardware.

 

I may just set up the 802.3ad bonding again to see if I see the same issue as I can put static routes into my firewalla router so I may be-able to have the same work around above as you as I had the bonding set up in me previous virtualised environment so no reason why it should not work here but I can live without it if need be.

 

I am just happy it is working now but I will have another go at bonding shortly to see if the same issue re-appears 

[DeekB]

I was reading this and wondering if this is possible with unraid 

 

https://access.redhat.com/solutions/70215

 

Add MACADDR= directive and specify the required MAC address to the /etc/sysconfig/network-scripts/ifcfg-ethX network configuration.

 

Like so:

 

but guessing the file location may be different in UnRaid

Edited May 11, 2022 by DeekB

[DeekB]

So have now set up 802.3AD bonding and everything is still working perfectly so far after an hour or 2 with restarts of the docker containers, service and server itself, it now seems to be rock solid!