I have 5 servers. Each server has a total of 5 connections, 1 on VLAN A which is where the Unraid web interface lives. The other 4 are on VLAN B which are for my VMs. 3 of my 5 servers are affected by this problem. For some reason, I lose network connectivity to the management interface (eth0 on VLAN A). When I log in to the console I can see that the gateway for VLAN B is set as the default route. I use the following command to set the gateway for VLAN A as the default route:

route add default gw 172.27.X.1 eth0

 

After I run this command I regain connectivity to the Unraid web interface and SSH. If I reboot then I lose connectivity again. I checked the routing table and it hasn't changed after the reboot. I can't even ping the gateway as I get a destination host unreachable error. I've triple checked the network settings and the IP, subnet mask and default gateway are all correct. At this point I am thoroughly confused. 

....didn't you set-up a bridge (main interface is br0 and IPs are attached to br0, instead of eth0)...thought this is standard, when using VLANs and VMs.

There was a bridge set up by default but I removed it because I didn't think it was necessary. I'll use bridges for the VM's though. Do I have to have a bridge set up for the web interface to work?

no you don't, but when a bridge is active, IPs are attached to that logical interface, not at the underlying physical ones (eth0, eth1, ...).

How many physical NICs do you have and are you sure that eth0 is not part of a bridge...If there is a hickup of some sort, IPs can get rearranged/reattached otherwise.

Standard is, that all physical NICs are attached to one bridge and VLANs are enabled / configured on that bridge.

Things won't change if physical cables are re-seated this way, I think.

 

Maybe you can provide diagnostics for each situation, when they occur.

I checked and there are currently no bridges. There are a total of 6 physical NICs, only five of those are connected. 

Is there a way to specify which interface the Unraid web interface should listen on?

23 minutes ago, Brad the Beast said:

I checked and there are currently no bridges. There are a total of 6 physical NICs, only five of those are connected. 

please post your diagnostics.zip

Also, can you give some info on how your setup outside unRaid is done, Switch & Router, especially with regard to using VLANs

So this is a university network. There are two networks set up for this lab. Connections are as follows:

Onboard NIC port 1, Unraid web interface/management. Switch port is on VLAN A, untagged. 

10Gbps PCIe NIC port 1, VM 1. Switch port is on VLAN B, untagged.

10Gbps PCIe NIC port 2, VM 2. Switch port is on VLAN B, untagged.

10Gbps PCIe NIC port 3 VM 3. Switch port is on VLAN B, untagged.

10Gbps PCIe NIC port 4, VM 4. Switch port is on VLAN B, untagged.

Switch uplink has VLANs A and B tagged.

The plan is for each VM to have it's own 10Gbps connection. 

undhpcvrlab3-diagnostics-20201105-1250.zip

Edited November 5, 20205 yr by Brad the Beast

...there is a bridge in your config, build from eth2, but is not fully configured (10G NIC #1)

currently eth1 is not present.

Also eth0 is on a different IP-network than eth3,4 and 5, so basically yes, there are two LAN segments active.

 

The routing table is not part of the diagnostic, though.

Regarding your default route, yes, this is set to the second network, via the gateway of the 10G NICs. That is written in the system log, when the dhcp-addresses get allocated.

But this should only present a problem when accessing the UI from a Browser/PC that is located in a third, different IP-network from your two and the firewall of the router prevents the packets coming in via that route.

 

In order to give a 10GB bandwith for each VM, I'd rather passthrough one 10G-Nic for each VM, so unRaid itself is not multi-homed and each VM will directly receive the IPs via DHCP from your router in VLAN-B

Edited November 5, 20205 yr by Ford Prefect

2 hours ago, Ford Prefect said:

But this should only present a problem when accessing the UI from a Browser/PC that is located in a third, different IP-network from your two and the firewall of the router prevents the packets coming in via that route.

So I still don't understand how I'm unable to ping the gateway of VLAN A from the interface that's on VLAN A.

Also I was troubleshooting one of the other machines. The only connection it has is the VLAN A management connection. All other network connections are unplugged. That machine still has the same issue. As part of my troubleshooting with this machine, I connected my laptop directly to the onboard NIC on the back of the server with all other network connections unplugged. I manually set a 169.254 address for both my laptop and the server and even then I still couldn't access the web interface, SSH or ping. 

Edited November 6, 20205 yr by Brad the Beast

if you configured the NIC with a fixed IP-Address and gateway and can*t ping the gateway from that interface, then maybe the netmask is different on both ends.

What*s the routers config? On eth0 in the diagnostics-file, your netmask is set to 255.255.255.0 if I recall correctly.

For the 10G NICs, DHCP ist defining the gateway and netmask, but maybe that config is also not correctly configured.

 

Also not sure if a zeroconf IP would work...for tests with directly linked hosts, I'd set an IP out of 192.x.x.x/24 manually, then start from there.

2 hours ago, Ford Prefect said:

if you configured the NIC with a fixed IP-Address and gateway and can*t ping the gateway from that interface, then maybe the netmask is different on both ends.

Checked. Netmask is the same on both ends. 

2 hours ago, Ford Prefect said:

Also not sure if a zeroconf IP would work...for tests with directly linked hosts, I'd set an IP out of 192.x.x.x/24 manually, then start from there.

This worked. I've made sure that all other interfaces are down. I then set the static IP that it's supposed to have and plugged the server back into the switch. It worked until I rebooted and now it's not working again. Same issue. Still can't ping the gateway. 

....I actually are out of my wits here.

Again all I can think of, is that ingress and egress traffic is going via different routes for your multi-homed setup.

Last thing maybe is to check routing table via CLI and confirm no double entries are there with different metrics where the lowest metric points to another interface.

If everything looks correct, then the problem is (must be) outside the host...so switch or router configs.

Maybe you can mirror the switch-port and let wireshark analyse it.

Edited November 6, 20205 yr by Ford Prefect

So I managed to figure it out. I had to set all other interfaces to be down (even though they were disconnected) and I had to set the optional metric for the interface I wanted to 1. Everything seems to be working as expected now. Thanks for your help @Ford Prefect!

...glad that you worked that out...just make sure that it stays that way after a reboot.