So yesterday was an adventurous day where I probably made more changes at once than I should have.

 

1) Changed firewall:  Untangle -> opnsense w/ zenarmor and surciata

2) Changed the host name of the servers

3) Changed the static dhcp ip for the server

 

I have 2 unraid servers configured very similar, both of them same subnet, same changes to both, etc.  I have direct 10g wire between them as well and they both use the same make/model 10g card (chelsio)

 

1 servers behaved exactly as expected (no issues) with the changes.   Works and accessible just as it was before.

 

The other server, no matter what I do, the interface will not pick up an IP address.  I can hardcode an IP and it takes it, but it doesn't look like the interface is active....and no matter what I do, I can not access it via the GUI except when connected directly to the server.

When I try to ping the gateway, I get the network is unreachable.

I can ssh between the servers via the direct link, but those are static ip's and it sets it up correctly everytime.

I changed the static dhcp entry on the firewall to a different IP in case it was somehow used already.

I tried removing the static dhcp and just have one assigned.

I've tried removing the network.cfg file

 

I've tried updating it with default values using this post:

 

One thing I have noticed that stays consistent is never changes from having a flag of 4099 and never shows RUNNING on br0/eth0.  Comparing it to the other server, it should be 4163 and show RUNNING.

 

Additionally, at the end of the eth0 entry is an error "device interrupt 236", originally it was 140, I don't know if the change is progress.  Comparing to the other server, I do know it shouldn't be there.

Scouring the web, I cannot find any information on the 140 or 236 interrupt's or even a hint at what they might mean.

I did find the flags are decimal versions of some hex values reflecting card settings, but none on how to control/change those values.

 

 

No matter what I do I can't get it working...at this point I haven't the foggiest idea on what to try or look at next.

I'm hoping somebody can shed some light my way...

 

I have attached diagnostics.

 

EDIT:  Forgot to mention, I believe it's when I remove network.cfg and reboot.  It assigns an IP from a different subnet and has all the interfaces bonded.  The interface still isn't active.   

While I will take anything at this point, I don't want any of them bonded, which is the way they both are/were configured when it was

working.

holmes-diagnostics-20240317-1744.zip

Edited March 17, 20242 yr by broncosaddict

Did you try resetting network.cfg on the Unraid USB stick?

/config/network.cfg

You can install unraid to another stick and then copy the network.cfg to your existing install and try setting up the network again?

Here's what mine looks like:

Quote

/boot/config# cat network.cfg 
# Generated settings:
IFNAME[0]="br0"
DHCP_KEEPRESOLV="yes"
DNS_SERVER1="192.168.1.251"
DHCP6_KEEPRESOLV="no"
BRNAME[0]="br0"
BRNICS[0]="eth0"
BRSTP[0]="no"
BRFD[0]="0"
PROTOCOL[0]="ipv4"
USE_DHCP[0]="no"
IPADDR[0]="192.168.1.3"
NETMASK[0]="255.255.255.0"
GATEWAY[0]="192.168.1.254"
METRIC[0]="1"
USE_DHCP6[0]="yes"
MTU[0]="1500"
IFNAME[1]="br1"
BRNAME[1]="br1"
BRNICS[1]="eth1"
BRSTP[1]="no"
BRFD[1]="0"
PROTOCOL[1]="ipv4"
USE_DHCP[1]="yes"
METRIC[1]="2"
MTU[1]="1500"
IFNAME[2]="eth2"
PROTOCOL[2]="ipv4"
METRIC[2]="21"
MTU[2]="1500"
IFNAME[3]="eth3"
PROTOCOL[3]="ipv4"
METRIC[3]="22"
MTU[3]="1500"
SYSNICS="4"

I have two cards, dual-port each.

My eth0 port is the 'main' one, eth1 is not connected, and eth2 and eth3 are passed-through to VM's.

Now, you said they are directly connected, so none are connected to a switch so they are available to other devices in your network?

Or you have dual ports and one port is connected to a switch and the other directly to the other NAS?

 

33 minutes ago, shpitz461 said:

Did you try resetting network.cfg on the Unraid USB stick?

/config/network.cfg

You can install unraid to another stick and then copy the network.cfg to your existing install and try setting up the network again?

Here's what mine looks like:

I have two cards, dual-port each.

My eth0 port is the 'main' one, eth1 is not connected, and eth2 and eth3 are passed-through to VM's.

Now, you said they are directly connected, so none are connected to a switch so they are available to other devices in your network?

Or you have dual ports and one port is connected to a switch and the other directly to the other NAS?

 

I've tried using the default parameters from that note I've referenced, I tried using the same network.cfg as the working server, tried removing it so it would rebuild, all nada.

 

You did have a couple of parameter I hadn't tried, I added them in and gave it a bounce, no joy.  Same as my tests with hardcoding the ip.  

 

Regarding the ports, yes they are both dual port.   Both have one going to a switch and the other going to the other server.  

 

I don't have any reason to suspect it, but I have a feeling something in the firewall may be screwing things up....I just don't know what.  I would also think both servers would be equally affected.  Aside from the dhcp static settings, it's pretty vanilla.  Once I had everything working/functioning I would go back and tighten the screws so to speak.

Currently the rules are let anybody on the LAN see/access everyone else on the LAN.....it's also all one subnet.

 

Here is the network.cfg I just rebooted with:

# Generated settings:
IFNAME[0]="br0"
DHCP_KEEPRESOLV="no"
DHCP6_KEEPRESOLV="no"
DNS_SERVER1="192.168.1.1"
BRNAME[0]="br0"
BRNICS[0]="eth0"
BRSTP[0]="no"
BRFD[0]="0"
PROTOCOL[0]="ipv4"
USE_DHCP[0]="no"
IPADDR[0]="192.168.1.7"
NETMASK[0]="255.255.255.0"
GATEWAY[0]="192.168.1.1"
METRIC[0]="1"
IFNAME[1]="br1"
BRNAME[1]="br1"
BRSTP[1]="no"
BRFD[1]="0"
BRNICS[1]="eth1"
PROTOCOL[1]="ipv4"
USE_DHCP[1]="no"
IPADDR[1]="10.10.10.2"
NETMASK[1]="255.255.255.0"
SYSNICS="2"

 

So all 4 ports are on the same subnet? I think this is bad practice.

Why would you need to directly connect the two on top of connection to the switch?

This is what my ifconfig shows:

Quote

ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.3  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 24:be:05:ce:a0:01  txqueuelen 1000  (Ethernet)
        RX packets 70648881  bytes 11384258916 (10.6 GiB)
        RX errors 0  dropped 2877  overruns 0  frame 0
        TX packets 82263069  bytes 147874216687 (137.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 24:be:05:ce:a0:01  txqueuelen 1000  (Ethernet)
        RX packets 75059578  bytes 12762809025 (11.8 GiB)
        RX errors 0  dropped 752  overruns 0  frame 0
        TX packets 130757839  bytes 151132101358 (140.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I noticed that your eth0 doesn't state RUNNING, mine does.

Is eth0 the port for direct connection between servers?

What happens if you remove the direct cable between servers and just have each one connect via the switch?

14 hours ago, shpitz461 said:

So all 4 ports are on the same subnet? I think this is bad practice.

Why would you need to directly connect the two on top of connection to the switch?

This is what my ifconfig shows:

I noticed that your eth0 doesn't state RUNNING, mine does.

Is eth0 the port for direct connection between servers?

What happens if you remove the direct cable between servers and just have each one connect via the switch?

2 ports on each server.   1 port on each server going to switch, 1 port direct connect to the other server (static IP, different subnet)

 

Why a direct connection?  Lots of data moving back and forth between them, keeps the traffic off the switch, is a good deal faster.....plus, I had nothing else to do with the ports 🙂 

 

eth1 is the direct connect with the other server, and yes, I noticed eth0 was missing the "RUNNING" piece (I posted a screenshot and talked about it in the first post)

 

This next question actually didn't have anything to do with the problem, but it put me in a position to see the problem!! (I'm such an idiot)

 

What happens when I remove the direct connect? 

 

I hadn't even thought to try that since I wasn't having an issue with that port, so I was going to give it a shot, I had nothing to lose.

I went to the server and traced both cables coming out to make sure I was going to pull the right one.  I had gone over the cables many, many times over the last couple of days, verifying where they were going and reseating the connections on both ends....but it never sunk in and clicked.

It wasn't going into the switch, it was going directly to the spare port on my opnsense server.  A few months ago, I moved it from the switch to the spare port.  It had been in the switch for a couple of years, so it was ingrained in my head that it was connected to the switch regardless of what my eyes were telling me. 

 

When I was running untangle I had bridged the interfaces so it was all the same network/subnet.   When I moved to opnsense on Saturday, I completely forgot about the creating the bridge....even though I had a screenshot of the old setup showing the bridge. 

Not only did I not bridge the interface, but I also disabled it.  So there was no way the server was going to be talking to anybody from that port.

 

Once I recreated the bridge, everything worked as expected.   I just had to go through and undo all the changes in unraid I made during troubleshooting.

 

So long story, short, Thanks for the nudge!!....I was finally able to sleep peacefully 🙂

Haha,nice fix!