Issus with secondary NIC connection on 6.12

[FoxBytes]

So I have my unraid server connected to two different networks through separate NICs; one is just connected to my regular house router and the internet, and the other is connected directly to my other server and my main computer through an SFP+ card. So on my main computer I can connected to it through the ip address assigned by my router, or I can get the faster speeds by connecting through the other ip address through the fiber connection.

Everything has been working perfectly up through 6.12.0-rc6, but then on rc7 the second connection suddenly stopped working. I figured it might be fixed before the release so I just restored back to rc6 and everything was fine again. However, the problem was there again for rc8 and now it appears to still be an issue on the official release.

I can still connect to my server through the router-assigned ip, but I can't get a connection through the other one, which is the one I have all my network drives mapped to.  As far I as I can tell from the dashboard though I can't find any indication that there is something wrong though, and since it was working all the way up through rc6 I'm not quite sure what I need to do in order to get it working again. Any ideas?
 

I hope this isn't duplicated somewhere, I did some searching through the forums but was unable to find anything that seemed similar yet.

Thanks for the help!

[MAM59]

without you telling us more details about those used cards and supplying "diagnostics", nobody will be able to tell you anything.

 

There are drivers that have been left out for 6.12, so you maybe affected. Usually you can find them back in the "Apps" section if you need them.

 

[FoxBytes]

Oh right! Sorry about that! These are the diagnostics files that were downloaded when I reverted back to rc6.

holodeck-diagnostics-20230616-2014.zip

 

As for the NIC, it says its an "Intel 82599EN"? Its this one on amazon: https://a.co/d/433FhSw
 

If it was a driver issue would it give me some sort of error or something? As far as I can tell unraid thinks everything is fine, its just not haha.

Thanks again!

[MAM59]

Hmm... I think the relevant parts are:

Quote

un 16 19:25:59 Holodeck rc.inet1: ip link set br0 up
Jun 16 19:25:59 Holodeck rc.inet1: ip link set eth0 down
Jun 16 19:25:59 Holodeck rc.inet1: ip -4 addr flush dev eth0
Jun 16 19:25:59 Holodeck rc.inet1: ip link set eth0 promisc on master br0 up
Jun 16 19:25:59 Holodeck kernel: br0: port 1(eth0) entered blocking state
Jun 16 19:25:59 Holodeck kernel: br0: port 1(eth0) entered disabled state
Jun 16 19:25:59 Holodeck kernel: device eth0 entered promiscuous mode
Jun 16 19:25:59 Holodeck kernel: br0: port 1(eth0) entered blocking state
Jun 16 19:25:59 Holodeck kernel: br0: port 1(eth0) entered forwarding state
Jun 16 19:25:59 Holodeck kernel: br0: port 1(eth0) entered disabled state
Jun 16 19:26:00 Holodeck rc.inet1: ip -4 addr add 192.168.1.175/255.255.255.0 dev br0
Jun 16 19:26:00 Holodeck rc.inet1: ip link set br0 up
Jun 16 19:26:00 Holodeck rc.inet1: ip link add name br1 type bridge stp_state 0 forward_delay 0 nf_call_iptables 1 nf_call_ip6tables 1 nf_call_arptables 1
Jun 16 19:26:00 Holodeck rc.inet1: ip link set br1 up
Jun 16 19:26:00 Holodeck rc.inet1: ip link set eth1 down
Jun 16 19:26:00 Holodeck rc.inet1: ip -4 addr flush dev eth1
Jun 16 19:26:00 Holodeck rc.inet1: ip link set eth1 promisc on master br1 up
Jun 16 19:26:00 Holodeck kernel: br1: port 1(eth1) entered blocking state
Jun 16 19:26:00 Holodeck kernel: br1: port 1(eth1) entered disabled state
Jun 16 19:26:00 Holodeck kernel: device eth1 entered promiscuous mode
Jun 16 19:26:00 Holodeck kernel: ixgbe 0000:17:00.0: registered PHC device on eth1
Jun 16 19:26:01 Holodeck kernel: ixgbe 0000:17:00.0 eth1: detected SFP+: 5
Jun 16 19:26:01 Holodeck kernel: ixgbe 0000:17:00.0 eth1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Jun 16 19:26:01 Holodeck kernel: br1: port 1(eth1) entered blocking state
Jun 16 19:26:01 Holodeck kernel: br1: port 1(eth1) entered forwarding state
Jun 16 19:26:02 Holodeck rc.inet1: ip -4 addr add 169.254.94.175/255.255.255.0 dev br1
Jun 16 19:26:02 Holodeck rc.inet1: ip link set br1 up
Jun 16 19:26:02 Holodeck rc.inet1: ip -4 route add default via 192.168.1.1 dev br0

So there are 2 cards, one 1Gbe (Intel 1000) card getting the address 192.168.1.175 and one 10Gbe (Intel x520) getting no address at all, defaulting to 169.254.94.175 (more or less random within the 169.254 range).

The default route points to a device (presumingly the router) 192.168.1.1 which is located on the 1Gbe LAN.

No route is set to 169.254 (besides the implied "I am member of this, so I can reach others with this address).

 

So far, all fine and ok. But now we come to the

9 hours ago, FoxBytes said:

but I can't get a connection through the other one, which is the one I have all my network drives mapped to.

Now we have to take a look at your "main machine". Does it also have 2 LAN cards with one going to the router and the other going to the SFP cage of the UNRAID box? If yes, you should be able to "reach" unraid under 169.254.94.175 directly.

 

Otherwise I need more details (aka almost the same diagnostics of the network settings) from that machine too)

 

BTW: your setup is not really clever, it is very error-prone (as you have might have noticed already by now). I would strongly recommend straighten it up by buying a 10G switch and attaching all devices to that single switch (not using the 1Gbe Card for unraid anymore).

 

[FoxBytes]

Thank you for the help! Sorry for the late reply. Yes, sorry, 169.254.94.175 is the address I gave it specifically, and the address I referred to by "the other one" haha. The other two computers on that network at similarly addressed.

Also yes actually both of the other computers have 2 NICs each, all of them are 10Gb cards that are linked together, and then their other cards are all also linked together through the main 1Gb lan with their other cards.

The two other machines on the 10Gb network are a Windows 10 and  TrueNAS scale. I'm sure I can figure out the truenas diagnostics but is there a preferred way of getting network diagnostics from a Windows install? As far as I can tell though its only the Unraid server that's not connecting, I can still connect Truenas to my windows machine fine.

And yeahh I definitely get that its not ideal haha, but I do like it for a couple reasons. I do actually have a 10Gb switch too, but its SFP/SFP+ only so I'd have to get some adapters to loop in the rest of my network. Also this way if its connecting through the 169.254 IP address, Its easy to be 100% sure at just a glance whether somethings going through at 10Gb/s or not, and I don't have to worry that its getting bottlenecked somewhere at my 1Gb router haha. I mean I'm sure there's another, better way to do it but its never given me much trouble until now. But I mean if you think it could help, I could try hooking them all into the same network? If its something wrong in between the NIC and unraid though I'd expect that nothing would even make it that far out, but I really don't know haha.

Now that I think about it the 10Gb switch does have routerOS though, which I do not know how to use very well haha, but I might be able to at least find out if its showing up there or something. I'll take a look at it!

Thanks again for your help! I really appreciate it!

[MAM59]

19 minutes ago, FoxBytes said:

and I don't have to worry that its getting bottlenecked somewhere at my 1Gb router

This cannot happen, the router could only "occupy" 1G, just 10% of the rest of the LAN.

Get some SFP modules (for shorter ranges buy "DAC" modules, they are cheap and fixed on both ends). Stay away from Twisted Pair Modules if possible, very expensive, error prone and a real energy / temperature hog. DAC or fiber are the way to go.

Also, this way, the client computers will automatically always talk 10G to Unraid, only packets for "the outside world" will be slowed down to 1G.

 

Drop your 1G connections completely and forget the 169.254... addresses.

19 minutes ago, FoxBytes said:

here a preferred way of getting network diagnostics from a Windows install?

from a command window issue "ipconfig /all", that should be sufficient.

(But after your last explanations, its not needed anymore. Your LAN is more or less an endless loop instead of a straight line)

 

19 minutes ago, FoxBytes said:

If its something wrong in between the NIC and unraid though I'd expect that nothing would even make it that far out, but I really don't know haha.

I guess it will be the loop thing. If you give a computer to chose between two connections to the same destination (the address does not really matter here, the upper layer "SMB" uses names internally, and even if you start a connection to a certain IP address, the answer may be send with the Hostname instead and followup packets may get rerouted. Thats a very dangerous and situation of uncertainty, and should be avoided at all costs)

So you might access 169.254.4.222 and get back the link for \\FILESERVER. And if you ask for \\FILESERVER next time, you get back 192.168.1.33 or so

 

Edited June 20, 2023 by MAM59