[SOLVED] Server No Longer Connects to LAN After 6.10.x OS Upgrade (TIFU)

[greaterbeing]

I'm pulling my friggin hair out over here after blue balling myself. I've searched everywhere and just can't find a solution.

 

Earlier this week I upgrade my ole Dell Precision T3500 from 6.9 to OS version 6.10.1. Upon reboot I immediately began facing connectivity issues.

 

Servers ran headless since 2018. Where I'd usually interface with it all through the web GUI, this time around I was completely unable to connect to it at through a local 192.168.1.x address. Pings to the unraid address from my main workstation and other computers on my local network would return "destination host unreachable"

 

So I plugged in a monitor, keyboard and mouse and booted in GUI mode to diagnose the issue.

 

Before I continue, I want to give everyone more background info on my network. The setup between the server and my router that had previously been working up until the upgrade to 6.10.x:

• Unraid had a static ip set to a 192.168.1.x address
• Bonding enabled "active backup(1)"
• Bridging enabled
• Default gateway and DNS servers pointing to my router (NETGEAR Router R8000P)
• There's an address reservation on my router for my server address

 

From unraid I could not ping my router or any other computers on my network. I would get the same error "destination host unreachable".

 

I tried switching to DHCP with no luck. Unraid could not resolve an IP address. 

 

Digging into the issue, I had read somewhere that people with similar issues (not specifically with 6.10.x) resolved it by disabling bonding. From what I can recall, bonding had always been enabled as an "active backup (1)" on my server, despite only having 1 NIC.

 

Disabled bonding and boom! Even when I had unraid set to DHCP it immediately set itself up with its old local ip 192.168.1.x address.

 

Swapped unraid back to its old static IP 

 

Now here's the FU on my part. I'll admit I have a problem when it comes to updates.

If I see a new stable build on an OS/Application I'm running I'll update it.

 

And as many of you know unraid 6.10.2 released yesterday.

 

My dumbass decided at the end of my troubleshooting mess, to update the OS again. 😭

 

Now I'm facing the same issue- except this time bonding had been disabled from before and it wasn't getting me out of jail.

 

Swapping from static to DHCP resolves a 169.x.x.x address. Which I think means unraid is unable to resolve it's own address on my network.

 

Pining to and from the server, I receive the same error "destination host unreachable".

 

I thought maybe my NIC was being blacklisted because it was maybe running broadcom tg3 drivers. However,

lsmod | grep tg3

shows no results.

 

I've tested the LAN cable with a laptop, assuming a bad cable. It works fine.

 

I've ordered a replacement NIC ($10 on ebay) in case that's the issue and the hardware's gone bad.

 

I've deleted the address reservation on my router for the 192.168.1.x address my server had reserved. No luck

 

Aside from what I've tried above- can anyone think of any other fixes or tests I should try?

 

Also- I'd like to be able to upload my config/diag dump file for everyone to take a look at, but I'm not sure how. The OS USB is seemingly unreadable on my windows machine, even with 3rd party data recovery and disk reader tools like Diskinternal's Linux Reader and UFS Explorer. And Without the ability to connect to my network, I can't SSH into the machine to grab them.

 

Any guidance on what I can try next to troubleshoot the issue or info on how I can get my diag files would be greatly appreciated

Edited May 28, 2022 by greaterbeing
solved

[greaterbeing]

After reading this thread, I solved my issue by disabling VT-d in the BIOS. I assumed that I wasn't affected based on on the fact I could not find any tg3 drivers. Sorry for making yet another post on this issue!

 

[JorgeB]

2 minutes ago, greaterbeing said:

I assumed that I wasn't affected based on on the fact I could not find any tg3 drivers

Was just going to post that if the NIC was being blacklisted the driver won't appear, if you don't mind please post the diagnostics just to check your hardware.

[greaterbeing]

13 hours ago, JorgeB said:

Was just going to post that if the NIC was being blacklisted the driver won't appear, if you don't mind please post the diagnostics just to check your hardware.

 

Certainly- 

dellt3500-diagnostics-20220529-0029.zip

[goodyear]

I’m not sure if this is related to 6.10, but I have a dual port mellanox x2 card that is not getting assigned an IP even though I’ve set eth0 up with a static IP.

 

I’ve set it up so that the two SFP+ ports are now eth0 and eth1 in bridged mode, with a static IP assigned to eth0, and the onboard NIC is eth 2 (unplugged).

 

But upon reboot, the server is unreachable via the network, but it boots up fine as seen via a directly attached monitor.

 

When I check via  ifconfig , I can see that br0 has the correct static IP, but eth0, eth1 and eth2 don’t have an IP. I can’t ping the default gateway either.

 

lsmod | grep tg3 doesn’t show anything.

When I restart the network via  /etc/rc.d/rc.inet1  restart  the network comes up with the correct configuration.

 

I’ve solved it by adding the network restart command as a custom script that’s run on first array start, but there is clearly an issue here.

 

Any thoughts on what could cause this behavior?

unraidserver-diagnostics-20220529-0800.zip

Edited May 29, 2022 by goodyear
Uploaded diagnostics

[JorgeB]

4 hours ago, greaterbeing said:

Certainly- 

Thanks, no signs of any corruption issues so far, Dell servers might not be affected, *should* be OK to use vt-d if you need it, if you don't leave it disabled for now.

[JorgeB]

2 hours ago, goodyear said:

I’m not sure if this is related to 6.10

Can't see what the problem is, but it's unrelated to v6.10.2, you should probably start a new thread.

[goodyear]

2 minutes ago, JorgeB said:

Can't see what the problem is, but it's unrelated to v6.10.2, you should probably start a new thread.

Will do!