Eth0 Does Not Exist -- After Reboot

[geeksheikh]

Hello,

 

Nothing in my server has changed. I seem to have had a strange power issue with my UPS last night and the server was off this morning. I booted it back up and got eth0 does not exist and ultimately ipv4 not set and ipv6 not set. This makes it impossible to get a diag bundle to you as I cannot get to the server.

 

No hardware changes at all and everything has been working fine for years. Any ideas how I can diagnose / fix the issue to get eth0 to be found so I can at least get an ip connection to unraid?

 

Any assistance would be much appreciated.

 

Thank you.

[JorgeB]

Please post the diagnostics.

[geeksheikh]

@JorgeB I got the diagnostics bundle off the server. 

 

I also noticed that in the logs it says eth0 is being renamed to eth3. After some more research I looked at the /etc/udev/rules.d/70-persistent-net.rules for udev and saw the lines duplicated. I tried dropping this file and restarting but the same file was created again. I tried several other things too to no avail. Perhaps you can determine the issue from the diagnostics bundle. Thank you very much for your help. I'm totally dead in the water right now.

70-persistent-net.rules pumbaa-diagnostics-20231022-1510.zip

Edited October 22, 2023 by srfnmnk

[geeksheikh]

I also see that in network.cfg the BONDNICS are "eth0 eth1" but it seems that those  have been renamed to eth2 and eth3 somehow. Eth2 and eth3 can also be seen in the ifconfig so I'm guessing this is the cause of the issue. I still am not entirely sure how to fix it or how this would happen but I thought I would share what I am seeing.

[geeksheikh]

After messing around with it for a very long time -- I finally got it back up...somewhat...but eth0 is still not found and the persistent-net.rules are still not showing up correctly. I cannot seem to figure out. I'm limping along with some random new eth2 (which used to be eth1) and a static ip address...

 

I got here by deleting the /boot/config/network.cfg file and rebooting and then just doing whatever it took in UI in settings -> network to get at least 1 interface working (apparently eth2)...

 

Still need to figure out how to get everything back to the way it was and what happened. Appreciate the assistance.

[JorgeB]

network-rules.cfg is not getting created, try deleting/renaming network.cfg and reboot, note that LAN settings will go back to default DHCP.

[geeksheikh]

I am on 6.11.5. I did delete network.cfg and reboot, that's how I got to this stage. It did reset my LAN back to DHCP, correct, but I am still getting eth2 and eth3 and no eth0. I also checked my backups and do not see a /boot/config/network-rules.cfg there historically either (from last week when it was working).

[JorgeB]

Without network-rules.cfg being created multiple NICs won't work correctly, but not sure why it's not getting created.

[geeksheikh]

So I'm clear, to where should it be generated? I don't see it in /boot/config/ and I don't see that it was there in the previous backups I have. 

 

What can I do to help us figure out why it's not getting created and/or create / restore one? The 70-persistent-net.rules is being generated each time but the contents of that file are incorrect.

[JorgeB]

16 minutes ago, srfnmnk said:

to where should it be generated?

/boot/config

 

 

[geeksheikh]

So what are next steps? How can we figure this out?

[JorgeB]

Not sure, try a different NIC or board if possible, that file should get auto created, no way I know of to create it manually.

[geeksheikh]

Please note that I don't have extra boards and nics lying around and even if I did, this board is in a prod server and buried in a rack so popping in a new board / nic is not trivial at all.

 

Furthermore, both NICs are still seen by the mobo and by the OS just not as eth0 (which is a special interface to linux / unraid) hence why I can't seem to just use eth2 / eth3. Any chance we can get some more eyes on this to see if they have a method to troubleshoot it further or resolve the issue?

 

I'm ok with completely resetting the network to default / etc but I just don't know where to go from here without potentially making things worse.

[JorgeB]

Try a new flash drive with a stock Unraid install, no key needed, just to see if it works, and since you are on v6.11.5 try with 6.12.4.

[geeksheikh]

would I need to upgrade anything or just add new flash with 6.12.4 directly? Will it keep all my configs with regard to docker / disks / etc?

Edited October 23, 2023 by srfnmnk

[JorgeB]

First try a new flash drive with a stock v6.12.4, if that works update you install to v6.12.4, if it starts working great, if not it's probably config related, you can try redoing the flash drive and copying just the bare minimum, like the ley array and pool assignments and docker templates.

[geeksheikh]

I will try this but I can't have the server down for extended periods of time. 

 

If I put in the new flash drive with 6.12.4 and it works -- what should I expect? Will all the configs come back or is that where I need to start restoring from backups? Either way how do I know which files to restore from backup? Is it just the array, pool assignments, docker templates or are there other things?

 

Sorry, I just don't have a mirrored test environment to test this on and am trying to figure out how to get back to limping along like I am now in case things don't go right. Thanks.

[JorgeB]

10 hours ago, srfnmnk said:

If I put in the new flash drive with 6.12.4 and it works -- what should I expect?

 

13 hours ago, JorgeB said:

if that works update you install to v6.12.4

 

[geeksheikh]

and if it doesn't work I should just be able to put my old flash drive back in and boot back to the way it is now, right?

[JorgeB]

Yep.

[geeksheikh]

one last question -- If it does "work" with the new flash drive -- would it make sense to just copy over the network-rules.cfg to the original flash drive and then it should work?

[JorgeB]

33 minutes ago, srfnmnk said:

would it make sense to just copy over the network-rules.cfg to the original flash drive and then it should work?

Worth a try.

[geeksheikh]

Great news, I was able to use the new flash drive to boot up and get the network-rules.cfg. After moving that to the /boot/config of the old flash -- the eth0 and eth1 were both recognized and the /etc/udev/rules.d/70-persistent-net.rules were also populated correctly.

 

I still have one file that looks strange, the /etc/rc.d/rc.inet1.conf file still has references to eth2 and eth3. I dropped the file and rebooted and those references still pop up. I'm attaching the file for reference. Any idea why these interfaces are still popping up in here?

 

I'm also attaching a post-fix diagnostics bundle.

 

Thanks again for the support

rc.inet1.conf pumbaa-diagnostics-20231024-1239.zip

[JorgeB]

I don't see the network-rules.cfg file in the diags, but great that it's working, can't really help with rc.inet1.conf, you could try updating to v6.12.2 since there are a lot of network changes, but make a flash drive backup before doing it.

[geeksheikh]

yeah -- been planning on doing upgrade but haven't had time to validate all backups yet. That has become a much higher priority with this issue and am working on the backup validations as we speak. I will upgrade and let you know.

 

RE: missing network-rules.cfg -- that's crazy that it's gone again. I put it there and it did work but I see it's missing now again too. I don't understand why it keeps getting removed. I can add it back but it seems after reboot it's gone again each time. This is very starnge. That said, my udev/rules.d file still looks correct now...but this is very strange.

 

I will attempt the upgrade and see if this resolves it and report back.