Network Card Assignment Woes


Recommended Posts

Somewhere I wrote a long list of all the problems I've experienced with Unraid's network management GUI, but I can't find it right now, so I'm just starting a smaller thread here, with one main specific issue.

 

I have an onboard NIC, which refuses to be eth0 by default.  It's extremely frustrating when you want to pass through your other NIC for latency reasons, or otherwise, want to remote manage your machine and for some reason when clicking apply on any changes, requires you to go directly to the unraid machine and log in with GUI mode as it loses connection.  Of course, I've been told that this is my fault (in the other thread) and it's my configuration issue.  Anything's possible I suppose - maybe you can help me figure it out.

 

So, to reset I delete the /boot/config/network.cfg and reboot

 

Of course all cards are then default assigned to a bond, which I don't want, so I turn bonding off and click apply.

 

At this point I have:

 

Eth0 - Assigned by the system to one of the dual port add on NIC ports

Eth1 - Assigned by the system to the onboard NIC) (From an Asus X399-A)

Eth2 - Nowhere to be seen, don't know why

 

In the interface rules, there is only one Mac address listed, which is the onboard NIC and which is assigned to ETH1.

 

I have today tried a different NIC (a dual port 10G rather than 1G, both by intel - same symptoms) 

 

My earlier, rather frustrated post started here after spending hours trying to use various suggestions to fix the issue.  The poster on that thread I recall told me it was simple and that I can choose it via the MAC address in the interface rules, however there is only one Mac address listed.

 

My expectation is that the GUI would see all ports and enable me to choose which NIC is default, which address is on which physical port and split out the dual ports on the add on card.  None of which seems to happen for me.

 

Something that confused me for a while - the interface description field isn't a physical interface description, rather a virtual interface description and therefore changes depending on configuration.  i.e. if I name the interfaces  internal / external so I don't have to look up Mac addresses all the time, the card can change depending on various factors including if I were to include eth0 or eth1 in it and if I were to choose a bond.

 

I have attached diagnostics, however bear in mind I can only do this while the interfaces are bonded, another nice little quirk is that while the card is in, neither interface seems to operate stand alone.  

 

And I'm again experiencing similar symptoms to my other post where:

 

1 - A reboot of the server is required (possibly only when changing from between bonding and non-bonding modes) to apply network changes, or possibly just because I can't get any communication over either interface with the card in and seperate IP addresses on the cards

2 - You can't make changes except on the local machine as you lose connection and can't restart the box remotely (at least for me and again possibly because I can't communicate with the external card in and individual addressing).

 

Highly frustrating.

 

And so I must run a bond between a 1G and 10G card, if the card is in - which is certainly not desirable as I'd like to use the 10G card for other things.

 

Any (helpful) thoughts?  In 25+ years of IT work, I don't recall ever having such issues with NIC assignment.  The only other slightly helpful thing I can think of was I had identified that on the dual 1G card, Unraid was assigning a driver that's explicitly stated by Intel to be for single port cards only - there could be something there I suppose.

 

And another thing I just discovered, communication IS happening over the onboard NIC when bonded and ironically (at least with the 10G card in) unplugging the onboard NIC stops all communication, even though the external Nic is part of the bond.

obi-wan-diagnostics-20190617-0203.zip

Link to comment

Check bios for network preference options ( prob none) and disable any network boot options if you have any

 

failing that, assuming you’re on 6.7.0 (on my phone and couldn’t access diag zip] and with the understanding that the pci nic is for pass through, then bind it per the new method outlined in the release thread and see if that forces the onboard into etho

Link to comment

Thanks, yes BIOS already checked and I did bind it on the old card (but not this one) - and it was still exhibiting odd behaviour to be honest.  My other thread was an attempt to list all the things that didn't work and address / make an enhancement request.  If we accept that this is not working ideally, I would ultimately like to help fix it properly so that the GUI can be used like it should be used, rather than hack it if you know what I mean.  I would have thought others might have had these issues too given the very common motherboard and NIC.

Link to comment

Have you tried deleting the network config, and then reboot with just onboard? If not, do that, make sure the array starts, then power down and add the other nic back in. It “should” add them after the onboard in eth numbering. 

 

I can take a look at diags  in the am of someone else doesn’t beat me to a solution.

 

 

Link to comment

Actually, I don't recall, but I was just wondering if I should do that.  I did delete the config, but not with the card out.  It's a good thought.  I still wish the GUI worked properly though, but this might go toward showing that it doesn't.  Also it probably doesn't account for why only a single port is showing up, though it could also reset from being taken out or something.  I did of course swap the card this morning, but chances are the Intel 10G and 1G use the same driver.

Link to comment
8 hours ago, Marshalleq said:

Actually, I don't recall, but I was just wondering if I should do that.

you should. also I spent some time with your logs:

 

 

I can see where the onboard changes during boot:

 

Jun 17 13:58:00 Obi-Wan kernel: igb 0000:05:00.0 eth1: renamed from eth0

 

I've never encountered this but just search for " eth1: renamed from eth0" reveals fixes and pinpoints the problem on the kernel, so not an unRaid problem. (ex: https://www.centos.org/forums/viewtopic.php?t=49338)

 

 

additionally, loading the out of tree driver for the 10gbe may cause network issues that are undesirable, including interface assignment (though I doubt it, but it is worth mentioning.) Yes, numerous others are running the same out of tree driver, but most likely using the 10gbe card as primary, which would have meant this issue would have gone unnoticed. 

ixgbe: loading out-of-tree module taints kernel.

 

but back to the issue at hand, I would recommend what I did before and delete the network.cfg file, power down, remove the ethernet nic, boot up, confirm all is good, power down, add card in, reboot, then see what happens. 

 

Normally the web gui can make interface assignment changes with no issue, but this may be a side case....

Link to comment

So done the card swap.  Actually rebooted a few times without the card in and without a config loaded to make sure.  So here's the thing, without an external card and only a single onboard Nic in the computer, no network configuration file loaded it starts with eth0 and eth1 bonded (even though there is not an eth0 and an eth1).  Going through the syslog, to my reading, eth0 is being renamed to eth1 and eth1 is considered to be in a down state.  Go figure.  Have updated the latest config in case you can see something I didn't.

obi-wan-diagnostics-20190617-1943.zip

Link to comment
2 hours ago, Marshalleq said:

So done the card swap.  Actually rebooted a few times without the card in and without a config loaded to make sure.  So here's the thing, without an external card and only a single onboard Nic in the computer, no network configuration file loaded it starts with eth0 and eth1 bonded (even though there is not an eth0 and an eth1).  Going through the syslog, to my reading, eth0 is being renamed to eth1 and eth1 is considered to be in a down state.  Go figure.  Have updated the latest config in case you can see something I didn't.

obi-wan-diagnostics-20190617-1943.zip 163.38 kB · 0 downloads

 

OK, so maybe try this:

 

with no network nic in

delete network.cfg again

make a backup copy of network-rules.cfg, like on your desktop

then modify the network-rules on the flash to the following (only modification is changing NAME="eth1" to "eth0"

 

# PCI device 0x8086:0x1539 (igb)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="10:7b:44:92:78:83", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

 

save

reboot and report back

 

My thought is that if it was initially booted with the nic card, then the onboard was somehow assigned as eth1 and for some reason is not being changed later when it's the only ethernet port available. And the forced change/rename in the network-rules causes it to make a bond with a now nonexistent eth0.

 

and if that doesn't work, delete the network rules-cfg all together and reboot (again with no nic)

Link to comment

OK, just tried it.  Stopped array.  Deleted config.  Renamed eth1 to eth0 in the network rules.  Rebooted.  Now only have eth0 which is good although it said the eth0 cable was unplugged for some reason.  Configured static IP again.  Rebooted again.  Now it's working.  So this 'seems' to be good.  Now doing an additional reboot to see if it sticks, if it does will add the additional card back in to see what happens.

Link to comment

So that worked, I now have eth0, eth1 and eth2.

 

Thanks for your help.  Other thoughts, which should probably be in a new thread include, "Questioning the wisdom of bonding a 1G and 10G interface together", "Why the unraid GUI has such a limited capability for network configuration compared to e.g. a QNAP NAS", "Why it requires so many reboots", "Why it loses connection and requires local console access".  But I'll leave those for a time when I can bring together the examples.  Perhaps some of it was caused by what you've just helped me solve.

 

Thanks again for your help!

Link to comment
5 hours ago, Marshalleq said:

Why the unraid GUI has such a limited capability for network configuration

Assigning interfaces is possible from the GUI (see Interface Rules).

5 hours ago, Marshalleq said:

Why it requires so many reboots

It shouldn't. With a correct procedure a single reboot is required.

5 hours ago, Marshalleq said:

Why it loses connection and requires local console access

When tossing interfaces, it requires to move the physical cable

Link to comment

"Assigning interfaces is possible from the GUI (see Interface Rules)." - yet it's still very limited and clearly didn't work for me.

 

"It shouldn't. With a correct procedure a single reboot is required". - Maybe I should have changed to 'why it requires -any- reboots' for a network change.  It shouldn't be so hard.

 

"When tossing interfaces, it requires to move the physical cable" -  All cards had a cable (yes 3 physical cables), so I don't think this is relevant (assuming I am correctly interpreting what 'tossing interfaces' means).

 

Basically I'm just trying to say, in this day and age, and with Unraid having so much reliance around docker and VM's networking we can probably do better (and probably should do better), via GUI.  Unraid is awesome, it would be more awesome with some semi-recent GUI network config tool.   Right now it's very very basic.  Attached the QNAP one below, just cause it's what I know about, being that QNAP is on linux, basic and based around open source community.  Good to think about, in terms of where we could be in the future, not where we must be, just where we could aim.  Amongst the NAS's QNAP does have an edge when it comes to customer oriented features so I tend to look toward them and ask how and why.  Why should we suppress thinking forward?

 

image.thumb.png.e6347e697021300772f99c9e1b93f4fc.png

Edited by Marshalleq
Clarity.
Link to comment

The big difference I can see is that QNAP NAS's are hardware with defined and clear cut w/ regards to the hardware like the number of network interfaces and possible roles of the interfaces. Also, I think most commercial NAS have a small reset button in case you bugger up the network config.

We don't have that in Unraid, which leads to the WebUI having to figure out just what is on the machine and work with it seems to get from the OS, and if it makes a mistake, there's no magic reset network config button. You either reconfigure locally (Console or GUI mode), or you unplug the USB, pop in another PC and fix/reset the config files.
Unraid has to make some general assumptions that would work for everybody out of the box, hence the initial bond all network interfaces together upon startup so that the network would work out of the box regardless of how many devices there are or which one is eth0 and if that interface has an actual cable plugged in. Unraid has no idea which of your on-board or addon nic would become eth0 and neither would you unless you've used the PC/server on a similar generation Linux OS before already. The fact that it works for a lot of people out of the box or with minimal difficulty is already a great step forward and it will probably get better, but lets face it we are not there yet. Last I checked the similar competition FreeNAS, or OMV do not have such a sparkling network configuration UI either.

One should also remember that QNAP spends a lot for the development and support, and makes a user pay for it, by being way more expensive than a similarly specced NAS made out of off the shelf consumer or even server parts.

I'm not knocking on the way you think, just pointing out that features tend to cost money due to development and testing, and you would have been shocked at how we got to the current networking configuration in the last 3 - 4 years - we only got custom docker IPs around 2 years ago, and we had back and forth with the community and the devs on which way to go and how to go. The current system is amazing and flexible, though not super user friendly.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.