Can't get Asus 10gbit NIC card to work

[BartDG]

Hi all,

 

Last week I've upgraded my Ubiquiti Unifi with a small 10Gbit switch.  I've also bought 10Gbit cards for my Windows PC, Unraid box and Qnap NAS. (an Asus ASUS XG-C100C for both the Windows machine and Unraid box, and a Qnap QXG-10G1T for the Qnap NAS - both cards are using the same chip, I just didn't want any compatibility issues, so I put a "real Qnap card" into the Qnap NAS)

 

Now, the Asus card in my Windows PC works fine, and also the Qnap card in the NAS.  I'm getting real 10Gbit speeds, awesome to see! 

 

But in the Unraid system, it doesn't seem to work.  I swapped out the card in my Windows PC, just to make sure I don't have a bad card.  I don't. In the Windows PC, both Asus cards work fine.

 

In my Unraid box, the lights of the 10Gbit NIC don't start blinking when I connect an ethernet cable, which I think is very weird.  It's like the card hasn't been turned on.  On the other hand, when I look in Unraid system devices, I can see the Aquantia corp card mentioned there. 

 

Under "network settings", I can see the card there also, under "eth1". 

But when I try to change any of the settings, like eg. adding a static IP address, the network totally locks up and my Unraid box becomes totally unreachable.  I can't even ping to it anymore, not the the 10Gbit card, but also not to the 1 Gbit Intel NIC I've been using for years.  The only way to solve this is to login to safe boot GUI and change it back again.

 

Am I missing something obvious here?  I seems to me like I am, but I have no clue what it could be.

 

Or could it be that the motherboard of my Unraid system is too old?  It is a pretty old setup.  The motherboard is an Asus P5E-VM DO , using a dual core Intel CPU.  It's only got one PCI-express slot, and maybe 10Gbit is asking too much from this board?

 

I'm using the most recent Unraid version (6.9.2) with all plugins updated to their latest versions.

 

I'll add a couple of screenshots from my network settings screen.

 

Thanks in advance!

Edited May 8, 2021 by BartDG

[SimonF]

Please  post diagnostics

[tjb_altf4]

Make sure you either disable your old NIC, or ensure the 2 NICs are on different subnets.

If the 10GbE is now your primary, or you have swapped around IP/subnets etc, make sure you also power cycle your network equipment.

[Zonediver]

1 hour ago, BartDG said:

But when I try to change any of the settings, like eg. adding a static IP address, the network totally locks up and my Unraid box becomes totally unreachable.

 

Using the Asus-NICs (unraid and Win10 over a direct-link) since Oct. 2019 without issues.

Beware: You need "different Subnets" for eth0 "and" eth1.

 

Example:

192.168.11.xxx for eth0 and

192.168.12.xxx for eth1 - Subnet eth0 is 11 and eth1 is 12

or

192.168.1.xxx for eth0

10.10.10.xxx for eth1

 

In your case, eth1 has no ip (???) - why?

 

Edited May 8, 2021 by Zonediver

[BartDG]

Could it be so simple as to disable my gigabit ethernet NIC in the BIOS?  So there's only the 10Gbit card left?

 

Edit : it doesn't.  As said the lights on the 10gbe card don't even blink, so in essence it just sits there and does not grab an IP address.

 

I'll also add Diagnostics to this post.

tower-diagnostics-20210508-1344.zip

Edited May 8, 2021 by BartDG
added info

[Zonediver]

9 minutes ago, BartDG said:

Could it be so simple as to disable my gigabit ethernet NIC in the BIOS?  So there's only the 10Gbit card left?

 

You need "TWO" adress-ranges - one for eth0 and one for eth1 !

But the question is: Do you need "two" networks?

If you say "no", then deactivate the NIC in the BIOS - if "yes", use two subnets.

 

FYI: The Asus 10GBit-NIC has "no" WOL - this function is not used by Asus (but the chip has it)

So if you like to wakeup your server from sleep, you must use either a second nic (onboard) or the QNAP-NIC for unraid, because

the NICs from QNAP have WOL.

Edited May 8, 2021 by Zonediver

[BartDG]

That's good info, thanks!

 

I really don't need two networks, so I can disable the 1 Gigabit NIC.  (which I did, but it doesn't make any difference - the lights on the 10 gbit card don't blink, so it just sits there and does nothing)

Thanks also for the info on WOL.  I don't really need WOL, but it's good to know.

 

I'll keep searching, but I'm starting to fear the motherboard I'm trying this on is simply too old.  I've just checked and the BIOS is from 2009. 🙈

[SimonF]

Card is not seeing a link. Is the switch port enabled?

 

Settings for eth1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
                            2500baseT/Full
                            5000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
                            2500baseT/Full
                            5000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    MDI-X: Unknown
    Supports Wake-on: pg
    Wake-on: g
        Current message level: 0x00000005 (5)
                               drv link
    Link detected: no

driver: atlantic
version: 5.10.28-Unraid
firmware-version: 3.1.88
expansion-rom-version: 
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes

 

These are the DHCP messages from the Log.

 

May  8 13:14:14 Tower dhcpcd[1425]: eth1: waiting for carrier
May  8 13:14:14 Tower kernel: 8021q: adding VLAN 0 to HW filter on device eth1
May  8 13:15:14 Tower dhcpcd[1425]: received SIGTERM, stopping
May  8 13:15:14 Tower dhcpcd[1425]: eth1: removing interface
May  8 13:15:14 Tower dhcpcd[1425]: dhcpcd exited
May  8 13:15:14 Tower rc.inet1: can't obtain IP address, continue polling in background on interface eth1
May  8 13:15:14 Tower rc.inet1: dhcpcd -b -q -t 10 -h Tower -C resolv.conf -4 eth1
May  8 13:15:14 Tower dhcpcd[1450]: forked to background, child pid 1461
May  8 13:15:14 Tower rc.inet1: ip link set eth1 up
May  8 13:15:14 Tower dhcpcd[1461]: eth1: waiting for carrier

Edited May 8, 2021 by SimonF

[BartDG]

Yes, the port is enabled.  I simply switch the same cable that's in my 1 Gigabit NIC to the 10 Gbit NIC.  So I'm sure it works, because it works perfectly with the 1 Gbe.

The switch is also a unmanaged switch, so there's nothing to set up.  That, and it works perfectly with my Windows PC and Qnap NAS.

Edited May 8, 2021 by BartDG

[SimonF]

It does look like the card is not being initalised correctly

 

@Zonediverdo you see any additional entries in your log than the ones above.

 

Have you tried adding ip on a different subnet just to see if it initialises?

 

I.e. 192.168.2.1 255.255.255.0 no gateway

[Zonediver]

22 minutes ago, SimonF said:

do you see any additional entries in your log than the ones above.

 

In my case, the NIC has a static ip and all what i see is this:

 

May  8 15:43:37 Horus kernel: atlantic 0000:05:00.0 eth1: atlantic: link change old 0 new 10000

 

That happens when i start my PC and both NICs are online.

But... some settings are different...

 

Port eth1 Information

Supported ports:[ TP ]

Supported link modes:100baseT/Full

1000baseT/Full

10000baseT/Full

2500baseT/Full

5000baseT/Full

Supported pause frame use:Symmetric Receive-only

Supports auto-negotiation:Yes

Supported FEC modes:Not reported

Advertised link modes:100baseT/Full

1000baseT/Full

10000baseT/Full

2500baseT/Full

5000baseT/Full

Advertised pause frame use:Symmetric

Advertised auto-negotiation:Yes

Advertised FEC modes:Not reported

Speed:10000Mb/s

Duplex:Full

Auto-negotiation:on

Port:Twisted Pair

PHYAD:

Transceiver:internal

MDI-X:Unknown

Supports Wake-on:pg

Wake-on:g

Current message level:0x00000005 (5)

drv link

Link detected:yes

MTU size:1500 bytes

 

Edited May 8, 2021 by Zonediver

[SimonF]

@BartDG Would you know the card firmware level?

 

Driver hasnt changed since March 2020 so dont think that is issue

Edited May 8, 2021 by SimonF

[BartDG]

27 minutes ago, SimonF said:

@BartDG Would you know the card firmware level?

Where can I see this?

[BartDG]

On another note, I've also noticed that TP-link now also makes such 10Gbit cards.  Are those more recent in their firmware?  Maybe also support WOL etc?  (don't need it right now, but you never know...)

[SimonF]

28 minutes ago, BartDG said:

Where can I see this?

Sorry it was in diags 3.1.88 which looks latest on site from March 21. Maybe current unraid driver doesnt support that fw.

 

 

Edited May 8, 2021 by SimonF

[BartDG]

No problem.  I've just switched  both cards to my Windows machine, but indeed, both run 3.1.88, so no newer firmware available.

Edited May 8, 2021 by BartDG

[SimonF]

1 hour ago, SimonF said:

Have you tried adding ip on a different subnet just to see if it initialises?

 

I.e. 192.168.2.1 255.255.255.0 no gateway

Did you try putting ip on nic?

[BartDG]

I'll try that tomorrow, got to go to work now. Thanks for the help so far!

[BartDG]

On 5/8/2021 at 6:24 PM, SimonF said:

Did you try putting ip on nic?

Sorry for the late reply, didn't get around to it yesterday.

 

I tried your suggestion, but nothing happens.  When setting this as a static address and switching the cable, the lights of the NIC don't start to blink.  It's like the card is turned off.

[SimonF]

29 minutes ago, BartDG said:

Sorry for the late reply, didn't get around to it yesterday.

 

I tried your suggestion, but nothing happens.  When setting this as a static address and switching the cable, the lights of the NIC don't start to blink.  It's like the card is turned off.

I have not been able to find out exactly the PCIe X16 slot version, but I think it may be only v1. Not sure if the NIC will go that low. Have you another PCIe card you can test in the slot?

[BartDG]

No, I don't.  But you may be right about that.  This board's only got two PCIexpress slots, one x16 and one x1.  All the rest are old school PCI slots.  That's saying something.

 

I think the time has come to upgrade my unRaid box. 😁