From: unRAID Server Release 6.2.0-rc5 - Mellanox config

[ajgoyt]

Upgraded to rc5 and my Mellanox connectx 10gb card says inactive in the network settings, I got to the network settings by hooking up to my 10gb switch on the 1gb side with cat 6 and to the mb lan port and got in, I performed a full parity check ran great with no errors, parity was successful and then rebooted thinking that its just a fluke that my 10gb card is not even lit up.

same issue no 10gb lights reset the sfp copper cables in card and the switch still no activity lights on the card.

 

if i go to the network settings it says this - D0:50:99:8A:9B:19 - Interface is shutdown (inactive) if i go to devices system devices it shows the following (below), My mellanox connectx card was just working in 6.1.9 , do i have to enable this 10gb card through some special settings i am not seeing? 

 

now i do have (2) 1gb lan ports on the MB and the other 1gb port does not light up, so maybe its the one mentioned above that's inactive?

 

0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0)

 

Diagnostics attached... did look around in the ethtool config and do see this...

 

Settings for eth2:

Supported ports: [ FIBRE ]

Supported link modes:  10000baseT/Full

Supported pause frame use: No

Supports auto-negotiation: No

Advertised link modes:  10000baseT/Full

Advertised pause frame use: No

Advertised auto-negotiation: No

Speed: Unknown!

Duplex: Unknown! (255)

Port: FIBRE

PHYAD: 0

Transceiver: external

Auto-negotiation: off

Supports Wake-on: d

Wake-on: d

Current message level: 0x00000014 (20)

      link ifdown

Link detected: no

 

driver: mlx4_en

version: 2.2-1 (Feb 2014)

firmware-version: 2.9.1000

expansion-rom-version:

bus-info: 0000:0b:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: yes

--------------------------------

 

AJ

tower-diagnostics-20160906-2154.zip

[bonienl]

Upgraded to rc5 and my Mellanox connectx 10gb card says inactive in the network settings, I got to the network settings by hooking up to my 10gb switch on the 1gb side with cat 6 and to the mb lan port and got in, I performed a full parity check ran great with no errors, parity was successful and then rebooted thinking that its just a fluke that my 10gb card is not even lit up.

same issue no 10gb lights reset the sfp copper cables in card and the switch still no activity lights on the card.

 

if i go to the network settings it says this - D0:50:99:8A:9B:19 - Interface is shutdown (inactive) if i go to devices system devices it shows the following (below), My mellanox connectx card was just working in 6.1.9 , do i have to enable this 10gb card through some special settings i am not seeing? 

 

now i do have (2) 1gb lan ports on the MB and the other 1gb port does not light up, so maybe its the one mentioned above that's inactive?

 

0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0)

 

sorry anyone got any ideas? Diagnostics attached...

AJ

 

Have you tried "Port Up" to make the inactive port active?

 

[ajgoyt]

Upgraded to rc5 and my Mellanox connectx 10gb card says inactive in the network settings, I got to the network settings by hooking up to my 10gb switch on the 1gb side with cat 6 and to the mb lan port and got in, I performed a full parity check ran great with no errors, parity was successful and then rebooted thinking that its just a fluke that my 10gb card is not even lit up.

same issue no 10gb lights reset the sfp copper cables in card and the switch still no activity lights on the card.

 

if i go to the network settings it says this - D0:50:99:8A:9B:19 - Interface is shutdown (inactive) if i go to devices system devices it shows the following (below), My mellanox connectx card was just working in 6.1.9 , do i have to enable this 10gb card through some special settings i am not seeing? 

 

now i do have (2) 1gb lan ports on the MB and the other 1gb port does not light up, so maybe its the one mentioned above that's inactive?

 

0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0)

 

sorry anyone got any ideas? Diagnostics attached...

AJ

 

Have you tried "Port Up" to make the inactive port active?

sorry how do you port up?  wait could it be that i have to stop the array then go into network settings and set the eth0 as 1st and then eth1 as 2 and so on?  - after the reboot i see more lan ports, see the screen shot. I am not running a connectx 4 im running a 2 , looks like unraid only has connectx 4 drivers? or maybe there backward compatible!

[bonienl]

Upgraded to rc5 and my Mellanox connectx 10gb card says inactive in the network settings, I got to the network settings by hooking up to my 10gb switch on the 1gb side with cat 6 and to the mb lan port and got in, I performed a full parity check ran great with no errors, parity was successful and then rebooted thinking that its just a fluke that my 10gb card is not even lit up.

same issue no 10gb lights reset the sfp copper cables in card and the switch still no activity lights on the card.

 

if i go to the network settings it says this - D0:50:99:8A:9B:19 - Interface is shutdown (inactive) if i go to devices system devices it shows the following (below), My mellanox connectx card was just working in 6.1.9 , do i have to enable this 10gb card through some special settings i am not seeing? 

 

now i do have (2) 1gb lan ports on the MB and the other 1gb port does not light up, so maybe its the one mentioned above that's inactive?

 

0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0)

 

sorry anyone got any ideas? Diagnostics attached...

AJ

 

Have you tried "Port Up" to make the inactive port active?

sorry how do you port up?

 

Go to network settings and under the corresponding interface is a button labeled 'Port Up'

 

(only available for ports other then eth0)

 

[ajgoyt]

Upgraded to rc5 and my Mellanox connectx 10gb card says inactive in the network settings, I got to the network settings by hooking up to my 10gb switch on the 1gb side with cat 6 and to the mb lan port and got in, I performed a full parity check ran great with no errors, parity was successful and then rebooted thinking that its just a fluke that my 10gb card is not even lit up.

same issue no 10gb lights reset the sfp copper cables in card and the switch still no activity lights on the card.

 

if i go to the network settings it says this - D0:50:99:8A:9B:19 - Interface is shutdown (inactive) if i go to devices system devices it shows the following (below), My mellanox connectx card was just working in 6.1.9 , do i have to enable this 10gb card through some special settings i am not seeing? 

 

now i do have (2) 1gb lan ports on the MB and the other 1gb port does not light up, so maybe its the one mentioned above that's inactive?

 

0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0)

 

sorry anyone got any ideas? Diagnostics attached...

AJ

 

Have you tried "Port Up" to make the inactive port active?

sorry how do you port up?

 

Go to network settings and under the corresponding interface is a button labeled 'Port Up'

 

(only available for ports other then eth0)

well i messed up i guess at the bottom for eth0 i set it to the mellanox 4 which has to be my 10gb card but in reality its a connectx 2 , rebooted and still no activity lights on the 10gb card, and now i cannot get into the main page,   on my monitor it say Device eth0 does not exist   so crap I guess ill try to fix by going back a version or? 

[bonienl]

Upgraded to rc5 and my Mellanox connectx 10gb card says inactive in the network settings, I got to the network settings by hooking up to my 10gb switch on the 1gb side with cat 6 and to the mb lan port and got in, I performed a full parity check ran great with no errors, parity was successful and then rebooted thinking that its just a fluke that my 10gb card is not even lit up.

same issue no 10gb lights reset the sfp copper cables in card and the switch still no activity lights on the card.

 

if i go to the network settings it says this - D0:50:99:8A:9B:19 - Interface is shutdown (inactive) if i go to devices system devices it shows the following (below), My mellanox connectx card was just working in 6.1.9 , do i have to enable this 10gb card through some special settings i am not seeing? 

 

now i do have (2) 1gb lan ports on the MB and the other 1gb port does not light up, so maybe its the one mentioned above that's inactive?

 

0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0)

 

sorry anyone got any ideas? Diagnostics attached...

AJ

 

Have you tried "Port Up" to make the inactive port active?

sorry how do you port up?

 

Go to network settings and under the corresponding interface is a button labeled 'Port Up'

 

(only available for ports other then eth0)

well i messed up i guess at the bottom for eth0 i set it to the mellanox 4 which has to be my 10gb card but in reality its a connectx 2 , rebooted and still no activity lights on the 10gb card, and now i cannot get into the main page,   on my monitor it say Device eth0 does not exist   so crap I guess ill try to fix by going back a version or?

 

Powerdown your unRAID system, and take your flash device out and put it in another PC. Then delete the files network.cfg and network-rules.cfg under folder config.

 

When you now boot unRAID it will start with default settings, which make all interfaces available.

 

From this point you can (re-)configure your desired network settings.

 

[ajgoyt]

Upgraded to rc5 and my Mellanox connectx 10gb card says inactive in the network settings, I got to the network settings by hooking up to my 10gb switch on the 1gb side with cat 6 and to the mb lan port and got in, I performed a full parity check ran great with no errors, parity was successful and then rebooted thinking that its just a fluke that my 10gb card is not even lit up.

same issue no 10gb lights reset the sfp copper cables in card and the switch still no activity lights on the card.

 

if i go to the network settings it says this - D0:50:99:8A:9B:19 - Interface is shutdown (inactive) if i go to devices system devices it shows the following (below), My mellanox connectx card was just working in 6.1.9 , do i have to enable this 10gb card through some special settings i am not seeing? 

 

now i do have (2) 1gb lan ports on the MB and the other 1gb port does not light up, so maybe its the one mentioned above that's inactive?

 

0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0)

 

sorry anyone got any ideas? Diagnostics attached...

AJ

 

Have you tried "Port Up" to make the inactive port active?

sorry how do you port up?

 

Go to network settings and under the corresponding interface is a button labeled 'Port Up'

 

(only available for ports other then eth0)

well i messed up i guess at the bottom for eth0 i set it to the mellanox 4 which has to be my 10gb card but in reality its a connectx 2 , rebooted and still no activity lights on the 10gb card, and now i cannot get into the main page,   on my monitor it say Device eth0 does not exist   so crap I guess ill try to fix by going back a version or?

 

Powerdown your unRAID system, and take your flash device out and put it in another PC. Then delete the files network.cfg and network-rules.cfg under folder config.

 

When you now boot unRAID it will start with default settings, which make all interfaces available.

 

From this point you can (re-)configure your desired network settings.

 

bonienl while waiting for you to reply i shutdown and took out the flash drive and then copied bzroot and bzimage to the root and rebooted and it went back to 6.1.9 , I still dont have activity lights on my 10gb all shares etc are ok by what i can tell but now my dockers are missing! My cache drive SSD is being read from the main page and my docker file 52 gb is still in the docker file, what do you suggest i do at this point?

 

1. upgrade back to RC5, then work on getting back my two dockers or?

2. stay with 6.1.9 and then fix the dockers and then to try to get the 10gb card working...

 

sorry i will wait till you reply,,, the only dockers i had were plex and emby!

[bonienl]

1. upgrade back to RC5, then work on getting back my two dockers or?

2. stay with 6.1.9 and then fix the dockers and then to try to get the 10gb card working...

 

sorry i will wait till you reply,,, the only dockers i had were plex and emby!

 

When upgrading to unRAID 6.2.0 you need to redo Docker installation due to changes in Docker itself (see this post http://lime-technology.com/forum/index.php?topic=47408.0)

 

My recommendation is to use unRAID 6.2.0 and follow the instructions to recreate Docker.

 

There is also more advanced network control in unRAID 6.2.0 which allows you to set your interfaces as you like.

 

[JorgeB]

I am not running a connectx 4 im running a 2 , looks like unraid only has connectx 4 drivers? or maybe there backward compatible!

 

mlx4_core is the driver, I have the Connect-X 2 and they work fine on v6.2, like Bonienl pointed out, set it as eth0 or press port up.

 

Also don't forget that on v6.2, as long as you have console access, through IPMI or local monitor, you can boot with the GUI option and change network settings with the inbuilt browser.

[ajgoyt]

I am not running a connectx 4 im running a 2 , looks like unraid only has connectx 4 drivers? or maybe there backward compatible!

 

mlx4_core is the driver, I have the Connect-X 2 and they work fine on v6.2, like Bonienl pointed out, set it as eth0 or press port up.

 

Also don't forget that on v6.2, as long as you have console access, through IPMI or local monitor, you can boot with the GUI option and change network settings with the inbuilt browser.

 

Ok I will have to go to work in awhile but I followed Bonienl advice - deleted the network settings and rebooted, my static ip i reset later , then i reinstalled rc5 and it installed fine rebooted again. Dockers are back Plex-Emby but need updating don't care i stopped them for now.  I am not touching nothing in Network settings as they look different then before! and i still have no activity or power light on my Connect x-2. Even though the connect x-2 card is not lit up it showing in devices. I am in on the MB 1gb lan port.

 

I have attached another diagnostic, and network settings it appears bonding is set to active backup 1  on eth0 , and the members are eth0,eth1,eth2 , if I look at eth1 and 2 - it says  D0:50:99:8A:9B:17 - interface is shutdown (inactive) probably because bonding is on? I am a little confused on what i should do? 

 

tell me if this is wrong...

Do i change the eth from the interface eth 0 or do i use the interface rules at the bottom for priority of eth?

I have about an hour before i need to get ready for work if someone can set me straight, Slap me lol i would appreciate it.....

 

 

 

 

tower-diagnostics-20160907-0645.zip

[JorgeB]

Do i change the eth from the interface eth 0 or do i use the interface rules at the bottom for priority of eth?

I have about an hour before i need to get ready for work if someone can set me straight, Slap me lol i would appreciate it.....

 

I believe you have to turn off bonding, these are my settings, connect-x 2 is eth1:

 

You can also choose which NIC is eth0, my eth0 is gigabit because I don't have a 10GbE switch, the connectx-2  is connected directly to my desktop, all other computers access the server using gigabit.

 

[ajgoyt]

Do i change the eth from the interface eth 0 or do i use the interface rules at the bottom for priority of eth?

I have about an hour before i need to get ready for work if someone can set me straight, Slap me lol i would appreciate it.....

 

I believe you have to turn off bonding, these are my settings, connect-x 2 is eth1:

Johnnie ok turned off bonding and unraid became unresponsive waited about 5 minutes, rebooted from the gui and bonding is off.

I ported up on eth2 which is the connect x-2 card and it thought for a bit and then the menu came back set the ip assigment to auto and it still says in red (at the very bottom) eth2 is down check cable. Do i set eth0 with the interface rules to connectx-2 or?

 

 

[JorgeB]

Do i set eth0 with the interface rules to connectx-2 or?

 

If the mellanox is your main NIC and it has internet connectivity set it to eth0, you have to reboot after.

[ajgoyt]

Do i set eth0 with the interface rules to connectx-2 or?

 

If the mellanox is your main NIC and it has internet connectivity set it to eth0, you have to reboot after.

it's been my main NIC since i switched to 10gb for my unraid and my workstation, So i did exactly that on the interface rules i set eth0 to the connectx-2 and the mb lan port to what the mellanox was (basically swapped) and it wanted a reboot, I did and still no activity lights on mellanox and im glad i have the gui on my monitor because its says eth0 is down. I think between the upgrade originally from 6.1.9 to RC5 my mellanox card died. Now tonight i can swap cards with this win10 workstation to confirm, I really don't want to disturb my sata pci card in unraid.  i think its just a fluke that the mellanox died... hopefully its not my PCIX8 slot.

 

I will be back tonight for more t/s to confirm my mellanox is DOA thanks for the help!

 

AJ

[ajgoyt]

Do i set eth0 with the interface rules to connectx-2 or?

 

If the mellanox is your main NIC and it has internet connectivity set it to eth0, you have to reboot after.

it's been my main NIC since i switched to 10gb for my unraid and my workstation, So i did exactly that on the interface rules i set eth0 to the connectx-2 and the mb lan port to what the mellanox was (basically swapped) and it wanted a reboot, I did and still no activity lights on mellanox and im glad i have the gui on my monitor because its says eth0 is down. I think between the upgrade originally from 6.1.9 to RC5 my mellanox card died. Now tonight i can swap cards with this win10 workstation to confirm, I really don't want to disturb my sata pci card in unraid.  i think its just a fluke that the mellanox died... hopefully its not my PCIX8 slot.

 

I will be back tonight for more t/s to confirm my mellanox is DOA thanks for the help!

 

AJ

 

Tom and John there is something going on with RC5 I swapped cards with the Win 10 workstation (as i type this) and the connectx-2 is working great, So for grins I put the Win 10 card in a different PCIX8 slot that was working just fine with my GT220 vid card and still no lights etc , the sys devices sees the card.  So I have verified my cards are just fine and SFP cables. 

 

I will run RC5 with my MB 1gb lan port till there is a fix or if you want I can do some testing for you. I have attached my Diagnostic to this.

please let me know with detailed instructions for any testing!

 

AJ

tower-diagnostics-20160907-7-14PM.zip

[ajgoyt]

Rebooted/Shutdown , for what ever reason WALLAH it works reset the interface rule for ETH0 for the Mellanox and rebooted and all is good.....  not sure what changed - At work i call this F*$#@ Magic!

 

AJ