I started setting up my Unraid server with the built-in ethernet on my system's motherboard.  I later added an SFP+ card (Intel X-520 chipset) and connected it to my switch (a UniFi Pro Max PoE switch).  Everything seemed to be going well - whereas file transfers with the ethernet maxed out close to the 2.5 Gbps limit of that port, after adding on the SFP+ card I was getting close to 5 Gbps.  I gave the SFP+ connection a fixed IP on the switch side, leaving it set to automatic within Unraid, and figured that was that.  However, after rebooting the system it seemed like it was favoring the ethernet port for file transfers again, so I unplugged the ethernet... and lost my connection to the server entirely, even though Unraid reported that its LAN IP was the fixed address assigned to the SFP+ connection.

 

Oddly, UniFi reported that the Unraid server was still online and attached to the correct SFP+ port, but it also showed an unknown MAC address on that port.  Plugging the ethernet cable back in gave me access to the server again, and if I forcefully the unknown MAC address through the UniFi Network application, the UniFi Network application would identify the correct Intel SFP+ card's MAC address for the Unraid server on the SFP+ port it is plugged into.

 

I thought maybe it was a fluke, so I removed the ethernet cable and the same thing happened again.

 

Bonding and bridging are both disabled in Unraid.  The SFP+ card may be a little faulty - it's a two-port card that I just recently got off of eBay, but only one port seems to be working - but I believe I've already transferred terabytes over it.  I'm already toying with contacting the seller to exchange the card; if we can attribute this purely to hardware failure then I definitely will, but the behavior just seems too odd for it be that.

 

If logs would be helpful, please let me know how to generate them, and in what circumstances you'd like them generated (presumably, after the ethernet is disconnected - but I will need to reconnect it to regain control of the server).

You can make the faster NIC eth0, Settings Network Settings - Interface Rules, reboot required, also make sure the other NIC is not configured, or at least using a different subnet if it is.

Hi Jorge, I appreciate the reply!  The SFP+ card's working port is set to eth0, with the ethernet port set to eth1.  A second ethernet port on the motherboard and the other, non-working port on the SFP+ card are set to eth2 and eth3, respectively, and aren't plugged in.  I was initially running them with "port down" but I also tried re-enabling the ports and it didn't seem to make a difference.

 

I'm thinking something is likely going on at the hardware level.  I'll replace the SFP+ card and report back (likely later this week).  I say that because I kept tinkering after making this post and the rogue MAC address seems to keep showing up even when the ethernet port is plugged in, and I actually could not get any data to transfer over the SFP+ card due to an IP conflict occurring for the SFP+ connection.  I can't explain why the connection worked the very first time I powered the system up with the SFP+ card installed, and it was stable enough to work for about 24 hours during which time multiple terabytes were transferred over it (confirmed by looking at eth0 in the dashboard and seeing peak transfer rates of 5 Gbps).  The event that triggered this instability was my modifying the ZFS ARC size which necessitated a reboot - all of this happened after the reboot.  A subsequent reboot didn't resolve it.  Since the card is already suspect (one port not working), hopefully a replacement will resolve it.

Using the original SFP+ card, which now seems to be working fine after the system was powered down over night... I think I have a resolution, but still can't explain it.  The solution has to do with assigning a static IP within Unraid, and not just at the router/switch level, and having only one device connected.

 

My network switch was recognizing two unknown MAC addresses as attempting to take the same IP and work on the same port as the actual SFP+ MAC address.  Every now and then the switch would throw an error about an IP collision, but for the most part it seemed content to have two or three devices with the same IP - for reasons I can't explain, because normally it would be constantly mentioning the collision.  Using a lookup database, the MAC addresses supposedly belong to TP-Link.  I don't have any TP-Link devices in my system.  As I think I mentioned, Unraid was reporting its IP as being the IP that I made a static assignment to the SFP+ port on, but I couldn't reach it at that IP.  I could only reach it from the IP I had assigned to the ethernet port (also set as static within Unraid), and if I unplugged the ethernet cable, the server became unreachable - even though it was showing in the switch as being actively connected.

 

The fix that I found was to set the IP for the SFP+ port as static within Unraid, and then reboot the system, removing the ethernet cable during the reboot process.  I've actually only once had success with Unraid dynamically taking an IP - not that an IP isn't assigned (I can see it within my router), but Unraid seems to expect that the IP is declared for it.  (The very first time I started Unraid I couldn't access it - I had to follow instructions online to modify the configuration on the USB drive, to assign a static IP.)  The very first time I booted with the SFP+ card then Unraid worked with the IP, but ever since then I had issues with the SFP+ card and connection until I set the IP as static within Unraid.  (It was always set as fixed within my router.)  Plugging in the ethernet cable seems to mess things up, as the connection favors the ethernet cable - even though the SFP+ port is listed as eth0, and even though Unraid reports the SFP+ connection's IP as its own on the main screen, and not the ethernet port's.  I've verified again that all ports (two SFP+ ports, and two ethernet ports) are not set to bridge or bond.

 

At this point, with just the SFP+ cable connected, the server has shown stability even after multiple reboots.  Realistically I'm not going to use the ethernet port at the same time as the SFP+ connection, so I'm just glad to have it working.  All the same, it's been a strange experience.

 

For reference, my Unraid server is using an Asus W680-ACE-IPMI motherboard, with IPMI card connected to the network via ethernet.  The SFP+ card is an Intel X520-DA2 (Lenevo-branded).  It's all wired to a UniFI Pro Max PoE 48-port network switch, managed by a UniFi Dream Machine router, with a Firewalla transparently connected between the switch and the router.  The Firewalla only deals with internet-bound traffic and probably has nothing to do with what was happening here, although as I write that, maybe I should have double-checked what was happening on the Firewalla side of things too.