**VIDEO GUIDE** Setting up 10Gbe networking between a PC and Unraid server


guyht

Recommended Posts

Hi All,

 

Seeing a lot of people trying to get 10Gbe networking setup with varying success so wanted to share my experience.  I have managed to get 10Gbe networking setup relatively easily between my Unraid server and my PC, without the need of a 10Gbe network switch.  This gives me full 10Gbe speeds between my PC and my Unraid server and 1Gbe speeds across the rest of the network.  

 

Requirements:

2X  Asus XG-C100C 10Gbe network cards - they can be found on Amazon and elsewhere for less than $100

1X Cat 6 network cable

Unraid 6.4.0_rc7 or later (for C100C driver support)

 

 

 

 

Thanks and feel free to ask any questions!

  • Like 1
Link to comment

I did a write up on this too because I found little to no info. I am happy that you posted this video. It would have helped me out a lot!

Not to hi-jack your thread, but here is my info as well.

 

I used fiber optic connections because I found the NICs to be cheaper. Copper works just as good.

 

Equipment (all retired enterprise equipment from ebay):

Cache Drives - 2x Kingston 240gb SSDs in raid 0

Unraid Nic - HP Dual Port 10Gb Ethernet PCIe Card for Proliant 468349-001 ($38.00)

Client Nics - HP 10GB MELLANOX CONNECTX-2 PCIe 10GBe ETHERNET NIC 671798-001 ($12.00)

Trancievers - 4x HPE BladeSystem 455883-B21 Compatible 10GBASE-SR SFP+ 850nm 300m DOM Transceiver ($20.00)

Cables - LC UPC to LC UPC Duplex 2.0mm PVC(OFNR) OM3 Multimode Fiber Optic Patch Cable ($6.00)

 

FYI - for the next person looking to do this here is the process.

If you are not able to completely saturate the 10gbe connection, it is likely because the hard drive at one of the ends is too slow. this is why my cache is 2x SSDs in raid 0. Another cause could be that the share you are using it not set to use the SSD cache.

 

Windows 10:

 

1. install the 10gb NIC and give it a static IP address in a different subnet (if your other NIC's IP is 192.168.0.6 - change the value of the 3rd octet to any unused number 1-254, such as 192.168.66.6)

 

2. turn on network discovery (without this your computer will not respond to pings or any other kinds of remote requests). Simply open a windows explorer window and click the "network option". it will attempt to find computers and devices on your network, if network discovery has not been enabled you will be prompted to do so.

 

3. Your done for now, but you will need to force windows to use the 10gb NIC for connections to unraid (keep reading)

 

 

Unraid:

 

1. install your 10gb NIC and make sure you can see it in the "system info". Make note of the "eth#" shown

 

2. go to "settings > network settings" and scroll down to the "eth#" from the previous step. If it is in a "bond group", the array must be stopped and the "eth#" removed.

 

3. assign the "eth#" a static IP address. this address must be in the same subnet/netmask as your windows PC (in other words, 192.168.66.* - see windows sample from above).

 

4. hit apply

 

5. use Putty or something similar to connect to your unraid server. Ping the windows PC's 10gb NIC using it's address (our sample is 192.168.66.6). Because you have multiiple NIC's you must specify what NIC to use for ping.

 

ping -I eth# 192.168.66.6 (the -i switch allows you to ping using a spcific NIC. replace the # with your NIC id).

 

if the windows 10 computer responds all is well. Go back to windows to complete the configuration.

if the windows 10 computer does not respond, you missed one of the above steps or have some other issue.

 

Back to Win 10 Again (rhymes right?)

 

There are 2 ways to force windows to use the 10gb connection. For the purposes of documentation (keeping all info in 1 place) I have both listed, but in no way would I ever recommend using method 2.

 

Method 1: add unraid server to your "host" file

go to "C:\Windows\System32\drivers\etc" and open the "hosts" file with notepad

scroll all the way to the bottom and on a blank line enter the unraid servers 10gb NIC address followed by a space then an alias (I like to use the Hostname)

 

192.168.66.6 Hostname

 

now save the file and attempt to ping "Hostname". You should find that "Hostname" now resolves to the IP address you have specified in the "host" file.

If you ever have need to access the unraid server from this PC but do NOT want to use the 10gb NIC you can simply enter another line in your "host" file.

 

192.168.66.6 Hostname-10g

192.168.0.6 Hostname

 

line 1: will resolve the alias "Hostname-10g" to the servers 10gb NIC address

line 2: will resolve the alias "Hostname" to the servers 1gb NIC address

 

Method 2: NOT RECOMMENDED

Windows 10 now has more than 1 NIC that can connect to the unraid server. How will it know what one to use?

You would normally go in and change the NIC priority, but this is no good in windows 10. Change the settings all you want and they will just revert back. You are expected to manually change the "metric" (although it's not well documented).

 

go to the "network connections" in the control panel and select the properties of the 10gb connection.

now select the IPv4 option in the "networking" tab, then select the "advanced" button in the new window.

You should now see the "advanced TCP/IP settings" window. At the bottom of this window uncheck the "automatic metric" button and set a value.

 

keep in mind that lower numbers have a lower priority. I manually set a value of "10" for the 10gb NIC and "20" for my wireless NIC so the priority for one is higher than the other. This method uses one connection as a "failover" for the other. It will always try the 10gb connection first and use the other if 10gb fails. The problem with this is that it uses 10gb for everything and does so at the expense of giving your other NIC a lower priority.

 

23783821_1391174887675472_87982198458295

 

 

Sending BIG files to/from Unraid? Configure Jumbo packets (OPTIONAL):

It's basically what it sounds like. You can configure the connection to send larger packets of data at a time. This is more of a feature for those using the 10gb connection exclusively to transfer large files.

 

for this to work jumbo packets must be configured the same on both ends (unraid and windows).

If using a 10gb switch, the port(s) used to make this connection must be configured to use jumbo packets as well.

 

Windows 10

Go back to your windows computer and in the 10gb NICs driver properties you should find an option to configure jumbo packets. Simply set the MTU to 9000 and apply your changes.

 

Unraid

Go to "Settings > Network settings". Scroll down to your 10gb's "eth#" and change the MTU to 9000 and hit apply.

Now Go to "Settings > Global Share Settings -> Tunable" (enable direct IO): set to Yes

Edited by smaka510
  • Like 2
Link to comment
7 hours ago, smaka510 said:

seems to work fine with the HP/MELLANOX cards. I transfer files like a beast now. I'm looking at a managed switch as we speak!

 

Mellanox is still the 10GB with SFP+/GBICs not Ethernet in the RJ45 sense.

 

 

I have 3 10GBT cards coming and a netgear 10GBT switch, will set it up and see how the magic works with 6.3.5. 

Link to comment
  • 4 months later...

I made the mistake of re-imaging my windows 10 PC and found this impossible to get working again. The key to this working with mellanox VPI cards is your ability to toggle between "Infinity Band (aka IB)" and "Ethernet (aka eth)". I don't know what Mellanox did with the drivers but the "Port Protocol" tab in the device manager is gone.

 

I found another driver for this in the device manger under "system devices" instead of "Network Adapters". This has the "Port Protocol" tab, but everything is greyed out. Mellanox no longer supports X-2 cards so getting help from them is out the window.

 

This is so frustrating

 

Do yourself a favor when you get this working.... Export your FUNCTIONAL Windows 10 drivers and save them in a place you can get to them later.

Link to comment
  • 2 weeks later...

** Update **

I have been able to get this working again. I'm not sure why Mellanox moved the "protocol". If you are using Mellanox Connex-x cards that support Infinity Band (IB) and cannot figure out how to change the virtual protocol from Ethernet to IB try the following.

 

note: IB mode requires a subnet manager run otherwise IB will not work. Also Both ends (host/server/switch) must be configured this way.

 

1. Open the device manager

2. find your mellanox card and go to properties

3. select the "Protocol" tab and toggle between IB and Eth as desired.

 

If you do not see the "Protocol" tab try the following:

1. in the device manager look for your mellanox NIC under "system devices"

2. open properties and select the "Protocol" tab

3. toggle between IB and Eth as needed.

Link to comment
  • 1 year later...

I am pulling my hair out trying to establish a peer to peer 10Gb connection on Mac. I have tripled checked that my cards and cables are correct. Can someone take a look and let me know if I am doing something obvious wrong? I would very much appreciate it.

 

Update 1: I noticed that when I plug a CAT6 from my 10Gb card (which is running on a PCI slot) to my 1Gb router and renew the DHCP, it still does not find a connection? Could something be broken on my Mac's end?

 

1271448899_ScreenShot2020-05-06at7_26_38AM.thumb.png.ca9ce5492b664618f7dc46c030eef79a.png1580032758_ScreenShot2020-05-06at7_25_02AM.thumb.png.0250a5fdc254f525865c62be691c293c.png1157408607_ScreenShot2020-05-06at7_24_41AM.thumb.png.4aeb520030411d244d2acb132cf9ec5d.png1494525761_ScreenShot2020-05-06at7_25_20AM.png.00592315504682b4fbc4fa4a3a5889ca.png2144131999_ScreenShot2020-05-06at7_26_03AM.png.05cce224e7167c2536be0344144b3546.png

Edited by marionza
Link to comment
5 hours ago, marionza said:

I am pulling my hair out trying to establish a peer to peer 10Gb connection on Mac. I have tripled checked that my cards and cables are correct. Can someone take a look and let me know if I am doing something obvious wrong? I would very much appreciate it.

 

Update 1: I noticed that when I plug a CAT6 from my 10Gb card (which is running on a PCI slot) to my 1Gb router and renew the DHCP, it still does not find a connection? Could something be broken on my Mac's end?

 

1271448899_ScreenShot2020-05-06at7_26_38AM.thumb.png.ca9ce5492b664618f7dc46c030eef79a.png1580032758_ScreenShot2020-05-06at7_25_02AM.thumb.png.0250a5fdc254f525865c62be691c293c.png1157408607_ScreenShot2020-05-06at7_24_41AM.thumb.png.4aeb520030411d244d2acb132cf9ec5d.png1494525761_ScreenShot2020-05-06at7_25_20AM.png.00592315504682b4fbc4fa4a3a5889ca.png2144131999_ScreenShot2020-05-06at7_26_03AM.png.05cce224e7167c2536be0344144b3546.png

From screen capture, there are 2 different setting in unraid, eth0,1 bond and not bond. Anyway if you found interface have port up/down button, then you need press it and let it in up state. Otherwise, port will in disable state.

 

Pls also confirm does NIC at iMac could link in up state if connect to router.

Link to comment
  • 1 month later...

Following your video guide, the MTU size for the 10 Gig NIC is set to 1500 (it's the same as the 1 Gig NIC since they are bridged). At least on the Mac side of things, I've seen this setting cost about 20-30% performance (e.g. 7 Gbps vs 9 Gbps).

 

Thoughts on changing it to 9000?

Would this adversely affect the 1 Gig NIC's connection to the router?

Link to comment
  • 10 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.