10gb Networking Expectations.


Recommended Posts

Hi

 

I recently added a single 10gb Mellanox NIC card to my Unraid server and my Freenas server.  As a test I'm trying to see if i can hit that mark.  I'm not getting anywhere close to it.

 

This is how I'm testing:

 

On the unraid server, i have a single SSD cache drive (rated 500MBs).  SSD is connected to a 2308 LSI controller (IT mode). A windows 10 VM on it.  Network settings in the VM and in Unraid has MTU to 9000.

 

Freenas server has a two intel ssds (also rated 500 MBs) set for Stripping. SSD is connected to a 2308 LSI controller (IT mode).   Created a share on it.  MTU is set to 9000.  Setup these tunable options per the internet to optimize settings:

(enter via CLI and even rebooted)

 

sysctl kern.ipc.somaxconn=2048
sysctl kern.ipc.maxsockbuf=16777216
sysctl net.inet.tcp.recvspace=4194304
sysctl net.inet.tcp.sendspace=2097152
sysctl net.inet.tcp.sendbuf_max=16777216
sysctl net.inet.tcp.recvbuf_max=16777216

sysctl net.inet.tcp.sendbuf_inc=32768
sysctl net.inet.tcp.recvbuf_inc=524288
sysctl net.route.netisr_maxqlen=2048

 

I tested by coping a big file between the VM in cache and the share on freenas.  Maybe i hit  500 MBs according to Windows copying GUI but it didn't last very long.  On average i'm getting anywhere between 150 - 200 MBs.

 

Then I tried iperf.  Setup the freenas as the server and the VM as the client.  Ran the tests a bunch of times and basically getting about the same.

 

Based on my Hardware and setup, is this the MAX I can get? In order to get the 10gb transfer speeds, are do i need a nvme on an m2 in order to get that type of transfer? is the sata SSDs the bottleneck?

 

Thoughts? Thanks in advance!!

 


 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.