Hello,

 

Recently I am working on my homelab network upgrade from 2.5G to 25G(just new to fiber world). The start point is that I move all my workspaces(lots of source code and pytorch models) to NAS thus I want a fast remote SMB access near local storage speed.

 

Back to the topic, here I have 2 25G NICs. One installed on my working PC and one on Unraid(my NAS). However the one installed on unraid seems doesn't have correct driver thus doesn't show in network settings. But I can see it in system device and can passthrough it to a VM.

 

Since the key point is to get a fast link between my working PC and NAS, without buying another expensive 25G switch, in theory I have 3 options:

1. Just install the 2 NICs on baremetal windows and unraid and use a fiber to connect. This is the most straightforward way.

2. Assign the NIC installed on Unraid to a VM, e.g. openwrt and just bond all the NIC ports and virtual NIC to make it perform like a switch.

3. Totally turn Unraid to a All-in-one server, including a openwrt VM perform like a router.

 

Since it looks my nic doesn't show up in network setting in Unraid, so 1# not work. Or maybe I can try to compile the driver for unraid, but I really don't know how to do that.

 

Now I switched to 2#, which mean a openwrt VM will perform like a switch and bond the 25G NIC ports and the VM's virtual NIC.

 

Then I tried to perform a ipref3 test from the workstation, here is the result:

1. From windows to Openwrt VM, it says around 20G bitrate

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  23.7 GBytes  20.4 Gbits/sec                  sender
[  5]   0.00-10.07  sec  23.7 GBytes  20.2 Gbits/sec                  receiver

2. From windows to Unraid Server(physically it's the same machine with Openwrt), it says around 3-5G bitrate. (With Virtio, others perform much worse, e.g. e1000 around 700~1000MB)

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  5.05 GBytes  4.34 Gbits/sec                  sender
[  5]   0.00-10.02  sec  5.05 GBytes  4.33 Gbits/sec                  receiver

3. From Unraid Server to windows

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.5 GBytes  14.2 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.5 GBytes  14.2 Gbits/sec                  receiver

 

 

Then I try to perform a CrystalDiskMark test toward the cache share on Unraid(2 x 256GB PCIe 3.0 SSD, PM961 and SM961, btrfs+raid1), it says:

[Read]
  SEQ    1MiB (Q=  8, T= 1):  1467.612 MB/s [   1399.6 IOPS] <  5710.67 us>
  SEQ  128KiB (Q= 32, T= 1):  1858.934 MB/s [  14182.5 IOPS] <  2254.66 us>
  RND    4KiB (Q= 32, T=16):   352.017 MB/s [  85941.7 IOPS] <  5949.30 us>
  RND    4KiB (Q=  1, T= 1):    13.625 MB/s [   3326.4 IOPS] <   300.44 us>

[Write]
  SEQ    1MiB (Q=  8, T= 1):   587.593 MB/s [    560.4 IOPS] < 14229.88 us>
  SEQ  128KiB (Q= 32, T= 1):   491.525 MB/s [   3750.0 IOPS] <  8508.06 us>
  RND    4KiB (Q= 32, T=16):   136.177 MB/s [  33246.3 IOPS] < 15352.36 us>
  RND    4KiB (Q=  1, T= 1):    13.689 MB/s [   3342.0 IOPS] <   298.99 us>

 

Here I have couple questions:

1. The samba test show good performance with SEQ read. However, 4k(Q1/T1) read/write is much worse than the result from testing locally. Is that because SMB or network? Now I almost keep all disk and SMB setting default in Unraid.

2. While the speed between my PC and openwrt VM is good(almost 25G), there is a big drop between PC and unraid. Is it the VM's virtual NIC the bottleneck? Is there any improvement can do?

3. I am just new to this area, is there any other better solution?

by defualt unraid advertises a 1 GB network.
 

You may need to find and get the data for 25G networking as it is uncommon. to fix/set the advertsiment on speed.

 

You may even need to install the melnox driver plugin to asis with some fiber comunication (more libs) as I don't think a driver nor setting are made for 25G on Unraid yet. Is it possible yes. 

In this case If you wish to use the machine in this way. I would recommend Proxmox. and pass a single vdisk imge disk for cache btrfs to use unraid dockers in a VM stated

 

Proxmox will have a bit beter setup for your 25G network and a overall better experience as a hypeV in terms of setting up labs and the forbidden router in a vm.

 

as proxmox virtual netwrok over its bridge can gurantee a 10G link between VM through its LocalHost...

 

review: