1 minute ago, kkdev said:

So this is not an Unraid specific issue, but yeah the ice driver from the kernel still sucks as recent as 6.14, which is higher than Unraid's current kernel. All the major OS' have added recent drivers (Proxmox/Ubuntu/Truenas), so if you want to use it, you can build the drivers yourself. @ich777 provides the kernel on his Github, so you can use it to build new drivers.

I personally used the provided kernel to build the ice drivers in the past (I have since switched to Mellanox nics). I currently use the kernel to build nvme rdma drivers though.

How about Thor kernel which is using 6.17? Might save the hassle and use mellanox like you, only downside I guess it'll need a fan on the heatsink? Did the e810 get hot?

1 minute ago, mikeyosm said:

How about Thor kernel which is using 6.17? Might save the hassle and use mellanox like you, only downside I guess it'll need a fan on the heatsink? Did the e810 get hot?

I have tested that kernel, but did not test it with ice (I tested it for proper battlemage support). It is as simple as swapping the files on your flash drive (backing up first of course).

You can easily swap back afterwards. You can tell that the ice driver sucks by using the Tips and Tweaks plugin. You will see that you cannot change the Nic Offload from off to on. Also, when you perform iperf3 test, you'll see that only 1 CPU/thread gets all the load.

4 minutes ago, mikeyosm said:

How about Thor kernel which is using 6.17? Might save the hassle and use mellanox like you, only downside I guess it'll need a fan on the heatsink? Did the e810 get hot?

I run all my nics in a server chassis with enough airflow, but SFP nics are pretty power efficient and do not require insane cooling needs (unless you use them with an RJ45 transceiver of course).

You can even get away with running them in a mini pc with restricted airflow and you will be fine. It says the max consumption with DACs is 9.8W and 13.7W with optics. You should start worrying about fans if you have to cool 20+W in a case with restricted airflow.

4 minutes ago, kkdev said:

I have tested that kernel, but did not test it with ice (I tested it for proper battlemage support). It is as simple as swapping the files on your flash drive (backing up first of course).

You can easily swap back afterwards. You can tell that the ice driver sucks by using the Tips and Tweaks plugin. You will see that you cannot change the Nic Offload from off to on. Also, when you perform iperf3 test, you'll see that only 1 CPU/thread gets all the load.

What were the temps like on the e810 and are you using a fan on the mellanox card?

30 minutes ago, mikeyosm said:

What were the temps like on the e810 and are you using a fan on the mellanox card?

I haven't checked temps because there hasn't been a need. The NICs will shutdown when the operating temps far exceeds what they can handle. You only want to worry about temps with HBAs, as they can actually result in data loss. NICs are pretty resilient. Like I said earlier, unless you are doing an RJ45 transceiver (even with that, the newer tranceivers that consume 1.8W are very efficient) or 100gbps NIC (even those aren't bad, I run Mellanox CX555A inside my workstation with a regular PC case) you should not worry too much about NIC temps.

On 10/10/2025 at 11:45 AM, kkdev said:

So this is not an Unraid specific issue, but yeah the ice driver from the kernel still sucks as recent as 6.14, which is higher than Unraid's current kernel. All the major OS' have added recent drivers (Proxmox/Ubuntu/Truenas), so if you want to use it, you can build the drivers yourself. @ich777 provides the kernel on his Github, so you can use it to build new drivers.

I personally used the provided kernel to build the ice drivers in the past (I have since switched to Mellanox nics). I currently use the kernel to build nvme rdma drivers though.

How's the performance with the RDMA enabled? I have two ConnectX-5 I'm wanting to get more performance out of.

37 minutes ago, james7360 said:

How's the performance with the RDMA enabled? I have two ConnectX-5 I'm wanting to get more performance out of.

I have a U.2 flash server of 24 NVME drives and use RDMA to access its storage. It is similar to iSCSI so you mount it as a block storage.

I am able to max out my 100gbps network. Remember that your switch has to support RDMA too. I attached an image a while back which I also posted on Reddit about how I was able to saturate my 100gbps link with RDMA and Truenas Scale. That was before I could build the RDMA modules for Unraid.

7 hours ago, kkdev said:

I have a U.2 flash server of 24 NVME drives and use RDMA to access its storage. It is similar to iSCSI so you mount it as a block storage.

I am able to max out my 100gbps network. Remember that your switch has to support RDMA too. I attached an image a while back which I also posted on Reddit about how I was able to saturate my 100gbps link with RDMA and Truenas Scale. That was before I could build the RDMA modules for Unraid.

Wow, that’s impressive!
My setup uses two MCX556A-EDAT 100g cards directly connected to each other no switch. On the Unraid side, I’ve got a 4-drive Z1 array with Gen 4 NVMe drives. I can get about 2-3GB/s over SMB currently.
Have you experimented with SMB Direct at all? is it any Improvement over SMB?

Last night I tried building my own kernel to add RDMA support, but ran into a “Verifying checksum” error on boot.

You should do a short guide on adding the drivers to the Unraid kernel. It would be much appreciated. You seem like you really know your stuff.

Also, thanks for your post. I was going to get the Intel E810, but decided against it.

I haven't experimented with SMB direct but from what I read about it, only windows supports it. There is barely any info on it for Linux, although last I heard, it was experimental. I can get around 6GB/s over SMB on Unraid/Truenas. SMB needs a decent single threaded performance, so if your Unraid server has decent single thread performance, you can easily get 5GB/s - 6GB/s using some TCP and SMB tunables.

As for the RDMA support, you likely run into the checksum problem because after building the kernel, you have to generate the checksum of your bzmodules and bzfirmware and replace it with the one on your flash drive. For me, I did not build a new kernel. I simply built all the NVME modules separately and added them to Unraid.

When I get the chance, I'll definitely create a guide on building the RDMA modules.

Oh, I see. That would make sense that would be Windows only.
I've been messing with tuning the last few days and I got it to be pretty decent. Using Robocopy, I'm getting almost 6 GB/s reads and 5 GB/s writes.
Using the normal Windows file transfer, getting 4 GB/s read and 3 GB/s writes. Not to bad. Probably still some room for improvement, but it's acceptable.
Fast networking is fun but its Definitely a pain to get it working.

On 12/13/2024 at 10:38 PM, kkdev said:

Unless you really know what you are doing, my advice is to avoid the Intel 800 series (the 700 series is fine) until Unraid moves to a kernel that has a better Ice driver. The current ice driver for the 800 series does not implement any hardware offloading, so unless your CPU is very very beefy (even with that, I am not sure), you can barely saturate the link. Unraid does not support users compiling their own drivers (it does not come with any build tools), and Slackware is not the most user friendly distro.

 

If you know what you are doing, head over to https://github.com/ich777/unraid_kernel and in the releases section, you can grab the Kernel for Unraid, install a Ubuntu docker container (I used the most recent Ubuntu version, 24.10), and use the kernel and the container to build the new driver. After, you write some bash scripts to swap the driver for the newly built driver, reload the module, and restart networking anytime unraid starts.

Has anyone ever tried this? I met another issue seems caused by the low version of ice driver: I cannot enable switchdev & sriov at the same time. When I tried to create sriov vfs after swith the mode to switchdev according to this: https://edc.intel.com/content/www/us/en/design/products/ethernet/appnote-e810-eswitch-switchdev-mode-config-guide/ there was a "Resource temporarily unavailable" error.

So, I want to try to build my own driver. However, the ice driver: https://github.com/intel/ethernet-linux-ice/releases needs something like linux-headers for building. I tried both: 1. build the ice source in the docker: https://github.com/ich777/unraid_kernel/pkgs/container/unraid_kernel; 2. copy the ice source code into unraid source code and then build the entire kernel. This exception was alerted:

drivers/ice/common.mk:71: *** Kernel header files not in any of the expected locations.
drivers/ice/common.mk:72: *** Install the appropriate kernel development package, e.g.
drivers/ice/common.mk:73: *** kernel-devel, for building kernel modules and try again.  Stop.

Can anyone help on this?

Edited November 14, 2025Nov 14 by Yunfei