kkdev

I haven't experimented with SMB direct but from what I read about it, only windows supports it. There is barely any info on it for Linux, although last I heard, it was experimental. I can get around 6GB/s over SMB on Unraid/Truenas. SMB needs a decent single threaded performance, so if your Unraid server has decent single thread performance, you can easily get 5GB/s - 6GB/s using some TCP and SMB tunables. As for the RDMA support, you likely run into the checksum problem because after building the kernel, you have to generate the checksum of your bzmodules and bzfirmware and replace it with the one on your flash drive. For me, I did not build a new kernel. I simply built all the NVME modules separately and added them to Unraid. When I get the chance, I'll definitely create a guide on building the RDMA modules.

I have a U.2 flash server of 24 NVME drives and use RDMA to access its storage. It is similar to iSCSI so you mount it as a block storage. I am able to max out my 100gbps network. Remember that your switch has to support RDMA too. I attached an image a while back which I also posted on Reddit about how I was able to saturate my 100gbps link with RDMA and Truenas Scale. That was before I could build the RDMA modules for Unraid.

I haven't checked temps because there hasn't been a need. The NICs will shutdown when the operating temps far exceeds what they can handle. You only want to worry about temps with HBAs, as they can actually result in data loss. NICs are pretty resilient. Like I said earlier, unless you are doing an RJ45 transceiver (even with that, the newer tranceivers that consume 1.8W are very efficient) or 100gbps NIC (even those aren't bad, I run Mellanox CX555A inside my workstation with a regular PC case) you should not worry too much about NIC temps.

I run all my nics in a server chassis with enough airflow, but SFP nics are pretty power efficient and do not require insane cooling needs (unless you use them with an RJ45 transceiver of course). You can even get away with running them in a mini pc with restricted airflow and you will be fine. It says the max consumption with DACs is 9.8W and 13.7W with optics. You should start worrying about fans if you have to cool 20+W in a case with restricted airflow.

I have tested that kernel, but did not test it with ice (I tested it for proper battlemage support). It is as simple as swapping the files on your flash drive (backing up first of course). You can easily swap back afterwards. You can tell that the ice driver sucks by using the Tips and Tweaks plugin. You will see that you cannot change the Nic Offload from off to on. Also, when you perform iperf3 test, you'll see that only 1 CPU/thread gets all the load.

So this is not an Unraid specific issue, but yeah the ice driver from the kernel still sucks as recent as 6.14, which is higher than Unraid's current kernel. All the major OS' have added recent drivers (Proxmox/Ubuntu/Truenas), so if you want to use it, you can build the drivers yourself. @ich777 provides the kernel on his Github, so you can use it to build new drivers. I personally used the provided kernel to build the ice drivers in the past (I have since switched to Mellanox nics). I currently use the kernel to build nvme rdma drivers though.

No worries, I have not tested the in-tree drivers to verify, but all major OS's except Unraid offer a decent ice driver. This is likely due to the fact that they include a recent driver as part of their Kernel, since Intel E810 nics are pretty popular in the enterprise space. Unless Unraids decide to do something about the driver, I'll stay away from using the E810 NICs with Unraid.

Update: I tested the Unraid 7.1 rc3 and the ice driver is still terrible. I don't know what is happening here with Unraid, or how Unraid sources and builds its kernel, but just about any other Linux OS I have tried does not have this problem (and I have tried many). I just wanted to put it out there so that nobody scratches their heads trying to solve this problem. I have sinced switched to Mellanox ConnectX5 and move some servers away from Unraid.

Yup, I've been a Mikrotik fan since 2015, it has never failed me. They always have the cheapest and low power equipment available. I run their router, together with 2 switches which serves my needs. The only service that has outside access on my network is Plex, and it is served behind a reverse proxy, with a separate suricata container in Proxmox monitoring the open port. I know that I can do some advance routing with Mikrotik, but in doing so, I lose some routing performance (I'm have 10gbps internet), so I try to keep my routing and firewall rules pretty light. The reverse proxy is also doing geo-blocking.

Hey @ich777 thanks for your Unraid kernel releases. They have been super helpful for me figuring out things with Unraid. I appreciate the good work!

Well that makes sense, but virtualizing a critical infrastructure such as your router is probably not the way to go, in my opinion. I don't know if you've heard of the Minisforum MS-01, but they are excellent and very performant (and has a low profile PCIE slot which fits an Intel E810-CQDA2!), and because they use mobile chips they are very low power. I played with OpnSense on them through a Local NAT (I only have 10gbps internet) for benchmarking and I was able to hit 25gbps with IDS/IPS enabled while not consuming more than 80W in my tests. Personally though, I use a Mikrotik router (maybe I will add OpnSense between the internet and the router in a transparent bridge mode in the future, but I have no use for IDS/IPS at least for now).

I did some tests and Truenas Scale 24.10 which uses a bit older version of the kernel 6.6 has a better ICE driver. I do not know whether they patched the kernel to include a better version of the ice driver (maybe they did, Truenas Scale is geared towards enterprise storage so I am sure they will make sure that most nics are supported), or it was included in that version of the kernel (I did not have time to do research). What I know for sure is that kernel version 6.8 has a good driver, and I have no idea when Unraid will update the kernel to include that driver. Since 6.6 is an LTS version, you will have to probably wait for Unraid 8, which will take a while. My advice to you is to get a Mellanox ConnectX-4 or 5, a Broadcom NIC, or the Intel 700 series (the 700 series has 25gb NICs) if you want to have 25gbps in your server. I currently use a Mellanox ConnectX-5 100gbps NIC in my Unraid server and I am able to hit 98gbps to another client running Proxmox. I have sinced moved the 800 series NIC to one of my client devices.

Unless you really know what you are doing, my advice is to avoid the Intel 800 series (the 700 series is fine) until Unraid moves to a kernel that has a better Ice driver. The current ice driver for the 800 series does not implement any hardware offloading, so unless your CPU is very very beefy (even with that, I am not sure), you can barely saturate the link. Unraid does not support users compiling their own drivers (it does not come with any build tools), and Slackware is not the most user friendly distro. If you know what you are doing, head over to https://github.com/ich777/unraid_kernel and in the releases section, you can grab the Kernel for Unraid, install a Ubuntu docker container (I used the most recent Ubuntu version, 24.10), and use the kernel and the container to build the new driver. After, you write some bash scripts to swap the driver for the newly built driver, reload the module, and restart networking anytime unraid starts.

A better driver has been upstreamed. I just don't know which kernel version(s) provided the fix. From looking at the latest version of Proxmox and Ubuntu, they are using an upstream driver, not a custom driver.

I was able to finally compile the driver last night (official linux driver from Intel) and get the full performance of the NIC, so like I said, it was a driver issue. Someone experienced something similar with an older kernel in proxmox, which prompted me to the issue: https://forum.proxmox.com/threads/default-ice-driver-kernel-6-poor-performance.133855/ If there is a way to have the driver fixed before the latest 7 stable, that will be great.