Poor SSD Cache Read Speed 2.5Gbe

[anibis2]

Hi,

 

I have spent many hours trying to resolve this issue to no avail. I have a 10th Gen Intel i7 / 48GB / Realtek 2.5Gbe card. 2x 4TB Samsung 860 EVO BTRFS cache and a single 500GB Hynix nvme drive, along with 6 spinners. When I copy Veeam backup files from the cache/nvme drives to a Windows PC (Server 2019 and Windows 11) I get about 130MB/s. When I write a (different) large file from those same Windows PC's to the cache drive I saturate the 2.5Gbe link. Direct copies between the nvme/ssd top out at 500MB/s or so, the top speed of the Samsung. There has been no activity on the spinners when running these tests. 

 

I also hit the 130MB/s cap when copying backups from the Unraid box (cache only) to my QNAP. When I work on the QNAP cache / empty array with my Windows machines I get 250MB/s or so. Unraid can read (and write) to the QNAP at the same 250MB/s speed.  

 

Diags are attached.

 

*EDIT*

 

Transfers between my two 2.5Gbe Windows machines saturate the link both ways. 

 

Thanks.

 

unraid-diagnostics-20220516-2310.zip

Edited May 17, 2022 by darcon
Added Windows Comparison

[JorgeB]

Start by running a single stream iperf test in both directions to see if the issue is network related.

[anibis2]

6 hours ago, JorgeB said:

Start by running a single stream iperf test in both directions to see if the issue is network related.

 

Here's Unraid > Windows 11:

 

 

And Windows 11 > Unraid

 

 

So, it is a network problem. Where should I go from here? Please keep in mind that I see this 130MB mark on my QNAP as well, so it's not a Windows issue. 

 

 

Edited May 17, 2022 by darcon

[JorgeB]

Realtek driver support with Linux has always been hit and miss, look at the post below, it might also help you:

 

 

[anibis2]

Thanks for that, I didn't come across it in my searches. 

 

When I disable bridging reads are 220MB/s+ which is more what I would expect. I do run VM's which was one of the features that brought me to Unraid in 2015, so disabling bridging is not an option. I did try with bridging enabled and no VM's powered on, same slow read speeds. The machine is also in the basement on the other side of the house, so running a second cable would be a pain. 

 

*EDIT*

 

I created a new route for my local subnet to go out eth0, that fixed the issue. I am not seeing any knock-on effects after adding this route (br0 to local subnet seems to be permanent). Am I missing anything?  

 

Edited May 17, 2022 by darcon

[anibis2]

The subnet route did the trick. For whatever reason br0 is slow for reads only, so strange. Maybe it's the Realtek 2.5Gbe NIC, or perhaps something with my config. So stoked that it's fixed! I spent a bit of money on 2.5Gbe / 4TB SSD's and it was disappointing for it to make no difference.  

[Reda.Saiko]

On 5/19/2022 at 6:34 PM, darcon said:

The subnet route did the trick. For whatever reason br0 is slow for reads only, so strange. Maybe it's the Realtek 2.5Gbe NIC, or perhaps something with my config. So stoked that it's fixed! I spent a bit of money on 2.5Gbe / 4TB SSD's and it was disappointing for it to make no difference.  

 

I have the same issue, if possible, can you please explain in details how you did solve it (I don't really understand the subnet part). This would help a lot of people like me who avoid Intel 2.5GbE nic known for its issues just to get this.

[anibis2]

15 minutes ago, Reda.Saiko said:

 

I have the same issue, if possible, can you please explain in details how you did solve it (I don't really understand the subnet part). This would help a lot of people like me who avoid Intel 2.5GbE nic known for its issues just to get this.

 

Go into Settings > Network Settings > Routing Table, then add a route pointing your local subnet (192.168.0.0/24 in my case) to br0.

 

 

It should look like that when you're done. There will be other routes for your docker subnets, as well as ipv6 if applicable. Just leave those alone.

Edited January 18, 2023 by darcon

[Reda.Saiko]

1 minute ago, darcon said:

 

Go into Settings > Network Settings > Routing Table, then add a route pointing your local subnet (192.168.0.0/24 in my case) to br0.

 

 

It should look like that when you're done. There will be other routes for your docker subnets, as well as ipv6 if applicable. Just leave those alone.

 

Thank you a lot. So with this setting no need ti disable the bridging? For now I am testing with bridging to No and my speed is 2.5 2-ways, but like you I have some dockers and vms.

 

[anibis2]

8 minutes ago, Reda.Saiko said:

 

Thank you a lot. So with this setting no need ti disable the bridging? For now I am testing with bridging to No and my speed is 2.5 2-ways, but like you I have some dockers and vms.

 

 

Correct, you can re-enable bridging and everything should work as it usually does.

[Reda.Saiko]

Just now, darcon said:

 

Correct, you can re-enable bridging and everything should work as it usually does.

 

Sorry, I am really lost here. I think I have missed something.

 

With these settings, I get 1 gbps download and 2.5 gbps upload.

 

And with these settings, I get 2.5 gbpsboth ways.

 

Thank you for helping me here it's really appreciated.