Slow r/w speeds with NVMe cache drive [Resolved]

[DesertCookie]

Edit: What ultimately solved my problem was reinstalling Windows - now I'm back on 1GbE but at least get around 100-110MB/s. The Windows installation was about a year old and only had seen new RAM and some new HDD and SSD storage. Let's hope the upgrade to 10GbE later this week goes well.

 

I recently installed a 1TB WD Blue M.2 NVMe SSD as cache, as well as a DeLock 2.5GbE network card and directly connected my PC (via its onboard 2.5GbE port to my server). Read and write speeds are down compared to before the update.

 

Here are some metrics:

• 20-40MB/s write and 20-80MB/s read (the same as via 1GbE but more stable)
• read and write speed are pretty much the same for array and cache (verified using the web UI)
• HDDs are capable of 80-170MB/s r/w (according to DiskSpeed docker) - averages around 120MB/s
• cache SDD is cabable of 1GB/s-1.1GB/s r/w (according to DiskSpeed docker)
• 2.5GbE is direct connect Cat.7 (1m), 1GbE is via two switches and a router (total of 25m of Cat.6)
• <1ms latency via 2.5GbE instead of 2-3ms latency via 1GbE
• previous no-name 240GB M.2 SATA SSD was a lot faster

 

Overall, the web UI feels a lot snappiere since accessing it via 2.5GbE - before traffic needed to go through a switch, to my router, and back to a switch - now it's direct. Windows 10 shows me 2.5GbE speeds. The M.2 SSD is connected via PCIe 3.0 x4. I really don't know where the bottleneck is here. Internally (e.g., copying from array to cache) is faster but not as fast as I expect from this hardware.

 

I'd be grateful if someone else could take a look at this. I'm happy to provide more information - I just don't know what more at the moment.

 

My system:

• ASRock X399 Taichi
• Threadripper 1900X
• 2x16GB DDR4-3600 (@3200 1.25V) - survived a memBench run
• 1x WD Blue 1TB as cache
• 2x WD Se 4TB as main array drives (no parity atm)

The system I connect from is a Ryzen 9 system with a B550 Mainboard with onboard 2.5GbE - plenty of power on that end.

syslog.txt

Edited May 10, 2021 by DesertCookie
1) Added info. 2) Added my solution.

[JorgeB]

Start by running a single thread iperf test to check network bandwidth.

[DesertCookie]

On 3/26/2021 at 1:21 PM, JorgeB said:

Start by running a single thread iperf test to check network bandwidth.

Using SpaceInvader One's video as a guide I got the following results:

1GbE:   74 Mb/s (spread of 38 to 120 Mb/s)
2.5GbE: 126 Mb/s (spread of 70 to 191 Mb/s)

 

iperf3_results.txt

Edited March 27, 2021 by DesertCookie
Spelling

[JorgeB]

Clearly a network problem, you're getting under 10% expected performance, since it's low with both NICs unlikely to be a NIC problem, are both using the same switch or is the 2.5Gbe link direct? Also try with a different source computer if available.

[DesertCookie]

30 minutes ago, JorgeB said:

[...] are both using the same switch or is the 2.5Gbe link direct? Also try with a different source computer if available.

The 2.5GbE link is direct, as I wrote. I only have a phone via WiFi 4 available as second client (or a VM in the server but that's obviously useless here).

 

I didn't change MTU as I didn't expect that to be relevant for only 2.5GbE. Could that cause an improvement?

Edited March 28, 2021 by DesertCookie
Added info

[JorgeB]

35 minutes ago, DesertCookie said:

I didn't change MTU as I didn't expect that to be relevant for only 2.5GbE. Could that cause an improvement?

It might help a little, but you should still be able to get close to line speed without it, so not the problem.

[DesertCookie]

I've retested today - after having made no changes, not even shutting down the computers since the last test. Now I get the following results:

1GbE:   412 Mb/s (spread of 277 to 859 Mb/s)
2.5GbE: 508 Mb/s (spread of 432 to 567 Mb/s)

While this is better it still doesn't represent what I would expect from this set up.

 

Edit: A little while later I found that 1GbE is mroe reliable. Whether it's problem of the ethernet cards or one of Unraid with the ethernet cards, I don't know. I'll be adding the name of the cards to the original post as soon as I stop my server for the next time.

Edited April 22, 2021 by DesertCookie