Slow cache drive write speeds in raid0 with new (to me) server

[bouis]

I have two 2tb sata SSDs in raid0 as a cache drive. I'd formerly used a fairly modern desktop motherboard and they worked as expected (1gigabyte/s read & write), but today I moved the unraid server over to an older supermicro box with an X9DRi-F motherboard and two v1 2620 xeons. It's got two sata iii ports, which I've connected the SSDs to. I also have a gigabit ethernet card (mellanox x-3).  But I only get 280-350 megabytes/s write speeds when copying to the cache drive over network. Formerly I was getting the expected 1 gigabyte/s.

 

So, I test the ethernet with iperf3 and it's giving me 1 gigabyte/s. I test the SSDs with diskspeed docker, and they're giving me 500mb/s read (each), 1000mb/s when both are tested simultaneously to check the controller. I have the trim plugin installed and tried manually trimming the cache.

 

I tried adding an HBA and connecting the SSDs to it, and I still get the same slow write speeds when copying files over network. The same HBA (9211-8i) can max out SATA3 SSDs in windows, though I don't think I ever tried it with unraid before now.

 

So what gives? Any thoughts are greatly appreciated.

argos-diagnostics-20190630-0728.zip

[bouis]

This is very strange. I put an NMVE drive in and tested it and got:

 

root@argos:/mnt# dd if=/dev/zero of=/mnt/testnv/test1.img bs=10G count=1 oflag=dsync
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 4.27589 s, 502 MB/s

 

The expected write speed is > 1500megabytes/s. However, the actual read speeds as tested by diskspeed plugin are the expected ~2750megabytes/s.

 

Could it be that the single-thread performance of the CPU is the bottleneck here? I don't have experience running these kind of SSDs on older hardware.

 

 

[Vr2Io]

On 6/30/2019 at 3:47 PM, bouis said:

But I only get 280-350 megabytes/s write speeds when copying to the cache drive over network.

This should be normal, bottleneck usually in storage side. 1st, two 860evo may not powerful enough. 2nd, some people i.e. @johnnie.black setup a ~10ish SSD cache pool could perform 1GB/s consistence write.

 

Some times ago, I add a Nvme ( benchmark write > 1GB/s ) also can't get consistance write speed, so now, my Unraid haven't any Nvme.

 

I have new setup a 9 spindle RAID0 cache pool and RW speed really good, it still in transfering data and testing, general test can saturate 10G NIC.

 

 

Edited July 2, 2019 by Benson

[bouis]

Thanks, but these two drives in RAID 0 were getting very close to 1GB/s sequential writes with the old (newer) motherboard and cpu.

 

I thought it was a SATA controller issue but that doesn't explain why the NVME drive is bottlenecked too. That thing will do > 2GB/s writes.

 

Tomorrow I am going to try it with an newer CPU (E5 2650 v2) and see if that changes anything.

 

 

 

[JorgeB]

On 6/30/2019 at 8:47 AM, bouis said:

two v1 2620 xeons

SMB transfers are single threaded, this CPU won't be able to handle 1GB/s even if the devices can.

 

Ideally you want a recent 3GHZ+ CPU, jumbo frames and direct i/o enable, with that I can get around 1GB/s transferring to a single fast NVMe device.

[bouis]

That makes sense. Still, it's quite a bit faster single-threaded than the CPUs that're in there. I'll report back how it works. Thanks again.

[JorgeB]

Just as an example, only difference between these two transfers to a 12xHDD btrfs raid6 pool was a CPU upgrade:

 

 

IIRC it was Celeron G1620 vs Xeon E3-1220v2, what makes the difference isn't the extra cores but the difference in single core performance, G1620 has a single core passmark rating of 1445 vs 1827 for the Xeon, there's around 25% difference in performance and the transfer speed increased by a similar amount, note that the CPU is still the bottleneck is this case, a faster one would still improve performance a bit more, but in this case good enough for me, to be able to get around 1GB/s on SMB transfers I recommend a CPU with a single core rating of around 2000.

 

 

Edited July 2, 2019 by johnnie.black

[bouis]

Thanks! Very helpful.

 

Maybe I should look into multi-threaded windows copy programs. It seems like there should be something with a shell extension that adds a "paste with two threads" dialogue in the right-click explorer menu.

[bouis]

Performance with the 2650 v2s is pretty disappointing. But if I run two threads I can max out the 10gb ethernet.

 

Unfortunately I can't find any windows utility that splits mass copies into multiple threads. I tried a bunch of them and none seem to have this functionality and a usable interface.

 

Looks like I'll just have to manually split my copies. It's not a big deal, I guess. And once I get the box filled up the performance issues won't matter and I'll probably get rid of one of the cache drives anyway. No sense in having a $300 drive sit idle.

Edited July 3, 2019 by bouis