Hi.

 

I've read several posts and topics regarding slow cache, and I have ran out of ideas.

 

My system is:

Dell Poweredge R720XD with Perc710 in IT mode.

5x 2TB SATA HGST with 2 parity disks.

1x NVMe on PCI-e adapter. Recognized and working fine.

10GbE

 

Writing to folders Multimedia with cache gives me only 250MB/s which is very slow, compared to my other UR NAS which writes 1,12GB/s. 

 

After reading some other posts here, I enabled "Disk Shares" from within settings.

I could now se ALL my drives, and also my cache drive. When writing directly to cache, I get full speed 1,12GB/s.

 

Why is it so, that when I write to my Multimedia folder (which I enabled to use cache) I get a slower speed then when I write directly to cache?

I also tried to write directly to disk1 which also gives me 1,12GB/s. How is this possible, when this is a SATA-3 6Gb/s disk?

 

 

Edited December 9, 20205 yr by j44rs4

18 hours ago, j44rs4 said:

Why is it so, that when I write to my Multimedia folder (which I enabled to use cache) I get a slower speed then when I write directly to cache?

There's always some overhead when writing to an user share, but for some reason some users see a much more large impact tan others, e.g. I get around 800MB/s vs 1GBs writing to user vs disk share, enabling direct i/o helps in some cases.

 

18 hours ago, j44rs4 said:

I also tried to write directly to disk1 which also gives me 1,12GB/s. How is this possible, when this is a SATA-3 6Gb/s disk?

Because first Unraid caches the writes to RAM, it will eventually drop to device speed if it's a large enough transfer.

Thx for your reply.

 

Unfortunately I've already activated Direct IO, with no better outcome.

 

I've tried serveral different disks (from nvme, SAS, SSD and SATA) as cache disk, but with the same performance.

 

Is seems like it's not using the cache drive correctly when writing to shared folder. Since I get better speed when writing directly to cache, than writing to shared folder through cache. 

 

1.PNG - Shows copy directly to cache

2.PNG - Shows copy to sharefolder

3.PNG -  Shows copy directly to disk1

 

11 minutes ago, j44rs4 said:

Is seems like it's not using the cache drive correctly when writing to shared folder.

Like mentioned there's always some overhead when using user shares, but the amount of the overhead can vary a lot with config/hardware used.

Is there no way I can tune the config/hardware?

 

If direct i/o doesn't help I don't know of anything else you can do other than trying different hardware, ideally hardware you already have availble, and that doesn't necessarily mean faster/better hardware, just different, though since SMB is single thread CPUs with better performance on a single core are usually better, i.e., a 3.5Ghz quad core is usually faster than an 8 core 2.5GHz CPU for this.

On 8/24/2020 at 2:28 AM, j44rs4 said:

Thx for your reply.

 

Unfortunately I've already activated Direct IO, with no better outcome.

 

I've tried serveral different disks (from nvme, SAS, SSD and SATA) as cache disk, but with the same performance.

 

Is seems like it's not using the cache drive correctly when writing to shared folder. Since I get better speed when writing directly to cache, than writing to shared folder through cache. 

 

1.PNG - Shows copy directly to cache

2.PNG - Shows copy to sharefolder

3.PNG -  Shows copy directly to disk1

 

We’re you able to find a work around or fix for this....

On 8/23/2020 at 8:38 AM, j44rs4 said:

enabled "Disk Shares"

Make sure you don't mix disks and user shares when moving / copying or you can lose data.

@Tekminute Sorry, no fix so far. the HW seems to be handling speed as you can see, but not when writing to cache. The only bottleneck I can think of must be the CPU (Intel Xeon CPU E5-2670 v2 @ 2.50GHz)

I have 64gigs of RAM, so that should be plenty enough.

 

But still I don't understand why writing to share through cache could make such a problem, while writing directly to all other disks, give me full speed.

Another test done this moment:

 

Writing to share Without cache drive, gives me a speed of 450-500MB/s, while activating the cache only give me 260MB/s.

 

Can someone explain to me why writing to share with cache enabled, is slower than writing directly to share?

 

I have 3 different cache disks I've been playing with. The first one, is a NVMe WDS250G3X0C, and the second one, is a SATA SSDd from Dell MZ7LH960HBJR0D3, and the third one is a SAS 300Gig disk 15000RPM.

 

All these disks can only perform a speed of 260MB/s

 

As for now, my system is slower with cache disk than without.

24 minutes ago, j44rs4 said:

Writing to share Without cache drive, gives me a speed of 450-500MB/s

Writes directly to the parity array, assuming HDDs, cannot be this fast.

Sorry forgot to mention that I did the test without parity disks.

You are totally correct.

 

With parity I'm down to 276MB/s.

 

6 hours ago, j44rs4 said:

With parity I'm down to 276MB/s.

Still not possible with HDDs. Maybe you mean Mb/s?

I'm talking MegaBytes (MB)

Hmmm, let med post you some pics then.

 

Pictures 1-3 is copy to share Test, without cache drive, on a 10Gb/s ethernet. As you can se on pic2, you'll see that I've disabled writing to cache.

 

If enable cache on the share Test, I get lower speed.

Are you sure that isn't just the RAM buffer? Does it stay at that speed all the way to the end of a very large file? How much RAM does your server have?

64Gigs of RAM

 

And that was a 20GB file.

 

But still not the same same, as I have on another system, with cache and 10Gbe

Edited August 31, 20205 yr by j44rs4

On 8/24/2020 at 2:22 PM, JorgeB said:

If direct i/o doesn't help I don't know of anything else you can do other than trying different hardware, ideally hardware you already have availble, and that doesn't necessarily mean faster/better hardware, just different, though since SMB is single thread CPUs with better performance on a single core are usually better, i.e., a 3.5Ghz quad core is usually faster than an 8 core 2.5GHz CPU for this.

 

Replaced my Intel Xeon E5-2670v2 2.5GHz with a Intel Xeon E5-2637v2 3.5GHz, and the writing is still the same.

 

Any other suggestions? Anyone?

On 9/3/2020 at 5:58 AM, j44rs4 said:

 

Replaced my Intel Xeon E5-2670v2 2.5GHz with a Intel Xeon E5-2637v2 3.5GHz, and the writing is still the same.

 

Any other suggestions? Anyone?

Any updates on this?  

Edited October 13, 20205 yr by SPOautos

Sorry, no.

 

I'm working out a new way to test, the speed.

 

I've figured out that Perc h710p controller only supports PCIe 2.0 even though they say, "When you flash your H710p to IT-mode, it will start to work as a PCIe 3.0"

Maybe that's for the PCIe adapter card and not the internal module.

https://www.dell.com/support/article/no-no/sln292279/list-of-poweredge-raid-controller-perc-types-for-dell-emc-systems?lang=en

I Also important to notice , is that this controller only supports 6Gb/s, so there is no way this controller can do 10Gb/s alone.

 

As for the m.2 disk I inserted, I used a PCIe-adapter, and still no perfomance.

 

I will try to add a H730P PCIe-card to my server, which indeed is a PCIe 3.0, and handles 12Gb/s.

I have 4x 2.5' SAS SSD, that also support 12Gb/s, and will try and see if that makes any impact on my performance. 

 

Will report back, when I'm done testing, so it might take a while. Hopefully before X-mas.

UPDATE.

 

After many ways of testing, I figured out that none of the suggestions mentioned in this thread had any impact on the system.

 

I also tried out with a Perc H730, to see if that could make any difference, since it has a PCIe 3.0 bus confirmed by DELL, and do support 12Gb/s bandwith. The test turned out to have no impact.

 

Did some more testing, and reset my BIOS to FACTORY settings, and still nothing.

 

Solution:

Made some minor changes in Bios, as you can see from posted pics.

1. Changed the System BIOS Settings -> System Profile Settings -> Performance

2. Changed the System BIOS Settings -> Processor Settings -> Dell Controlled Turbo - ENABLED

3. Changed the System BIOS Settings -> Integrated Devices -> Memory Mapped I/O above 4GB - ENABLED

 

Setting number 3. made the biggest impact.