Speed with and without cache

[dnLL]

I have an array with 4x4TB WD Red drives (CMR) and 2x500GB WD Blue SSDs, all hooked up to the same SATA controller (onboard).

 

Here is my "issue": it's faster to write on shares with cache disabled. For example, I have a share named qbittorrent that has cache enabled and another named www with cache disabled. Copying the same ~2GB file to both, starting with the share with cache off, it copied 25% faster than on the share with cache on.

 

root@server:~# ls -l /mnt/*/qbittorrent/myfile*
-rw-rw-rw- 1 nobody users 2910110155 Jan  2  2016 /mnt/cache/qbittorrent/myfile.zip
-rw-rw-rw- 1 nobody users 2910110155 Jan  2  2016 /mnt/user/qbittorrent/myfile.zip
root@server:~# ls -l /mnt/*/www/myfile*
-rw-rw-rw- 1 nobody users 2910110155 Jan  2  2016 /mnt/disk3/www/myfile.zip
-rw-rw-rw- 1 nobody users 2910110155 Jan  2  2016 /mnt/user/www/myfile.zip
-rw-rw-rw- 1 nobody users 2910110155 Jan  2  2016 /mnt/user0/www/myfile.zip

What's going on?

 

I do have md_write_method set to rewritable_method, which is specifically why I'm doing these tests as I'd like to maybe try reverting it to default to spare my drives a little bit.

[dnLL]

Raw numbers.

 

With cache: 83.04 MB/s

Without cache: 109.57 MB/s

 

This is on gigabit so I'm pretty much getting to the limit of the network (which should be 125 MB/s for gigabit). Also note that I timed the transfer, not really trusting anything else.

 

Oh and the two SSD cache drives are mirrored... maybe that's the issue?

Trim is done on a daily basis at 1:00am.

Edited July 27, 2020 by dnLL

[JorgeB]

New beta aligns partitions for non rotational devices on the 1MiB boundary, you can try that, it can be considerably faster for some devices, requires wiping current pool though.

[dnLL]

New beta aligns partitions for non rotational devices on the 1MiB boundary, you can try that, it can be considerably faster for some devices, requires wiping current pool though.

Those SSDs are rated for over 500MB/s r/w performances. You are saying that Unraid could be bottlenecking my drives to the point where they can't even maximize Gigabit bandwidth?

What about turbo write, should I disable it since I have a cache? And what about the internal HDD cache that we can also write to, any reason not to use it considering I do have a UPS?

[JorgeB]

14 minutes ago, dnLL said:

You are saying that Unraid could be bottlenecking my drives to the point where they can't even maximize Gigabit bandwidth?

I'm saying that some SSDs suffer from bad performance when not 1MiB aligned, especially when used on a pool.

 

15 minutes ago, dnLL said:

What about turbo write, should I disable it since I have a cache?

Turbo write only maters when writing to the array, though it will mater when moving the data from cache to the array.

 

16 minutes ago, dnLL said:

And what about the internal HDD cache that we can also write to, any reason not to use it considering I do have a UPS?

Not sure what you mean, if you mean the small HDD RAM cache it will already be used, unless it's disable on the device.

 

[dnLL]

12 hours ago, johnnie.black said:

I'm saying that some SSDs suffer from bad performance when not 1MiB aligned, especially when used on a pool.

 

Turbo write only maters when writing to the array, though it will mater when moving the data from cache to the array.

 

Not sure what you mean, if you mean the small HDD RAM cache it will already be used, unless it's disable on the device.

 

I did some more tests, with or without cache I'm getting around 115 MB/s read and write with CrystalDiskMark. That's pretty much gigabit's practical limitation. I read that WD Red drives can do more than that so I guess I'm just getting bottlenecked by the network and that's all. That pretty much means that my cache is only useful for loading times for VM disks and such.