Slow write speed to share with cache (avg 25-30 MB/s) - /mnt/user is slow, /mnt/cahe super fast

[ChrisK2]

I have set up a share that uses an SSD as cache. However, something seems to be very wrong. My writing speed to /mnt/user/share are around 25-30 Megabyte/s, even though I'm having an SSD cache and the "Use Cache Pool" setting at "Yes".

 

I noticed that my qBittorrent was always at a maximum of 25 Megabyte/s. I have 750 Mbit/s internet and always get 500 Mbit/s+ in speed tests and reaches speeds at 70 Megabyte/s+ on my other computer.

 

I tried to search and found out more people who seem to have had the same problem, but I couldn't find any solution. I've tried setting Tunable (md_write_method) to reconstruct write, but that doesn't affect speed. (I guess it shouldn't either, as the data is written to cache and the parity is not involved.)

 

My server specs:

Motherboard: Supermicro X12STH-LN4F

CPU: Intel Xeon E-2356G

Memory: 2*32 GB ECC RAM

Hard drives: 3*Seagate Exos X16 16 TB 7200 RPM (1 parity, 2 data) - Data is xfs encrypted

SSD cache: 1 TB Samsung 970 EVO Plus NVMe M.2 - SSD is btrfs encrypted
Unraid version: 6.10.0-rc4

 

I ran some dd tests and these are the results:

 

# dd if=/dev/zero of=/mnt/user/media/test-user bs=1K count=5000000 conv=fdatasync
5000000+0 records in
5000000+0 records out
5120000000 bytes (5.1 GB, 4.8 GiB) copied, 209.546 s, 24.4 MB/s

 

# dd if=/dev/zero of=/mnt/cache/media/test-cache bs=1K count=5000000 conv=fdatasync
5000000+0 records in
5000000+0 records out
5120000000 bytes (5.1 GB, 4.8 GiB) copied, 8.50469 s, 602 MB/s

 

# dd if=/dev/zero of=/mnt/disk1/media/test-disk1 bs=1K count=5000000 conv=fdatasync
5000000+0 records in
5000000+0 records out
5120000000 bytes (5.1 GB, 4.8 GiB) copied, 23.7756 s, 215 MB/s

 

I've verified that the dd test to /mnt/user is written to the share, and not disk.

 

I've attached the diagnostics, collected after I ran the dd tests. No reboot. Below are some images of my settings.

 

 

Disk Settings:

 

Disk 1:

 

Cache:

 

Share settings:

 

Global Share Settings:

tower-diagnostics-20220401-2004.zip

Edited April 1, 2022 by ChrisK2
Formatting

[ChrisK2]

Some other topics with same/similar problem, but without any solution.

 

 

 

 

Edited April 2, 2022 by ChrisK2

[ChrisK2]

Ran a dd with Tunable (enable Direct IO) turned to yes, with no difference at all:

 

# dd if=/dev/zero of=/mnt/user/media/test-user bs=1K count=5000000 conv=fdatasync
5000000+0 records in
5000000+0 records out
5120000000 bytes (5.1 GB, 4.8 GiB) copied, 189.436 s, 27.0 MB/s

 

Edited April 2, 2022 by ChrisK2

[ChrisK2]

Some more information, it seems that the shfs process is taking some CPU when doing the dd tests, if that's of any interest.

 

Both these screenshots from when running dd to /mnt/user/.

 

 

 

Edited April 2, 2022 by ChrisK2

[JorgeB]

Just a couple of observations, using dd to test /mnt/user doesn't always give relevant results, also it's known that user shares always have some extra overhead vs disk shares, some users see a much bigger difference than others, though 25MB/s would be extra slow, usually only users with 10GbE notice the difference. 

 

If you you transfer data from another PC to a share using cache do you also get 25MB/s?

[ChrisK2]

Thank you for your answe, @JorgeB!

 

I've setup my PC and ran tests against the share and directly against the cache. Both were actually at 96-98 Megabyte/s, so the problem doesn't seem to be the cache, after all. I also tried local dd, with larger block size, and that seemed to increase the speed as well. Can it be that the small block size, when running locally, increases I/O? And that's hard for the shfs to handle?

 

However, I still seem to be limited to 25-30 Megabyte/s in all torrent clients I've tried. qBittorrent, rTorrent, Deluge. From my other PC, I reach speeds at 60 Megabyte/s+ with no problem. Could it be that the torrent clients has high I/O, and that, as in the dd with a block size of 1kB, leads to poor performance?

[JorgeB]

20 hours ago, ChrisK2 said:

I also tried local dd, with larger block size, and that seemed to increase the speed as well. Can it be that the small block size, when running locally, increases I/O?

Like mentioned dd is not a reliable way to test that.

 

20 hours ago, ChrisK2 said:

However, I still seem to be limited to 25-30 Megabyte/s in all torrent clients I've tried. qBittorrent, rTorrent, Deluge.

Sorry, can't really help with that.