(SOLVED) Cache: Performance and max filesize


Toskache

13 posts in this topic Last Reply

Recommended Posts

A few weeks ago I upgraded my PC and the unraid-Server with 10G network interfaces.

Until then, I had only configured one SSD as a cache disk. This cache disk had the file system "xfs" and was of course the bottleneck to exhaust the 10G performance.

So I installed an identical, second SSD and configured it as a cache pool (btrfs, raid0).
Strangely, this had no effect on the write performance over the 10G interface. I get max ~ 300 MB/s when transferring a large file to a share (with cache = prefer). Is there anything else I'm missing here? Diagnostics data is attached.

 

During the test with dd:
 

sync; dd if=/dev/zero of=/mnt/cache/testfile.img bs=5G count=1; sync

I realized, that the max. filesize for the cache is 2G. Is that correct? I used to write larger files to the unraid-NAS, so how will the cache behave?

nas.fritz.box-diagnostics-20201012-1601.zip

Link to post
11 minutes ago, JorgeB said:

Have you done an iperf test? See what you get for a single stream.

The iperf-performance seems to bee ok:

 

toskache@10GPC ~ % iperf3 -c 192.168.2.4
Connecting to host 192.168.2.4, port 5201
[  5] local 192.168.2.199 port 50204 connected to 192.168.2.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   1.00-2.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   2.00-3.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   3.00-4.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   4.00-5.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   5.00-6.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   6.00-7.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   7.00-8.00   sec  1.15 GBytes  9.89 Gbits/sec
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec
[  5]   9.00-10.00  sec  1.15 GBytes  9.89 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  sender
[  5]   0.00-10.00  sec  11.5 GBytes  9.89 Gbits/sec                  receiver

iperf Done.

 

Edited by Toskache
Link to post
19 minutes ago, Toskache said:

I realized, that the max. filesize for the cache is 2G. Is that correct?

No, what made you think that limit applied?   The limit is going to be imposed by available space on the cache.

Link to post
26 minutes ago, itimpi said:

No, what made you think that limit applied?   The limit is going to be imposed by available space on the cache.

Ok, it was probably a mistake of mine. I was just surprised that files> 2G were never generated during the performance tests with "dd".

Link to post

Sorry for the late response.

@JorgeB I performed a retest with iperf:

unraid as iperf3-Server:

toskache@Hacky ~ % iperf3 -c 192.168.2.4
Connecting to host 192.168.2.4, port 5201
[  5] local 192.168.2.26 port 53229 connected to 192.168.2.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   519 MBytes  4.35 Gbits/sec
[  5]   1.00-2.00   sec   504 MBytes  4.22 Gbits/sec
[  5]   2.00-3.00   sec   491 MBytes  4.12 Gbits/sec
[  5]   3.00-4.00   sec   498 MBytes  4.17 Gbits/sec
[  5]   4.00-5.00   sec   499 MBytes  4.18 Gbits/sec
[  5]   5.00-6.00   sec   437 MBytes  3.66 Gbits/sec
[  5]   6.00-7.00   sec   384 MBytes  3.22 Gbits/sec
[  5]   7.00-8.00   sec   424 MBytes  3.56 Gbits/sec
[  5]   8.00-9.00   sec   472 MBytes  3.96 Gbits/sec
[  5]   9.00-10.00  sec   501 MBytes  4.20 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  4.62 GBytes  3.97 Gbits/sec                  sender
[  5]   0.00-10.00  sec  4.62 GBytes  3.97 Gbits/sec                  receiver

iperf Done.

unraid as iperf3 client:

toskache@Hacky ~ % iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.2.4, port 45284
[  5] local 192.168.2.26 port 5201 connected to 192.168.2.4 port 45286
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.02 GBytes  8.73 Gbits/sec
[  5]   1.00-2.00   sec  1.04 GBytes  8.92 Gbits/sec
[  5]   2.00-3.00   sec  1.04 GBytes  8.91 Gbits/sec
[  5]   3.00-4.00   sec  1.04 GBytes  8.95 Gbits/sec
[  5]   4.00-5.00   sec  1.04 GBytes  8.97 Gbits/sec
[  5]   5.00-6.00   sec  1.05 GBytes  8.98 Gbits/sec
[  5]   6.00-7.00   sec  1.04 GBytes  8.94 Gbits/sec
[  5]   7.00-8.00   sec  1.04 GBytes  8.92 Gbits/sec
[  5]   8.00-9.00   sec  1.04 GBytes  8.94 Gbits/sec
[  5]   9.00-10.00  sec  1.04 GBytes  8.93 Gbits/sec
[  5]  10.00-10.01  sec  6.23 MBytes  9.08 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  10.4 GBytes  8.92 Gbits/sec                  receiver

 

So there is a very asymmetric performance.

At the same time you can see in the unraid dashboard that there are drops on the recipient side:

image.png.c91693a8aed583dc5ccd880de06e84d9.png

I tried allready different cables an switch-ports. Both interfaces (unraid and PC) are working with an MTU of 1500.

Unraid is on version 6.9.0-beta30.

I don't know where the drops came from. 😞

 

I also enabled disk-shares and tested the cache-pool directly:

5GB-File:

image.png.b09097d7ffa25de0fa06ad989beb88d4.png

 

That seems to be the perfomance of a single SATA-SSD even though the cache-pool is set to raid0:

image.png.744993f7182d7e713cf56d04f140da6a.png

 

At least the reading speed should be much higher, right?

 

And: Thank you for your support!

 

 

 

 

Link to post
On 10/12/2020 at 4:08 PM, Toskache said:

During the test with dd:
 


sync; dd if=/dev/zero of=/mnt/cache/testfile.img bs=5G count=1; sync

I realized, that the max. filesize for the cache is 2G. Is that correct?

Never seen anyone go for a block size of 5G before, that’s literally orders of magnitude larger than commonly seen...

 

How about a reasonable block size and increased count to match instead?

Link to post
1 hour ago, gubbgnutten said:

Never seen anyone go for a block size of 5G before, that’s literally orders of magnitude larger than commonly seen...

You are completely right! I Testet with various combinations of bs and count. The last one I copied in the thread realy not reasonable.

With

dd if=/dev/zero of=/mnt/cache/testfile.img bs=32K count=32000; sync

I get 490 MB/s which is the ca. 90% of the double-performance of one single SATA-SSD. So the cache performance seems to be fine. For more performance i have to go with M.2/NVMe.

 

The only problem now are the dropped RX packets. With 6.9.0-beta25 i did not see any droped rx-packets. I posted it in the beta-thread.

Link to post
  • Toskache changed the title to (SOLVED) Cache: Performance and max filesize

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.