Writing to cache fast, writing to cached share slow

[BrttClne22]

Hoping one of you smart souls can shed some light on an issue I've found today. I've done a lot of Googling, but no dice.  I had been experiencing some iowait lockups when radarr/sonarr moved a file from an unassigned disk to my 'media' share with the cache setting to 'Yes'.  I confirmed that turning the cache setting to 'Off', mitigated the iowait issues but I would prefer it be written to the cache first.

 

Using a write speed test script posted by @johnnie.black, I ran two separate tests against my cache drive.

 

When formatted as BTRFS (sorry, I don't have the exact results for these):

/mnt/cache/test.dat     450 MBps

/mnt/user/media/test.dat     60 MBps

 

I ditched BTRFS, thinking that may be the problem. I didn't really have a use case for it anyways as I only have a single cache drive atm. Here are the same tests with XFS

writing 10240000000 bytes to: /mnt/cache/test.dat
3681144+0 records in
3681144+0 records out
3769491456 bytes (3.8 GB, 3.5 GiB) copied, 5.00031 s, 754 MB/s
6741953+0 records in
6741953+0 records out
6903759872 bytes (6.9 GB, 6.4 GiB) copied, 10.0016 s, 690 MB/s
9420809+0 records in
9420809+0 records out
9646908416 bytes (9.6 GB, 9.0 GiB) copied, 15.0049 s, 643 MB/s
10000000+0 records in
10000000+0 records out
10240000000 bytes (10 GB, 9.5 GiB) copied, 16.1416 s, 634 MB/s
write complete, syncing
removed '/mnt/cache/test.dat'

 

writing 10240000000 bytes to: /mnt/user/media/test.dat
327258+0 records in
327258+0 records out
335112192 bytes (335 MB, 320 MiB) copied, 5.00024 s, 67.0 MB/s
664063+0 records in
664063+0 records out
680000512 bytes (680 MB, 648 MiB) copied, 10.0015 s, 68.0 MB/s
993568+0 records in
993568+0 records out
1017413632 bytes (1.0 GB, 970 MiB) copied, 15.0029 s, 67.8 MB/s
1321207+0 records in
1321207+0 records out
1352915968 bytes (1.4 GB, 1.3 GiB) copied, 20.0045 s, 67.6 MB/s
1652369+0 records in
1652369+0 records out
1692025856 bytes (1.7 GB, 1.6 GiB) copied, 25.0059 s, 67.7 MB/s
1988734+0 records in
1988734+0 records out
2036463616 bytes (2.0 GB, 1.9 GiB) copied, 30.0073 s, 67.9 MB/s
2316253+0 records in
2316253+0 records out
2371843072 bytes (2.4 GB, 2.2 GiB) copied, 35.0089 s, 67.7 MB/s
2649459+0 records in
2649459+0 records out
2713046016 bytes (2.7 GB, 2.5 GiB) copied, 40.0102 s, 67.8 MB/s
2984946+0 records in
2984946+0 records out
3056584704 bytes (3.1 GB, 2.8 GiB) copied, 45.0115 s, 67.9 MB/s
3321507+0 records in
3321507+0 records out
3401223168 bytes (3.4 GB, 3.2 GiB) copied, 50.0129 s, 68.0 MB/s
3651156+0 records in
3651156+0 records out
3738783744 bytes (3.7 GB, 3.5 GiB) copied, 55.0142 s, 68.0 MB/s
3980471+0 records in
3980471+0 records out
4076002304 bytes (4.1 GB, 3.8 GiB) copied, 60.0157 s, 67.9 MB/s
4314105+0 records in
4314105+0 records out
4417643520 bytes (4.4 GB, 4.1 GiB) copied, 65.0171 s, 67.9 MB/s
4647805+0 records in
4647805+0 records out
4759352320 bytes (4.8 GB, 4.4 GiB) copied, 70.0184 s, 68.0 MB/s
4981473+0 records in
4981473+0 records out
5101028352 bytes (5.1 GB, 4.8 GiB) copied, 75.02 s, 68.0 MB/s
5315667+0 records in
5315667+0 records out
5443243008 bytes (5.4 GB, 5.1 GiB) copied, 80.0215 s, 68.0 MB/s
5645479+0 records in
5645479+0 records out
5780970496 bytes (5.8 GB, 5.4 GiB) copied, 85.0229 s, 68.0 MB/s
5974952+0 records in
5974952+0 records out
6118350848 bytes (6.1 GB, 5.7 GiB) copied, 90.0242 s, 68.0 MB/s
6310587+0 records in
6310587+0 records out
6462041088 bytes (6.5 GB, 6.0 GiB) copied, 95.0256 s, 68.0 MB/s
6638667+0 records in
6638667+0 records out
6797995008 bytes (6.8 GB, 6.3 GiB) copied, 100.027 s, 68.0 MB/s
6965658+0 records in
6965658+0 records out
7132833792 bytes (7.1 GB, 6.6 GiB) copied, 105.029 s, 67.9 MB/s
7297553+0 records in
7297553+0 records out
7472694272 bytes (7.5 GB, 7.0 GiB) copied, 110.03 s, 67.9 MB/s
7625676+0 records in
7625676+0 records out
7808692224 bytes (7.8 GB, 7.3 GiB) copied, 115.033 s, 67.9 MB/s
7953216+0 records in
7953216+0 records out
8144093184 bytes (8.1 GB, 7.6 GiB) copied, 120.035 s, 67.8 MB/s
8282188+0 records in
8282188+0 records out
8480960512 bytes (8.5 GB, 7.9 GiB) copied, 125.037 s, 67.8 MB/s
8613901+0 records in
8613901+0 records out
8820634624 bytes (8.8 GB, 8.2 GiB) copied, 130.04 s, 67.8 MB/s

Just to prove that it is in fact set to use the cache drive (I also double checked the file system of /mnt/cache/media as it was writing):

 

 

Diagnostics attached. Thanks for your help!

 

zeus-diagnostics-20180515-2039.zip

[John_M]

Your experience reminds me of this thread.

 

 

In practice I see no actual problem. Maybe the test is somehow not valid.

[BrttClne22]

Thanks for pointing that out, very interesting thread. After reading it, I did a write test of a 25 GB file from my Windows 10 VM (running on a separate NVME drive) to the media share (Samsung Pro SSD) and speeds were VERY similar to the /mnt/cache test. That puts my mind a little more at ease.