Cache overhead?

[skaterpunk0187]

I have been doing some testing with the cache. Drive is a Samsung 970 evo plus 500 GB NVMe new on 1/25/2020. Data being transferred is on a very fast NVMe drive as well. Data is a 27 GB mkv file. The 970 not set as cache drive mounted and shared with unassigned devices the file transfers at 1.10 GB/s. Forget to mention 10 GBE network and Unraid version 6.8.2. I set the 970 to be the cache drive transfer same file I get the speed of 450-500 MB/s say 470ish MB/s. Also no other data on cache drive Docker and VMs turned off and only one share set to use cache only for testing. I remove from cache mount and share with unassigned devices plugin again bam right back to 1.10 GB/s. I also formatted btrfs when used under unassigned devices so file system was the same for both. attached Pictures of both speed. Does the cache really have near a 50% performance overhead? This would not be noticed on a 1 GBE, just a heads up for anyone thinking of going 10 GBE. Not that 450 MB/s is bad you just wont get the full 1 GB/s speed.

P.S. I apologize if this has been discussed already I spent two days here reading forums and couldn't find anything 

[JorgeB]

There can be some overhead when writing to a user share, try enabling disk shares and writing to \\tower\cache to compare.

 

Also, enabling Direct IO can in some cases help with this issue ( Settings -> Global Share Settings).

[testdasi]

It's shfs (i.e. the magic behind /mnt/user). Any io going through shfs won't exceed about 450MB/s (about 500MB/s for multiple streams from my own testing), which people won't notice for SATA SSD and HDD but certainly will with NVMe cache.

 

The way to have your cake and eat it (i.e. have NVMe in cache + bypass shfs for network transfer) is to access /mnt/cache/ directly. This can be achieved by enabling disk share or creating custom smb config.

Also, for cache = only shares (e.g. appdata, vdisk etc.), you should point the docker / VM paths to /mnt/cache instead of /mnt/user.

Edited January 30, 2020 by testdasi

[JorgeB]

Any io going through shfs won't exceed about 450MB/s

That can vary with hardware/config used, I can usually get around 800/900MB/s on user shares. 

 

 

 

[testdasi]

1 hour ago, johnnie.black said:

That can vary with hardware/config used, I can usually get around 800/900MB/s on user shares. 

Hmmm and here I am thinking my 450MB/s was already improved from the 6.8.0 update.

Back to more tweaking. 😅

[skaterpunk0187]

11 hours ago, testdasi said:

The way to have your cake and eat it (i.e. have NVMe in cache + bypass shfs for network transfer) is to access /mnt/cache/ directly. This can be achieved by enabling disk share or creating custom smb config.

This does achieve the transfer speed of over 1GB/s. The down side is the mover wont move the files, or if it does not the the right directories I would assume. Unless you created the same directory structure for example \\tower\cache\directory\testdirectory1 then transfer file to testdirectory1, then when mover runs it will be moved to /mnt/usr/directory/testdirectory1.

11 hours ago, johnnie.black said:

That can vary with hardware/config used, I can usually get around 800/900MB/s on user shares. 

 

 

 

What's your hardware setup and config? My hardware is older enterprise but its no slouch.

[testdasi]

32 minutes ago, skaterpunk0187 said:

This does achieve the transfer speed of over 1GB/s. The down side is the mover wont move the files, or if it does not the the right directories I would assume. Unless you created the same directory structure for example \\tower\cache\directory\testdirectory1 then transfer file to testdirectory1, then when mover runs it will be moved to /mnt/usr/directory/testdirectory1.

The mover will move the files if the share is set to cache = Yes.

[JonathanM]

2 hours ago, skaterpunk0187 said:

Unless you created the same directory structure for example \\tower\cache\directory\testdirectory1 then transfer file to testdirectory1, then when mover runs it will be moved to /mnt/usr/directory/testdirectory1.

All files in /mnt/cache/directory are already in /mnt/user/directory, no mover action needed. If the 'directory' user share is set to Cache : Yes, then when mover runs the files will be moved from /mnt/cache/directory to /mnt/diskX/directory, where the diskX used is determined by split level, allocation, min free space, includes or excludes. The files will always appear in /mnt/user/directory regardless of whether they are on the cache drive or one of the array disks.

 

All root folders on all array disks and cache pool are user shares, and will appear in /mnt/user.

[JorgeB]

9 hours ago, skaterpunk0187 said:

What's your hardware setup and config?

Supermicro X11SSM-F, G4560, 6 SSD RAID5 cache pool

[JorgeB]

Couple of more observations, regarding hardware, and obviously besides a fast device/pool, the only other thing you want is an high clock CPU with a fast single thread rating, Samba is single threaded, so e.g. a fast dual core CPU is better then a low clock 8 core Xeon.

 

Also, and since 99% of the time my transfers to this server are in the other direction, from Unraid, not to Unraid, it's been some time since I've paid attention to write speed, fuse overhead appears to be more pronounced with v6.8.x compared to v6.7.x, did a quick test and currently I'm getting more like 700MB/s write speed to a user share, I remember getting around 800/900MB/s with v6.7.x.

 

User share vs disk share

 

 

 

 

[skaterpunk0187]

7 hours ago, johnnie.black said:

Couple of more observations, regarding hardware, and obviously besides a fast device/pool, the only other thing you want is an high clock CPU with a fast single thread rating, Samba is single threaded, so e.g. a fast dual core CPU is better then a low clock 8 core Xeon.

Yes that is true, but if the CPU speed was my bottleneck it would also be the bottleneck using samba to transfer to an unassigned device or the cache directly. Using htop to monitor CPU usage. Transfering a 27gb file to \\tower\media\movies @450ish MB/s no core spikes over 25% on CPU. Transfering same 27gb file to \\tower\cache\media\movies @1.03 GB/s CPU get one core to spike to 70-75%. My CPU clock speed. I also have an identical server other then RAM and CPU.

RAM is only 64GB and CPU is 2x Xeon E5-2667 v2 base clock 3.30GHz and boosts up to 4GHz. Ran same test exact same results for speed and CPU usage. I/m not complaining 450 MB/s is still 400% faster then 1gbe.

7 hours ago, johnnie.black said:

Also, and since 99% of the time my transfers to this server are in the other direction, from Unraid,

That's gotta be rough going that direction. I hope your array is all SSD. Unraid is only my archival/backup storage and Plex server.<--Thats why I transfer such large files to Unraid.

[JorgeB]

19 minutes ago, skaterpunk0187 said:

That's gotta be rough going that direction. I hope your array is all SSD.

Not for this one, but transfers are mostly from the large SSD cache pool, HDD array is for streaming video only, so normal HDD read speeds are enough.