Unraid bypassing cache midway through transfer when cache is only half full

[gregnostic]

I'm in the process of ripping a bunch of my Blu-rays and I've run into an issue where Unraid is bypassing my cache and writing files directly to the array when the cache reaches about half-full. As long as the cache remains at that capacity, ALL new file transfers are bypassing the cache and being written directly to the array.

 

My cache drives are WD Blue 2TB SSDs, and I've been using them for over a year with no issues. I've been able to write nearly the full 2TB to the cache drives in the past (the last time I recall doing this was likely on 6.7.x).

 

I'm including a screenshot of the most recent example of this happening. When I took the screenshots, I was trying to transfer over 1TB of Blu-ray rips to the server. The first half of the transfer went along at a reasonable clip--about 105 MB/s consistently for about two hours. Then, all of a sudden, the transfer falls off to anywhere from 20 MB/s to 55 MB/s. I look in Unraid and see that cache drive is no longer receiving writes (at least not from this transfer) and it's going to an array disk instead.

 

Note in the screenshot that I've included a LibreSpeed speedtest result (LibreSpeed is hosted on Unraid in Docker). The file transfer is still happening and the speedtest was being run with that going on in the background. This speedtest thus shows spare throughput between the client and server, so it's not a networking issue.

 

I've also seen this issue happening in recent days trying to rip discs directly to Unraid (my usual workflow). When it reaches roughly ~960 GB of storage space used, new files ripped start getting written to an array disk. Because that can't keep up with the data from three simultaneous rips, I started ripping to my local machine, hence this batch transfer.

 

All these transfers are happening over SMB.

 

Server: Unraid 6.8.3, Xeon E5-2630v4 (10-core, 2.2GHz), SuperMicro X10DRi w/Intel i350 LAN, 32GB RAM, etc.

Client: Windows 10 v1910, Ryzen 3800X, 16 GB RAM, 2x 4TB HGST NAS in RAID1

 

I'm happy to provide any other relevant information if it'll help figure out what the heck is going on here.

Edited May 7, 2020 by gregnostic
Fixing typo in title

[trurl]

7 minutes ago, gregnostic said:

provide any other relevant information

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post.

[gregnostic]

Whoops. I knew I forgot something!

consul-diagnostics-20200506-2159.zip

[itimpi]

You might want to check what value you have set for Minimum Free Space for the shares in question?     It is not obvious, but the larger of the global share setting and the individual share setting is used to decide if the file should go to the cache.    It is also a good idea to use the suffixes rather than entering just a numeric value (especially as that value is not an absolute number but the number of KB) as it can be easy to get the number of zeroes wrong.

 

There has been a feature request raised that the User Share value for Minimum Free space should not apply to the cache, and that the cache should only use the value set under Settings >> Global Share Settings, but I have no idea if that is going to happen.

[gregnostic]

14 hours ago, itimpi said:

You might want to check what value you have set for Minimum Free Space for the shares in question?     It is not obvious, but the larger of the global share setting and the individual share setting is used to decide if the file should go to the cache.

Thanks for the answer. I'm going to have to think about how I want to approach this problem because I don't want to fill up my disks to the brim. If I do what that field's help text says and use the size of the largest files I deal with (about 50 GB), that's 99.5% usage on a 10 TB disk, which is generally a bad thing to do to a hard drive. (That's why I set the value as high as I do.)

 

I had actually briefly considered this as a possibility but I dismissed it because the help text on the Minimum Free Space field has this to say:

Quote

The minimum free space available to allow writing to any disk belonging to the share.

To my mind, that doesn't include the cache. It makes no sense to me whatsoever to include the cache in this value.

 

14 hours ago, itimpi said:

It is also a good idea to use the suffixes rather than entering just a numeric value (especially as that value is not an absolute number but the number of KB) as it can be easy to get the number of zeroes wrong.

I have "1 TB" as the input. It displays in the UI as "1 TB" but it apparently stores the value as a plain number.

 

14 hours ago, itimpi said:

There has been a feature request raised that the User Share value for Minimum Free space should not apply to the cache, and that the cache should only use the value set under Settings >> Global Share Settings, but I have no idea if that is going to happen.

I sure hope this happens. This behavior makes next to no sense. Worse, the help text is simply wrong.

[Baoh]

I have run into this problem as well.  Here is the link to the current feature request though it is perhaps not as clear as to the problem statement as this thread is: https://forums.unraid.net/topic/90303-a-shares-minimum-free-space-setting-should-not-apply-if-it-uses-the-cache-drive/?tab=comments#comment-838175&searchlight=1

 

For what it's worth, this is a huge problem as I have 512gb of cache (2x512gb ssd), but ~20TB of storage.  A reasonable minimum free space parameter for my shares might be half or all of my cache.

Edited May 7, 2020 by Baoh

[gregnostic]

I actually just finished adding a response to that very thread.

 

If you have a 16 TB hard drive and want to set a comfortable 5% space free, you're looking at setting a fixed value of 800 GB. If your cache disk is only 512 GB, it's never going to touch your cache and always write directly to the 16 TB drive. Applying a fixed byte value to a setup with variable disk sizes doesn't work.

[John_M]

3 hours ago, gregnostic said:

If I do what that field's help text says and use the size of the largest files I deal with (about 50 GB), that's 99.5% usage on a 10 TB disk, which is generally a bad thing to do to a hard drive.

50 GB free on an XFS-formatted disk that's essentially used in read-only mode is plenty. I fill up disks on the basis that once full the contents will seldom if ever change and manually add files until they have as little as 10 GB free and it's fine. There was an update to XFS some years ago that used up some of the free space, but only a small percentage of the 10 GB.

[gregnostic]

23 hours ago, John_M said:

50 GB free on an XFS-formatted disk that's essentially used in read-only mode is plenty. I fill up disks on the basis that once full the contents will seldom if ever change and manually add files until they have as little as 10 GB free and it's fine. There was an update to XFS some years ago that used up some of the free space, but only a small percentage of the 10 GB.

Ultimately, it's true that I could change how I work with files to work around this problem. I could create two media shares--one for files that change and one for files that don't--and assign those shares to different disks to maximize usage on the disks with purely static data. But if I did that, I'd have to stop organizing files based on what they are and start organizing them based on how they behave. And frankly, that's not appealing to me.

 

I like being able to have a media share where I can put my media files, which sometimes change (e.g. recorded TV shows that get deleted after being watched). I like not having to worry about whether I plan to keep something forever or just for a while.

 

If Unraid allowed me to set a minimum free space based on a percentage rather than a fixed byte value, I'd be able to retain all that flexibility while giving up a minimal amount of storage space. To my mind, losing just 5% of my storage space is a fair trade for that. It might not be for everyone, and that's fine. But I'd really like to have the option.

[trurl]

On 5/7/2020 at 4:35 PM, gregnostic said:

I actually just finished adding a response to that very thread.

I have responded on that thread.