File transfer limited to size of cache when cache is set on a share

[aglyons]

I thought about how to go about searching for this in the forum and couldn't come up with anything. My apologies if this is a rehash.

 

I have two 250GB SSD cache drives in a btrfs pool. Total 500GB. If you try to transfer in 1.2TB of data, Windows complains that there isn't enough space. Regardless of the fact that the array is 60TB in size.

 

Is this as expected or is this something that is being looked into?

[JonathanM]

Each data drive and pool is a distinct file system that doesn't allow single files to span multiple drives or pools. You need to set the minimum free space so Unraid knows to select the next preferred destination after that free space is reached. Alternatively you could set that specific share to go directly to the array instead of going to the cache first and then moving later. If you are regularly filling the cache pool, you probably would be better off going directly to the array, or massively expanding the cache pool size.

[trurl]

38 minutes ago, aglyons said:

I have two 250GB SSD cache drives in a btrfs pool. Total 500GB.

default is btrfs raid1, so unless you have changed to raid0 you have a mirror with total capacity 250GB.

 

Attach diagnostics to your NEXT post in this thread

[aglyons]

So I do have a min free space set for drives. In this scenario, I was copying a folder with multiple files totaling 1.2TB. Windows saw that there was only 500GB of space on the target share because it was the cache that it was looking at. 

[aglyons]

Just now, trurl said:

default is btrfs raid1, so unless you have changed to raid0 you have a mirror with total capacity 250GB.

 

Attach diagnostics to your NEXT post in this thread

I changed it.

[trurl]

Each pool (cache) has a Minimum Free setting. This doesn't control how much space is kept free, it controls how much space a pool must have remaining for Unraid to choose the pool. If the pool doesn't have Minimum remaining, Unraid will choose an array disk (overflow) for cache-prefer or cache-yes shares.

 

Each user share also has a Minimum Free setting. If a disk has less than Minimum, Unraid will choose another disk as long as Split Level and Included disks allow it.

 

If you want more specific advice based on your configuration

6 minutes ago, trurl said:

Attach diagnostics to your NEXT post in this thread

 

[aglyons]

knoxx-diagnostics-20220508-2339.zip

[trurl]

You have a user share O-------e set as cache-prefer. Prefer means you prefer that the share stays on cache. Is that what you intend?

 

Your cache pool has no Minimum, some of your shares have no Minimum, and the shares that do have Minimum may not have that set high enough.

 

Minimum should be set larger than the largest file you expect to write to the share (or pool)

 

11 hours ago, trurl said:

If the pool doesn't have Minimum remaining, Unraid will choose an array disk (overflow) for cache-prefer or cache-yes shares.

 

Each user share also has a Minimum Free setting. If a disk has less than Minimum, Unraid will choose another disk as long as Split Level and Included disks allow it.

Unraid has no way to know how large a file will become when it chooses a disk (or pool) for it. If a disk (or pool) has more than Minimum, it can be chosen, and if the file won't fit the write will fail.

 

[trurl]

For example

 

A user share has Minimum Free 10G. A disk has 15G free. Since that is more than Minimum, the disk can be chosen (depending on Included/Excluded, Split Level, Allocation Method). If you write a 20G file, and the disk is chosen, the write will fail due to out-of-space.

 

Another example

 

A user share has Minimum Free 20G. A disk has 30G free. Since that is more than Minimum, the disk can be chosen (depending on Included/Excluded, Split Level, Allocation Method). If you write a 25G file, and the disk is chosen, the write will succeed since it will fit, and after that the disk will have 5G free, less than Minimum, so another disk will be chosen next time (if Split Level, Include/Exclude allows another).

[aglyons]

3 hours ago, trurl said:

For example

 

A user share has Minimum Free 10G. A disk has 15G free. Since that is more than Minimum, the disk can be chosen (depending on Included/Excluded, Split Level, Allocation Method). If you write a 20G file, and the disk is chosen, the write will fail due to out-of-space.

 

Another example

 

A user share has Minimum Free 20G. A disk has 30G free. Since that is more than Minimum, the disk can be chosen (depending on Included/Excluded, Split Level, Allocation Method). If you write a 25G file, and the disk is chosen, the write will succeed since it will fit, and after that the disk will have 5G free, less than Minimum, so another disk will be chosen next time (if Split Level, Include/Exclude allows another).

 

Ok, I think I figured out what is going on here. There STILL is an issue with Unraid in a way¹.

 

I use Teracopy as my file copy handler. An option that it has is to check free space before a copy/move procedure. The built in Windows copy/pate I think does the same thing. It checks for free space prior to starting the operation.

 

Unraid reports back the capacity of the cache, not the capacity of the share being transferred to. Copying a test folder of 1348GB (1.22TB on-disk) returned an error needing an extra 800GB of space. 500GB (cache size) + 800GB (additional space needed error) = 1300GB 

 

 

IMO, Unraid should be returning the free space available on the share not the cache size. 

 

I'm curious with regards to how the cache system works. I understand the mover and that it runs on a schedule. But, is the mover/cache system smart enough to monitor the current capacity of the cache and start moving files if capacity is getting too high, before the schedule kicks in?

 

¹ **UPDATE**

As I write this and poke around the UI while testing, I noticed an interesting quirk which probably answers my own question. I noticed that if you have a share set to use cache (not 'cache only' but any cache setting), the available space shown in the share panel shows the available capacity of the cache, not the array. The effect of returning the cache capacity rather than the array capacity means that you will hit a hard limit on the size of the file transfer you want to accomplish. Exactly the situation I am hitting now.

 

As I understand it, the purpose of using cache is to help speed up transfers. Like a fast loading holding space. This seems to work differently.

 

The problem is if you turn cache off, transfer rates are terrible. I upgraded to 10GBe and with no cache, transfers will start in the 500MB/s range but very quickly plummet to < 90MB/s even with Turbo Write enabled. On a pipe that can handle 1250MB/s and SATA3 drives that can do 750MB/s (both in theory, yes), 90MB/s is a far far cry from potential. I would be happy with seeing a sustained 400-500MB/s.

Edited May 9, 2022 by aglyons

[JorgeB]

13 minutes ago, aglyons said:

I noticed that if you have a share set to use cache (not 'cache only' but any cache setting), the available space shown in the share panel shows the available capacity of the cache, not the array.

This will happen if the share only exists on cache, create the share with use cache="yes" and run the mover once and it will also exist in the array.

[aglyons]

2 minutes ago, JorgeB said:

This will happen if the share only exists on cache, create the share and run the mover once and it will also exist in the array.

OK, that's good to know. BUT, the same thing still happens. Unraid reports the capacity of the cache.

 

I just changed one share to use cache and ensured that it showed the full array capacity in the shares page.

 

I tried the copy/paste again. Same error as before, approx 800GB more space needed to copy the pasted files.

 

[aglyons]

It's interesting that mapped SMB connections in Windows does show the correct array capacity.

[aglyons]

OK, so the definitive answer....

 

https://wiki.unraid.net/Cache_disk#Amount_of_data

 

"Amount of data

The final consideration in choosing a cache drive is to think about the amount of data you expect to pass through it. If you write ~10 GBs per day, then any drive 10 GB or larger will do (a 30 GB SSD may be a good fit in this case). If you write 100 GB in one day every few weeks, then you will want a cache drive that is larger than 100 GB. If you attempt a data transfer that is larger than the size of your cache drive, the transfer will fail."

[trurl]

Best approach is to not use cache for initial data load. If you intend to write more than cache can hold, don't cache. It really just gets in the way since files have to be moved from cache. Default mover schedule is once per day. Setting it more often won't really help because it is impossible to move from fast cache to slow array as fast as you can write to cache

[itimpi]

3 hours ago, aglyons said:

If you attempt a data transfer that is larger than the size of your cache drive, the transfer will fail."

This statement is not true!    You can transfer more data than the size of the cache disk as long as:

• you have the Use Cache setting for the shares to end up on the array but go via the cache set to Yes, or Prefer for files you want to finally end up on the cache if room permits but go the arrsy otherwise.
• You have the Minimum Free Space value for the cache set to be larger than any single file to be transferred.  A typical recommendation is twice the size of the largest file to give some headroom.   You do NOT want the default value of 0 for this setting.   Personally I do not think a value of 0 should be allowed by Unraid and the default should be something like 10% of the size of the pool as a better default that would suit most people.
• Files are being transferred one at a time (which would be typical).  If multiple files are being transferred in parallel the the Minimum Free Space value must be adjusted to a larger value to allow for the number of parallel transfers that can occur.

Aa long as this is true, then when Unraid sees the free space fall below the Minimum Free Space setting for the cache it will start by-passing the cache for new files and write them directly to the array.    It is true, however, that there is not much point in having a cache drive that is smaller than the amount of new data you write between mover runs.
 

For the initial load onto a new Unraid system it is advantageous from a performance perspective to not have a parity drive assigned as most files will probably end up by-passing the cache so you do not want the performance penalty of updating parity (unless you are prepared to accept it to keep the data on the array drives protected from the outset).

 

 

 

 

[JorgeB]

Like itimpi mentioned, you can start a transfer larger than cache as long as everything is configured correctly, also

 

14 hours ago, JorgeB said:

create the share with use cache="yes" and run the mover once and it will also exist in the array.

this was incorrect, shared created with cache="yes" if first created in the array, so available space will be correct.