Different file write policies (share floors) for cache and array - per share

[TheLinuxGuy]

While copying data to my new unraid server to a share which has cache+array I keep my writes denied once the cache fills up. This is an inconvenience for me - if there's space in the secondary storage the writes should instead go to the array until mover frees up the cache.

 

The server I am moving from is my mergerfs + snapraid setup (https://github.com/TheLinuxGuy/free-unraid) and this is absolutely one of the best features of mergerfs (https://github.com/trapexit/mergerfs).

 

I am not sure why unraid doesn't have it on its FUSE implementation so here's a feature request for it.

Edited August 7, 2023 by TheLinuxGuy
clarity in title, as requested.

[JorgeB]

11 minutes ago, TheLinuxGuy said:

if there's space in the secondary storage the writes should instead go to the array until mover frees up the cache.

It will go to the array if you configure the minimum free space for the share, we usually suggest configuring it to twice the file size you expect to copy to that share.

[TheLinuxGuy]

5 minutes ago, JorgeB said:

It will go to the array if you configure the minimum free space for the share, we usually suggest configuring it to twice the file size you expect to copy to that share.

 

I think the way this is setup is a bit confusing; the help text for that setting is not cache specific (so it feels ambiguous)
 

Quote

The minimum free space available to allow writing to any disk belonging to the share.

Choose a value which is equal or greater than the biggest single file size you intend to copy to the share. Include units KB, MB, GB and TB as appropriate, e.g. 10MB.

Why not have a separate file setting for this "Primary" and "Secondary" storages? This is how mergerfs is flexible in.

In mergerfs - I configure my slow disks ("array" in unraid parlace) with that setting; while my "cache" has a different configuration. Note how I ensure each individual array disk must always keep 300G free before the next candidate is chosen for writes. NVME cache has 4G reservation. 

 

Could we get this flexibility in unraid? Cache and Secondary storage settings per disk member of each.

# add fstab slow-storage
/mnt/disk* /mnt/slow-storage fuse.mergerfs defaults,nonempty,allow_other,use_ino,category.create=msplfs,cache.files=off,moveonenospc=true,dropcacheonclose=true,minfreespace=300G,fsname=mergerfs 0 0

# add fstab /cache nvme
/cache:/mnt/slow-storage /mnt/cached fuse.mergerfs defaults,nonempty,allow_other,use_ino,noforget,inodecalc=path-hash,security_capability=false,cache.files=partial,category.create=lfs,moveonenospc=true,dropcacheonclose=true,minfreespace=4G,fsname=mergerfs 0 0

 

[JorgeB]

5KB won't work unless the files you are copying are smaller than that, which I assume they are not.

[TheLinuxGuy]

1 minute ago, JorgeB said:

5KB won't work unless the files you are copying are smaller than that, which I assume they are not.

If I set this to 4Gb (matching my nvme cache mergerfs setting), is it possible to ensure each array disk (non-cache) to have a different min space reservation (300GB)?

[JorgeB]

33 minutes ago, TheLinuxGuy said:

s it possible to ensure each array disk (non-cache) to have a different min space reservation (300GB)?

Not at this moment, share floor applies to all devices that make part of that share.

[TheLinuxGuy]

11 minutes ago, JorgeB said:

Not at this moment, share floor applies to all devices that make part of that share.

 

Would you then consider then this feature request to be valid?

 

I appreciate that you helped clarify one of the existing unraid features that I wasn't aware of; I don't think the previous posts troubleshooting addressed my Feature Request for more flexibility to control how the cache actually works in each share.

My other feature request is also related to the cache and using quotas rather than waiting for the entire cache disk drive to fill up.

[JorgeB]

6 minutes ago, TheLinuxGuy said:

Would you then consider then this feature request to be valid?

I believe there is already a feature request to have different share floors for array and pools, but if you don't find it suggest changing the request title to that, to be more clear.

[itimpi]

1 hour ago, TheLinuxGuy said:

if there's space in the secondary storage the writes should instead go to the array until mover frees up the cache.

This will happen if you set the Minimum Free Space for the pool (cache) irrespective of whether you have set a Min9mum Free Space for a share.

[JorgeB]

9 minutes ago, itimpi said:

This will happen if you set the Minimum Free Space for the pool (cache) irrespective of whether you have set a Min9mum Free Space for a share.

It will happen if pool floor is set higher than the share floor, not like the OP wanted, he wanted a lower setting for the pool, and that's not currently possible, since the higher of the two settings will take effect.

[itimpi]

5 minutes ago, JorgeB said:

It will happen if pool floor is set higher than the share floor, not like the OP wanted, he wanted a lower setting for the pool, and that's not currently possible, since the higher of the two settings will take effect.

I have been saying for a long time that the pool setting should be used when for a pool for caching purposes and the share setting used when writing to the main array.   
 

The only time the share setting should apply to a pool is if the share is set to only be on the pool.

[JorgeB]

11 minutes ago, itimpi said:

I have been saying for a long time that the pool setting should be used when for a pool for caching purposes and the share setting used when writing to the main array.   

That would probably be an easy way to fix this issue, unless I'm missing other use cases.

[TheLinuxGuy]

3 hours ago, itimpi said:

I have been saying for a long time that the pool setting should be used when for a pool for caching purposes and the share setting used when writing to the main array.   
 

The only time the share setting should apply to a pool is if the share is set to only be on the pool.

 

I actually agree with this...

 

6 hours ago, JorgeB said:

I believe there is already a feature request to have different share floors for array and pools, but if you don't find it suggest changing the request title to that, to be more clear.

 

I did not find it. I did come across another FR about quotas from 2021: 

and this from 2014

 

I'm not sure how to rename the title of this thread. Maybe option will show up after I post this