Hate to request, but please open a new defect report on this.  In all my testing the per-share cache floor and global cache floor settings seem to work correctly.

Why? It is the same bug.

 

Description: Even with a floor set the cache disk can run out of space (which should not be possible)

 

The consequences of running out of space are a symptom and not the bug and to be expected really.

Why? It is the same bug.

Which bug exactly?  I cannot reproduce the original issue in -beta10a.

 

Description: Even with a floor set the cache disk can run out of space (which should not be possible)

Not strictly true.  If an existing file on the cache device is extended it will be possible to run out of space.  If the 'cache' share is written directly, obviously it can run out of space.

 

The consequences of running out of space are a symptom and not the bug and to be expected really.

I don't understand what you mean by this statement.

 

--

 

In testing I did run across an interesting behavior.  First, I set up an array with a single disk1 and a single cache disk, each 120GB.  Next, created a share called 'test' with 'Use cache disk' set to Yes.  I then set the "Min free space" under Cache settings to 110GB and starting copying files to the 'test' share via windows explorer.  Once free space on cache got below 110GB, the next files created caused the "cache disk full" messages to get logged and files started getting created on disk1.  This is how it's supposed to work and also shows how I cannot reproduce your original issue.

 

Next I deleted all the files, and then set Cache 'Min free space' to blank (or zero), and now set 'Min free space' for the 'test' share to 110GB and started copying files.  This time once 10GB was written to the share, windows popped up a window saying "no more space on device" - but, it also got into a "loop" repeating trying to create the file over and over again.

 

What's happening here is this: The 'Min free space' for a share says there must be that amount of free space on any device that it wants to create a file on.  If that device does not have enough free space, it checks the next device.  In my case I only have one array disk, so it reported "no more space".  However with debugging turned on I noticed that windows, in response to this, executes a 'statfs' on the share - meaning, "tell me how much free space is left".  The size of the file it was trying to create was around 3GB and statfs reported there was 110GB free (because that's how much free space there really was on disk1).  Windows therefore was continually retrying file create, saying, "What is wrong with this bloody server - he's saying there's 110GB free but giving me 'out of space' when I try to create a 3GB file!"

 

Ok, I don't think this is the issue you are reporting but it raises an interesting observation about how "free space" should be computed.

 

At present, without going into corner-case details, free space for a share is computed by simply adding up the free space of all the devices that share exists on, or could exist on.  But this presents a problem when the share nears filling up.  It could be there is "lots" of free space, but the biggest file that can be created is far smaller than this number and will lead to problem described above.

 

I think what the s/w needs to do is this: when client asks "how much free space on this share?" it should return the size of the largest possible file that can be created on the share, taking into account current actual free space on all the devices, as well as the various floor values, and whether cache is involved or not.  Doing this will avoid the problem described above but will produce some "strange" free space numbers.

 

For example, say you have a share than can span 3 disks: disk1 with 10GB free, disk2 with 20GB free, and disk3 with 30GB free.  Ignoring floors etc., current code will report 60GB free, but it will be impossible to create any file larger then 30GB.  Hence I think it should report there is 30GB free.  Now suppose you copy a 30GB file.  Afterwards it will now report 20GB free.  Not sure if this will cause it's own problems but I think that is how it should work.

 

Make sense?

The consequences of running out of space are a symptom and not the bug and to be expected really.

I don't understand what you mean by this statement.

 

This is just me admitting I have a few times rambled about consequences that happen after space runs out which is irrelevant to the actual bug of space running out.

 

For example, say you have a share than can span 3 disks: disk1 with 10GB free, disk2 with 20GB free, and disk3 with 30GB free.  Ignoring floors etc., current code will report 60GB free, but it will be impossible to create any file larger then 30GB.  Hence I think it should report there is 30GB free.  Now suppose you copy a 30GB file.  Afterwards it will now report 20GB free.  Not sure if this will cause it's own problems but I think that is how it should work.

 

Make sense?

 

This is actually a very clear description of an angle I haven’t considered before and could explain at least one variant of what is happening to me. I would open this whole idea up for it up to debate as it seems safer to use the smaller number but the macro effect could be serious change for people.

 

For sure when cache and array space get low weird things seem to happen and I suspect that quite a bit of my problem may come from this. I have an extra 4TB on order and will follow this with doubling my cache drive later.

 

My concern is that this is a very tricky bug to nail down and I don’t want to hog time (although to be fair we seem to be catching some interesting things in the process).

 

Is there any chance adding some extra logging because it occurred to me is that the symptoms of "out of disk space" and a "generic (28) error" essentially match every space related bug we could ever think of and gets us no closer to nailing it down. Perhaps even listing the free space of each component with the (28) could help out long term regardless.

 

You may want to report free space as 2 numbers, something like "Total free space: 100.4GB (Largest free space chunk: 30.2GB)".

any news on a fix. unless i dont use my cache drive unRAID doesn't honor the floor which in turns kills docker and sometime corrupts appdata as well.