unRAID OS Version: v6b6

 

Description: Even with a floor set the cache disk can run out of space (which should not be possible)

 

How to reproduce: Set the  cache disk Min free space to 2000000 and copy data to the array.

 

Other information: I note that the help section of the GUI now says "If no suffix, GB is assumed." which makes the above figure a nonsense. The forum and the wiki? states this defaults to kB:

 

Units are 1024-byte blocks.  So if you set Min free space to 1000 that would be 1000 x 1024 = 1,024,000 bytes.

 

Once this error is seen /mnt/user0 becomes empty breaking the mover script.

ls -al /mnt/user0
total 0
drwxrwxrwx  2 root root  40 Jul 21 09:22 ./
drwxr-xr-x 19 root root 380 Jul 21 09:22 ../

 

As you would expect the syslog now has a recurring message of "cache disk full" however if I set min free space to "1KB" and free up a few GB of space the "cache disk full" continues.

 

If I remove the cache dir and add it again the mover still skips everything, ls /mnt/user0 is still empty but the cache full message is gone. I can also now copy files to user shares and see the data on the cache drive directly

 

If I stop the array and start it again everything works as normal.

The suffix in the cache floor field is indeed broken (it gets completely ignored).

 

Also the help text is also wrong: absence of a suffix indeed means a count of 1024-byte units. So 2000000 in there means:

 

2000000 x 1024 = 2,048,000,000 bytes

 

The KB, MB, GB, TB are in base 10 units, eg, KB = 1000.

 

This has all been corrected - musta been drunk when I coded that and hung over when tested 

 

But I cannot reproduce the problem you see where cache runs out of space causing /mnt/user0 to go away...

 

Any more details, e.g., what protocol (NFS, SMB, ?).

I am having problems myself making user0 go away again. It seems like it was perhaps an edge case caused by the mover + mc + docker running at the same time as apart from those and SMB nothing else is used.

 

However I can cause it to fill up but that would be expected if the suffix get completely ignored.

 

Let me change it from 20GB to 10000000 and see if that solves the issue.

So randomly user0 became empty again. I can only suppose it is still related to docker + mover but there is nothing in the logs at all to help.

 

Stopping and starting the array brought it back as before.

 

However this cannot be a coincidence:

 

/dev/sdc1      120G  119G  4.0K 100% /mnt/cache

 

so a floor setting of 10000000 was somehow ignored.

 

Update 2: All docker containers are lost in the GUI as well although i can still see them in the command line.

 

 

'docker' is set as a 'use cache' Only share?  And 'docker' dir does not exist on any array devices, correct?

 

Also, you must reference the docker folder as it exists directly on the disk, eg,

/mnt/cache/docker

not

/mnt/user/docker

 

I'm assuming this is what you do because docker won't work otherwise.  Also assuming /dev/sdc is your cache disk?  If so, what else is writing to it?  Something else: split-level is still in effect with the cache disk, one could argue that's a bug.

docker is set to /mnt/cache/apps/docker.

 

The "apps" parent share is set to cache only

Data in the the "apps" share exists only on the cache drive and not on any other drive.

 

/dev/sdc is my cache disk.

 

The process that actually fills the cache drive up is a log down loader which admittedly  is a bit dumb and will just fill every byte if given the chance.

The process that actually fills the cache drive up is a log down loader which admittedly  is a bit dumb and will just fill every byte if given the chance.

 

Is it just appending to the same log file?  Once a file is on a device, the file itself cannot be split between different devices.  If an exiting log file grows it will eventually hit "Out of Space" condition.

No sorry the log files are just lots of tgz (few tens of MB each) that accumulate until the daily mover kicks in. I think the thing that broke the camels back this time was that i copied a load of media up that reduced the available cache space this specific day and then the log process promptly filled the drive up as nothing stopped it doing so.

 

For info all my docker containers are completely b0rked. The images are still there but unRAID cant see the containers and i get variations of a theme on:

 

c18fba1d953af40e6fe1f74731208b76c1135876bd26cf9456ffa673dea7e9bf
2014/08/24 07:33:24 Error: Cannot start container c18fba1d953af40e6fe1f74731208b76c1135876bd26cf9456ffa673dea7e9bf: Cannot find child for /Deluge
The command failed.

 

Docker really really doesnt not play well if it runs out of space this is something we need to look at seriously as an out of space SAMBA message is one thing but a completely broken docker with no easy recovery is quite another.

After some effort the only fix I could come up with for docker was to remove every container and images (removing containers alone was not enough) and then add them all back in again.

 

The docker equivalent of ctrl-alt-delete

 

Note: since I am now one version out of date I will not report against this until I see it again with b7. Can I just confirm that we expect the cache floor settings to be fixed with b7?

 

 

After some effort the only fix I could come up with for docker was to remove every container and images (removing containers alone was not enough) and then add them all back in again.

 

The docker equivalent of ctrl-alt-delete

Do we actually have two different bugs here?  One having to do with 'floor' settings, the other having to do with behavior of docker if it's file system becomes full?  I'm guessing the first is causing the second in your case, but if the first is fixed, the second could still happen due to something else.  If this seems right to you, please break out the docker bug into a separate report.

 

Note: since I am now one version out of date I will not report against this until I see it again with b7. Can I just confirm that we expect the cache floor settings to be fixed with b7?

Yes.

I think you are right we have in fact four bugs here:

 

1. Cache drive floor free space isn't working (even if i dont use any units)

2. Units floor free space aren't interpreted correctly

3. Documentation, forum and webgui documentation do not agree on what value is assumed if a unit isnt specifiied

4. Docker simply cannot handle if its install location runs out of space. Recovery of docker from this failed state seems to require a complete reinstall of every container and image dependency.

 

I would suggest 1-3 are all part of one larger interelated bug fix and 4 is a new ticket. I will go on this assumption and start a new ticker for number 4.

I had this error today on 6b7. If I store a location for appdata on a btrfs drive in the array:

"/mnt/disk9/appdata" works but "/mnt/user/appdata" gives an out of space in a docker container log

 

Could be related?

 

Docker is not supported to operate when it's install path is specified to a user share.  Please only install docker to a disk directly when using an array device.

 

With the cache pool feature, you can have two or more disks in a BTRFS RAID1 cache pool with Docker installed to /mnt/cache/docker and that works, but not /mnt/user/cache/docker or /mnt/user/docker.

I had this error today on 6b7. If I store a location for appdata on a btrfs drive in the array:

"/mnt/disk9/appdata" works but "/mnt/user/appdata" gives an out of space in a docker container log

 

Could be related?

 

Docker is not supported to operate when it's install path is specified to a user share.  Please only install docker to a disk directly when using an array device.

 

With the cache pool feature, you can have two or more disks in a BTRFS RAID1 cache pool with Docker installed to /mnt/cache/docker and that works, but not /mnt/user/cache/docker or /mnt/user/docker.

I think he is talking about data as accessed by a docker container, rather than where docker itself is installed. Of course our docker containers must be able to access the array or they wouldn't be very useful.

Yeah he almost certinaly is sicne "/mnt/user/appdata" is the default config data directory of the enhanced docker plugin i.e. almost everyone uses it so if its not supported almost no one is

Yeah he almost certinaly is sicne "/mnt/user/appdata" is the default config data directory of the enhanced docker plugin i.e. almost everyone uses it so if its not supported almost no one is

Another thread about this.

Confirming this issue still exists with b7. I left some files copying up to the array using mc from a USB attached disk this time and even though cache floor is set to "10G" I end up with a No space left on device (28) error

 

df -h | grep cache
/dev/sdc1       120G  118G  587M 100% /mnt/cache

 

 

Unlike last time docker still works and user0 still exists although we haven't quite completely filled the drive this time which probably explains that.

 

Edit: removed the obvious brain fart

 

This should be fixed in -beta8, changing to 'solved'.  Could be re-opened if that's not the case.

That was a FAST fix 

... wish all bugs were that simple !!

 

This should be fixed in -beta8, changing to 'solved'.  Could be re-opened if that's not the case.

I think there is still a problem in beta 8!

 

I  have min free space set to 50GB

.

I started copy some largish (~4GB) files to the array, and monitored the free space.  When the free space dropped below 50GB I started getting messages of the form

Sep  1 19:01:47 DJW-UNRAID shfs/user: cache disk full

in the syslog.  However on inspection the files were still being written to the cache disk, whereas my understanding is that they should now be written directly to the the array data drives bypassing the cache disk?  I got one message of the form shown above each time a new file was copied - and each time it ended up on the cache disk.

Yeah that's not good, I'll reopen.

Marking this fixed in -beta9.  If you still find a problem, please post here and I'll re-open.

Still broken in beta9

Still broken in beta9

 

What part?  There were a few issues reported in this thread.

This bit:

 

Description: Even with a floor set the cache disk can run out of space (which should not be possible)

 

In cache settings i have Min. free space: 10G

 

It is very obvious for me when it doesnt work because SQL fails as there isnt even enough room on the cache drive for mariadb to make the tmp file to return an answer to any query.

 

 

Woke up this morning to find cache drive full and array full. What a mess that caused, docker crashed and i couldn't even mount a disk via usb and udev. I had to manually move some files from the array which cleared up 1GB which in turn let me mount a disk locally to clear more space.

 

For me at least the cache floor is still 100% broken.

 

Edit: Acytually when I thjink about it I am not sure unRAID could have handled this one any different. Just to be clear though even with lots of array space free cache drive floor is not honored.