Since the 6.3 beta testing I've had this issues where the rootfs would fill up to 100% At this point that all dockers would crash out and the system is pretty much useless. At this point a diagnostics command fails due to lack of space. The only option I have at this time is to reboot the server.

 

This time I've managed to catch it at 86% please find diagnostics attached. Any help would be appreciated.

vhs-diagnostics-20170226-1126.zip

Edited February 26, 20179 yr by Azza666

Try searching for rootfs as this issue has come up before.  Hopefully, one of those solutions will be applicable for you. 

Thanks for the reply. I've had a good look around these topics. They are either unresolved or they are unrelated to the issue I am experiencing.

 

ive looked at my diagnostics but they don't see anything out of the ordinary. But some of the stuff in there is way over my head 

Upon further investigation within var there is a bunch of stuff which looks like btrfs volumes. If that expected. Looks like a lot of docker stuff from what I can see. Also looks like var is mounted as part of rootfs.

df -h

 

You will see there are other items mounted within rootfs that do not take up space on rootfs, namely all the /var/lib/docker and down, those are mounted loopback from your Docker Image file. You should also have /var/log mounted as it's own volume and using tmpfs .

 df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           128M  3.6M  125M   3% /var/log

 

so it doesn't appear to be out of control logging. I'm not sure what other useful information the diagnostics tell us.

 

at this point I'm still sitting at 86% but would expect to be at 100% before the day is out.

 

her is a full df -h

df.txt

Edited February 26, 20179 yr by Azza666

Got to 100% so had to reboot 

The checksum plugin has been known to cause this problem in the past when it has been installed but not actually configured to do anything.   If this is the problem the solution is to remove the plugin.

 

I had been using the File Integrity plugin. I removed this earlier following advice from another topic even if is not the plugin mentioned. Just to see if there is impact.

 

thanks for the suggestion anyway.

Edited February 26, 20179 yr by Azza666

If you have removed the Dynamix File Integrity plugin then it is not the one which caused problems, so you can probably reinstall it.

 

The problem was with the Checksum plugin which pre-dated the Dynamix File Integrity plugin.    If you don't have ithe Checksum plugin installed then your problem must be some other issue.

24 minutes ago, remotevisitor said:

Checksum plugin

I still use this without problem.

Just now, trurl said:

I still use this without problem.

As do I, but if I see it installed and there's memory issues then I also recommend to uninstall it.

But, back on topic, best wild guess cause is this line in the log

Quote

Feb 18 01:51:07 VHS root: container "libresonic" hostpath "/mint/cache/appdata/libresonic/other-media" does not exist

 

I'm going to take a flyer and say that because of that, libresonic is storing stuff within RAM which is causing the issue.  It should be /mnt/cache/...

Edited February 26, 20179 yr by Squid

15 minutes ago, trurl said:

I still use this without problem.

 

The problem with the Checksum plugin appears to manifest when it is installed but not configured.    Therefore if you have actually configured it then I'm not surprised you have no problem.    Anyway, as this appears to not be the problem with the OP, I suggest we don't side-track this thread any further and get back on track with help on the OP..

 

Squid, well spotted got the /mint typo..... Looks like that may be a lick lt candidate.

I've modified the docker setting and will monitor my server closely over the next few days.

Well I fixed the path error pointed out in a previous post for libresonic.

 

no luck. Now after a day after a reboot I'm at 86% of 32G used.

 

latest diagnostics attached. Once again any help would be very well received 

vhs-diagnostics-20170227-1832.zip

Thanks to everyone who offered suggestions. Someone mention bad paths being stored in memory. I checked all my other dockers and they were fine.

 

so I checked plugins and the appdata backup was referencing a path to a drive which used to be attached (no longer there) via unasssigned devices! 

Thanks to everyone who offered suggestions. Someone mention bad paths being stored in memory. I checked all my other dockers and they were fine.
 
so I checked plugins and the appdata backup was referencing a path to a drive which used to be attached (no longer there) via unasssigned devices! 

That'll do it. Have to admit that I didn't consider that possibility. Will update

Sent from my LG-D852 using Tapatalk