Weird error messeges every 2 days like clockwork

[ryguy7]

Every two days my unriad setup starts acting weird.  I first noticed the problem with the unassigned devices plugin on my main screen.  It would show error messages like the ones below.  I assumed it was related to the plugin so I deleted it.  Then I went to update docker list and noticed this message popping up when I try to update applications.  It seems like I am running out of space in a system partition or something.

 

"

Warning: file_put_contents(): Only 0 of 60 bytes written, possibly out of free disk space in /usr/local/emhttp/plugins/community.applications/include/exec.php on line 130 Fatal error: Uncaught exception 'UnexpectedValueException' with message 'RecursiveDirectoryIterator::__construct(/var/lib/docker/unraid/templates-community-apps/templates/gfjardimsRepository): failed to open dir: No such file or directory' in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php:78 Stack trace: #0 /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php(78): RecursiveDirectoryIterator->__construct('/var/lib/docker...', 4096) #1 /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php(193): DockerTemplates->listDir('/var/lib/docker...', 'xml') #2 /usr/local/emhttp/plugins/community.applications/include/exec.php(137): DockerTemplates->downloadTemplates('/var/lib/docker...', '/tmp/community....') #3 /usr/local/emhttp/plugins/community.applications/include/exec.php(1061): DownloadCommunityTemplates() #4 /usr/local/src/wrap_post.php(17): include('/usr/local/emht...') #5 {main} thrown in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 78

"

 

Its really annoying because once these error messages start popping up I am unable stop the array or reboot, unless I force it to from commandline.  Does anyone know what these messages mean?

 

Thanks

[Squid]

First off, if You're quoting error messages from CA it's best to do it in the CA thread so that it's not hit and miss on me seeing it.

 

Until I get home and see the actual lines in code that are referenced I can tell you that it's either the tmp fs filled up or the docker.omg is full or corrupted.

 

Since its every two days odds on it's your tmp fs and you've got a container of plugin or something saving files to ram

 

Sent from my LG-D852 using Tapatalk

 

 

[Squid]

Strike that.  Had a better look at the messages on my laptop, and its your docker.img file full / corrupt.

 

And, there is generally zero reason to hit the Update Applications button within CA (as the pop up states)

 

 

[ryguy7]

Its not just the Community Applications,  thats just the easiest problem to reproduce.  Once these error messages start popping up, that's when I know I need to reboot the system.  Once they appear, hard drives start disappearing, all the buttons from the main page of the webgui disappear etc.    Here's an image of what my main page looks like http://i.imgur.com/020itWx.png

[Squid]

Ah ok.  In that case I would try and grab the diagnostics from the UI.

 

If that fails and you've got the powerdown plugin installed and updated (and you should),

powerdown -r

and the diagnostics will save to the flash drive

 

TBH, this has happened to me something like twice during the whole 6.x series.  But, since it was never a reoccuring issue for me I've never attempted to diagnose what's going on.  Best guess without diagnostics however is that your cache drive dropped offline which can have some strange effects.  But that's only a wild guess

 

[Squid]

And if it is repeatable every two days (and diagnostics don't work), what I would do is hook a monitor and keyboard up to the server, log in locally and enter in:

tail -f /var/log/syslog > /boot/syslog.txt

 

That'll continually copy the syslog to your flash drive which will give us something to look at next time it happens

 

 

[ryguy7]

I guess I will post a log next time it happens, but I have been having this problem for a few weeks now.  I did capture a log last week.  Since this log was captured, I did replace the cache drive (sdd) and make a new docker image/apps.  The problem persists.

 

Anyways here it is.

 

Thanks

 

Its filled with "No such file or directory"  Does that mean a docker is trying to access a folder that doesn't exist and its filling up all my space?

syslog.zip

[Squid]

Best here is that sdd dropped offline, but the log doesn't go back far enough to see what happened.  It keeps talking about corruption on sdd1, and since the cache drive is the only drive that is referred to like that (the others get referred to a mdx - and it also complains about corruption on md7), it seems to follow through.

 

Start that tail command so that we can then see what leads up to everything.

 

[ryguy7]

It happened again.  I ran the command you specified.  This was all that was in the logfile:

 

Apr 16 19:41:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:42:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:43:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:44:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:45:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:46:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:47:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:48:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:49:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:49:12 battleserver sshd[7804]: Accepted password for root from 192.168.1.8 port 59342 ssh2
Apr 16 19:50:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:51:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null
Apr 16 19:52:01 battleserver crond[1660]: exit status 2 from user root /usr/local/emhttp/plugins/dynamix.system.stats/scripts/sa1 1 1 &> /dev/null

[ryguy7]

The error messages that keep filling the webgui screen say out of diskspace.  I ran a command to show disk space

 

root@battleserver:~# df -h /usr/local/emhttp/

Filesystem      Size  Used Avail Use% Mounted on

-              7.8G  7.8G    0 100% /

 

As somone that is not familiar with linux can someone explain what this means? 

Thanks

 

 

[remotevisitor]

It means that the root file system "/" in which the directory "/usr/local/emhttp" resides has completely filled up (100% in use).

 

As had been suggested earlier, this means the you have some program which is incorrectly writing files to the / file system.

 

I would suggest after a reboot you do the command

  du -sx /*

And repeat it every few hours and check to see if a number reported in the 1st column for a directory keeps increasing.

 

If a directory does show it increasing repeat the command on that directory. Eg if /home is continually increasing then run the command "du -sx /home/*".    Repeat this until you have identified the directory which holds the files causing the problem.

 

Then you might have some idea of what program is writing the files.

[ryguy7]

Well it's been a day and a half and the issue seems to be resolved.  The directory is not filling any more.  For future reference in case anyone else has a similar issue, the offending plugin was checksum tools.  I had it misconfigured or something, either way I just removed it and the issue has resolved.

[Squid]

Well it's been a day and a half and the issue seems to be resolved.  The directory is not filling any more.  For future reference in case anyone else has a similar issue, the offending plugin was checksum tools.  I had it misconfigured or something, either way I just removed it and the issue has resolved.

One of the reasons why I don't suggest installing it anymore

 

Sent from my LG-D852 using Tapatalk