Log rotation for docker or limit log size?


Recommended Posts

I don't believe there is log rotation enabled in unraid's implementation of docker. There doesn't seem to be a limit on log size either.

 

I kept running out of space in my docker image and finally realized that a few of the containers were filling up the image with their logs. One container had a log that was 2.8GB (couchpotato)

 

The temporary solution was to reinstall the container and it reset the files in /var/lib/docker/containers and got rid of the large logs. It freed up a ton of space, too. (For any users interested, the easiest way to reinstall is to click on the container image in the unraid gui, select edit, don't change any settings and just hit save, it will reinstall the container with the same settings)

 

Is there anyway a limit on size can be implemented, or log rotation? Or perhaps an option to move the logs somewhere else where there is more storage available?

 

Thanks

 

PS. If anyone's interested in checking how big their logs are, type this in the unraid terminal and it will list the largest logs:

du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60

  • Like 1
  • Upvote 3
Link to comment

Docker 1.8 added log rotation  https://docs.docker.com/reference/logging/overview/

--log-opt max-size=50m

 

 

To clear log you can execute:

 

echo "" > $(docker inspect --format='{{.LogPath}}' <container_name_or_id>)

 

edit: the above does not show correctly on tapatalk - add the container name to the end before executing.  Use Aptalca's command to list the log sizes then turn advanced view on in Docker tab and match docker container id to the docker name and run the command above using the Docker name.

Link to comment
  • 2 weeks later...

PS. If anyone's interested in checking how big their logs are, type this in the unraid terminal and it will list the largest logs:

du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60

 

Thanks for the command line to locate the runaway log file. I'm new to docker, and incorrectly assumed the logs were within the docker container itself. I tried searching around inside my BitTorrent Sync container (by running the 'docker exec -it Sync bash' command), but couldn't find the problem. I'm syncing so many files with BTSync, that the log would grow to 50GB during the initial "indexing" after starting BTSync. Thanks to you, I finally figured out what was happening and modified my Sync template to include '--log-driver=none' on the startup command line. I've lost my BTSync logs for now, but I no longer have to keep growing my docker.img size to accommodate my increased use of BTSync. Thanks!

Link to comment

Telnet/SSH to unRAID.  Issue the command in Aptalca's post:

 

du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60

 

It lists the logs for the containers and their sizes.  Locate the Docker Container ID in the list of a desired (large) log to delete:

 

du_log_highlight.png

 

Go to the unRAID GUI Docker tab.  Turn on advanced view.  Find the container that matches the ID from the 'du' command.

 

container_ID.png

 

 

To delete the log, issue the command below with the name of the container (case sensitive)

 

echo "" > $(docker inspect --format='{{.LogPath}}' sabnzbd)

 

du_log_clean_highlight.png

  • Like 1
  • Upvote 2
Link to comment

OK and if I have containers in Aptalca's command that are not in the unraid GUI, I can just delete them with MC as they were from old containers?

 

Thanks for the walkthrough by the way :)

 

Not exactly sure how old container/images and their logs are handled.  Can try deleting them as you say or wait for someone more familiar with it to chime in.

Link to comment

I think the issue is generally containers that haven't mapped their noisy daemon logs out to the host and dont rotate or destroy. In this scenario the daemon logs accumulate within the container and consume loopback image space until it is either manually deleted (bad) or the container reset (better but not great).

 

Certainly we should at the very minimum provide a GUI way for user to know where this space is going because currently we just alert saying essentially "your almost out of space go work out the docker command line to know where its gone".

 

Also I am off the firm opinion docker container that have this issue should be fixed as it is a bug. Pythin apps that use the native HTTP server are particularly bad for this.

Link to comment

I think the issue is generally containers that haven't mapped their noisy daemon logs out to the host and dont rotate or destroy. In this scenario the daemon logs accumulate within the container and consume loopback image space until it is either manually deleted (bad) or the container reset (better but not great).

 

Certainly we should at the very minimum provide a GUI way for user to know where this space is going because currently we just alert saying essentially "your almost out of space go work out the docker command line to know where its gone".

 

Also I am off the firm opinion docker container that have this issue should be fixed as it is a bug. Pythin apps that use the native HTTP server are particularly bad for this.

These logs are not within the containers. The syslogs of the containers are mapped to the host where a user can access them without having to exec into the containers.

 

Under unraid's implementation, these logs are saved within the docker.img but separately from the containers

 

That is part of the reason why it causes confusion because when the logs get large, the containers don't, but the docker.img gets full. The users get confused about why the image is ballooning when the containers themselves aren't.

Link to comment

If I remember correctly the real culprit is the docker logs which is not syslog, but rather the stdout and stderr of process id 1 ie the app docker is running

so if wrappers are used and the wrappers allow the stdout and stderr to just bubble up that will cause the docker.img to balloon. A well written docker will either have provisions for syslog output, or a logfile/logdir which can be mapped to the host volume.

Link to comment

I think the issue is generally containers that haven't mapped their noisy daemon logs out to the host and dont rotate or destroy. In this scenario the daemon logs accumulate within the container and consume loopback image space until it is either manually deleted (bad) or the container reset (better but not great).

 

Certainly we should at the very minimum provide a GUI way for user to know where this space is going because currently we just alert saying essentially "your almost out of space go work out the docker command line to know where its gone".

 

Also I am off the firm opinion docker container that have this issue should be fixed as it is a bug. Pythin apps that use the native HTTP server are particularly bad for this.

These logs are not within the containers. The syslogs of the containers are mapped to the host where a user can access them without having to exec into the containers.

 

Under unraid's implementation, these logs are saved within the docker.img but separately from the containers

 

That is part of the reason why it causes confusion because when the logs get large, the containers don't, but the docker.img gets full. The users get confused about why the image is ballooning when the containers themselves aren't.

 

OK this is interesting and exactly not what I thought the (and my) issue is.

 

Can we show some proof of this so that others can confirm?

 

Edit: I just remoted in to have a quick look and it seems to confirm the above. I think we would still benefit from showing the output of a real world example other can check. If no one else gets to it I will do it when i return as I am sure I am being hit with this the same as everyone esle

Link to comment

Just found 30GB worth of logs in my couchpotato container!

 

Thanks for the fix, but we really should have a way to at least inspect the docker image within the webgui to see what is taking up the space, if not be able to clear old logs without having to SSH into the machine

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.