Log rotation for docker or limit log size?

[aptalca]

I don't believe it would. What does the log for that container say?

[bonienl]

53 minutes ago, jang430 said:

Does the log being full prevent my Docker apps from starting?  It's happening, Docker apps aren't opening, at least 1 closed.

 

Yes, this is possible. Docker log files are stored in the same file system as system logs. When full it will cause an issue for Docker.

 

[aptalca]

 
Yes, this is possible. Docker log files are stored in the same file system as system logs. When full it will cause an issue for Docker.
 

I thought docker log files were stored within the docker image

[bonienl]

9 minutes ago, aptalca said:

I thought docker log files were stored within the docker image

 

 

Both Docker and VM log files are in /var/log. These are the services.

 

Log files of the containers themselves are in the docker image.

[jang430]

I clicked on the log on the upper right, a new window opens.  Even after a long time, nothing comes out anymore.  I decided to restart the whole Unraid machine.  After restart, log went down to 2%.  Docker apps are up and running once again.  

 

I hope the log doesn't go full once again.

[aptalca]

 
Both Docker and VM log files are in /var/log. These are the services.
 
Log files of the containers themselves are in the docker image.

Right.

To clarify, the docker logs described in the original post of this thread refer to the container logs and not the docker service log.

The container logs reside in the docker image and are essentially the system logs of each container. They can and often do get out of hand when the container encounters an issue and the log is spammed repeatedly.

Afaik, on unraid, these logs never get rotated or cleaned, and can fill up the docker image without the user realizing it. I would like to reiterate the feature request of adding log size limit functionality to the docker service for the container logs.

The current version of docker supports the log option "max-size" for the json-file log driver.

https://docs.docker.com/engine/admin/logging/json-file/

Thanks

[dmacias]

Right.

To clarify, the docker logs described in the original post of this thread refer to the container logs and not the docker service log.

The container logs reside in the docker image and are essentially the system logs of each container. They can and often do get out of hand when the container encounters an issue and the log is spammed repeatedly.

Afaik, on unraid, these logs never get rotated or cleaned, and can fill up the docker image without the user realizing it. I would like to reiterate the feature request of adding log size limit functionality to the docker service for the container logs.

The current version of docker supports the log option "max-size" for the json-file log driver.

https://docs.docker.com/engine/admin/logging/json-file/

Thanks

@Squid also pointed out that you need max-file=1, otherwise the log gets rotated and is still in the image.

[aptalca]

That's a good point. One thing I'm not sure about is whether the rotated logs are compressed. If they are, it may be good to set it to 3 or 5 for archiving purposes since they would take much less space when compressed.

[aptalca]

@limetech @eric @jonp

 

I just wanted to make sure this didn't get lost in the pile.

 

To summarize, unraid currently does not have log rotation features of docker enabled for the json logging driver, which can cause issues with log files getting too large and filling up the docker image.

 

Docker has built-in log rotation / limitation feature, which can be enabled through configuring the service with the following options

--log-opt max-size=10m    #with k, m, or g
--log-opt max-file=3

source: https://docs.docker.com/engine/admin/logging/json-file/#options

 

I would appreciate it if this would be incorporated in the next stable

 

Thanks

[jonp]

7 hours ago, aptalca said:

@limetech @eric @jonp

 

I just wanted to make sure this didn't get lost in the pile.

 

To summarize, unraid currently does not have log rotation features of docker enabled for the json logging driver, which can cause issues with log files getting too large and filling up the docker image.

 

Docker has built-in log rotation / limitation feature, which can be enabled through configuring the service with the following options

--log-opt max-size=10m    #with k, m, or g
--log-opt max-file=3

source: https://docs.docker.com/engine/admin/logging/json-file/#options

 

I would appreciate it if this would be incorporated in the next stable

 

Thanks

 

I will forward this to the rest of the team for feedback.  Thanks for posting and including all the details!

[ars92]

Thanks for this. 

 

Turns out my sonarr linuxserver docker was occupying 6.5g of log space due to it being set as trace level for whatever reason

 

Changed it to info, edited the container and now the docker.img is only 32% filled. 

 

Guess the log rotation is still not implemented in 6.3.5? Or maybe its the docker maintainer who hasnt implemented the support?

[bonienl]

Log rotation is available on Docker level only, meaning when you start the Docker service it applies the Log setting to all containers. Currently this feature is available in unRAID version 6.4-rc7 and later.

 

[aptalca]

Thanks bonienl, that's great

[ars92]

Log rotation is available on Docker level only, meaning when you start the Docker service it applies the Log setting to all containers. Currently this feature is available in unRAID version 6.4-rc7 and later.
 

Great thanks!

In less than 12 hours i can see it has grown to 400mb now, looking forward to 6.4!!

Sent from my LG-H990 using Tapatalk

[Nem]

whats the ETA on 6.4?

 

If its still far from release, is the recommended way of dealing with this still to reinstall the container?

[CyberMew]

On 11/01/2018 at 11:54 AM, Nem said:

whats the ETA on 6.4?

 

If its still far from release, is the recommended way of dealing with this still to reinstall the container?

It's released!

[Rookie]

On 6/27/2017 at 6:51 PM, jonp said:

--log-opt max-size=10m #with k, m, or g --log-opt max-file

Where exactly do I put this in order to fix my docker.img file problem? And please confirm that once I do this I need to restart the docker img for it to take effect

[Squid]

Where exactly do I put this in order to fix my docker.img file problem? And please confirm that once I do this I need to restart the docker img for it to take effect

You can set the options via the GUI (settings, docker, advanced view)

Sent from my LG-D852 using Tapatalk

[Osiris]

co=$(docker inspect --format='{{.Name}}' $(docker ps -aq --no-trunc) | sed 's/^.\(.*\)/\1/' | sort); for c_name in $co; do c_size=$(docker inspect --format={{.ID}} $c_name | xargs -I @ sh -c 'ls -hl /var/lib/docker/containers/@/@-json.log' | awk '{print $5 }'); YE='\033[1;33m'; NC='\033[0m'; PI='\033[1;35m'; RE='\033[1;31m'; case "$c_size" in *"K"*) c_size=${YE}$c_size${NC};; *"M"*) p=${c_size%.*}; q=${p%M*}; r=${#q}; if [[ $r -lt 3 ]]; then c_size=${PI}$c_size${NC}; else c_size=${RE}$c_size${NC}; fi ;;  esac;  echo -e "$c_name $c_size"; done

Single command to produce a list of log-sizes per container.

 

 

[AetherSmith]

Hey there, I believe I've been seeing this issue with the Resilio Sync docker, but even after setting the log rotation options in the GUI it doesn't seem to have helped. Switched the container back on yesterday, and by this evening the JSON log had reached 5.2G. Specifically, the offending file is the one with a filename like /var/lib/docker/containers/<docker ID>/<docker ID>-json.log; are the GUI options for something else?

[rmilyard]

So I just started running into this issue with 6.6.5. My docker image hit 85%. I deleted them with commands from first post. 

 

Just wondering if there is a fix in current version on UnRAID. And if so how do I implement it?

[Squid]

1 hour ago, rmilyard said:

Just wondering if there is a fix in current version on UnRAID. And if so how do I implement it?

For a number of releases, if the logging is the problem

https://forums.unraid.net/topic/57181-real-docker-faq/?tab=comments#comment-564326