dnLL Posted January 4, 2020 Share Posted January 4, 2020 Here: root@server:~# df -h /var/log Filesystem Size Used Avail Use% Mounted on tmpfs 256M 167M 90M 65% /var/log root@server:~# du -sm /var/log 3 /var/log du doesn't find much and there is no mountmoint under /var/log/* either. I looked it up on the web and people mentionned it usually happens if there are deleted files/handles locked by a process such as: root@server:~# lsof | grep "/var/log/" | grep deleted dockerd 6282 root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 2047 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 2047 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6286 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6286 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6287 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6287 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6288 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6288 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6290 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6290 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6291 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6291 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6295 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6295 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6296 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6296 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6297 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6297 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6298 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6298 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6299 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6299 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6317 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6317 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6318 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6318 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6319 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6319 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6320 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6320 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6321 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6321 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6322 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6322 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6323 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6323 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6324 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6324 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6325 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6325 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6578 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6578 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6579 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 6579 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 8090 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 8090 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 10337 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 10337 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 12624 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 12624 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 17100 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 17100 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 17643 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 17643 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 19395 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 19395 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 20294 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 20294 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 25049 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 25049 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 28520 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 28520 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 31502 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 31502 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 32693 dockerd root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) dockerd 6282 32693 dockerd root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 1082 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 1082 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 3031 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 3031 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6301 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6301 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6302 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6302 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6303 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6303 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6304 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6304 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6305 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6305 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6306 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6306 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6307 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6307 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6308 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6308 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6309 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6309 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6310 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6310 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6311 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6311 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6312 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6312 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6314 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6314 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6315 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6315 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6316 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6316 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6976 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6976 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6977 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6977 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6978 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 6978 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 7304 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 7304 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 8412 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 8412 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 9110 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 9110 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 9111 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 9111 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 10407 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 10407 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 11343 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 11343 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 13038 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 13038 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 13371 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 13371 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 15705 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 15705 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 16646 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 16646 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 20586 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 20586 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 32149 container root 1w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) container 6300 32149 container root 2w REG 0,28 171606437 25553 /var/log/docker.log.1 (deleted) Right... so the log rotation works I guess... but the deleted files apparently still take space on /var/log (at least that's what I'm thinking)? I didn't try (yet) to kill/restart the dockerd engine (which would remove the locked handles shown above) since I'd like to find a better way. My server has 200+ days of uptime so yes, rebooting once in a while would obviously prevent the issue but that's not an actual fix, so I'm looking for a better solution. Quote Link to comment
BRiT Posted January 4, 2020 Share Posted January 4, 2020 Simply stop Docker using the Web UI then start it. Quote Link to comment
dnLL Posted January 4, 2020 Author Share Posted January 4, 2020 Simply stop Docker using the Web UI then start it.That won't prevent it from happening again. Of course I could do a script to do this every once in a while but I can't have Plex randomly reboot when busy, so most likely I'd need to keep doing it manually. I can't be the only one with this problem. Quote Link to comment
BRiT Posted January 4, 2020 Share Posted January 4, 2020 31 minutes ago, dnLL said: That won't prevent it from happening again. Of course I could do a script to do this every once in a while but I can't have Plex randomly reboot when busy, so most likely I'd need to keep doing it manually. I can't be the only one with this problem. What is your docker.log even filled with? Doing it once will get you by your issue for now. I never said to do it randomly, nor repeatedly. You could try the following, adjust 320m to be whatever increased size you require. for your /var/log/ partition. Once you're satisfied, you can then add it to your /boot/config/go script so it happens automatically upon startup. # resize log partition mount -o remount,size=320m /var/log I use 192M on mine and here's what mine looks like: #df -h /var/log Filesystem Size Used Avail Use% Mounted on tmpfs 192M 488K 192M 1% /var/log Quote Link to comment
dnLL Posted January 5, 2020 Author Share Posted January 5, 2020 30 minutes ago, BRiT said: What is your docker.log even filled with? Doing it once will get you by your issue for now. I never said to do it randomly, nor repeatedly. You could try the following, adjust 320m to be whatever increased size you require. for your /var/log/ partition. Once you're satisfied, you can then add it to your /boot/config/go script so it happens automatically upon startup. # resize log partition mount -o remount,size=320m /var/log I use 192M on mine and here's what mine looks like: #df -h /var/log Filesystem Size Used Avail Use% Mounted on tmpfs 192M 488K 192M 1% /var/log As shown in my original post I have 256MB for mine which is already double the size. Expanding it further only buys time and doesn't solve the issue. I will take a look tonight at what's in the docker logs but 200MB in 7 months ain't that bad overall considering how we use Plex alone. Quote Link to comment
dnLL Posted January 5, 2020 Author Share Posted January 5, 2020 The file is actually empty. root@server:~# ls -lh /var/log/docker.log -rw-rw-rw- 1 root root 0 Sep 13 04:40 /var/log/docker.log Something else is taking space... I have no idea how? root@server:~# du -sm /var/log/* 1 /var/log/apcupsd.events 1 /var/log/apcupsd.events.1 1 /var/log/apcupsd.events.2 1 /var/log/apcupsd.events.3 1 /var/log/apcupsd.events.4 0 /var/log/btmp 1 /var/log/btmp.1 0 /var/log/cron 0 /var/log/debug 1 /var/log/dmesg 0 /var/log/docker.log 0 /var/log/faillog 1 /var/log/lastlog 1 /var/log/libvirt 1 /var/log/maillog 0 /var/log/messages 0 /var/log/nfsd 1 /var/log/nginx 0 /var/log/packages 1 /var/log/pkgtools 0 /var/log/plugins 0 /var/log/removed_packages 0 /var/log/removed_scripts 0 /var/log/samba 0 /var/log/scripts 0 /var/log/secure 0 /var/log/setup 0 /var/log/spooler 0 /var/log/swtpm 1 /var/log/syslog 2 /var/log/syslog.1 2 /var/log/syslog.2 1 /var/log/wtmp That doesn't make sense at all, df says I have 168MB used. Quote Link to comment
BRiT Posted January 5, 2020 Share Posted January 5, 2020 And you have no way of seeing what's in /var/log/docker.log.1 ... So this is quite annoying to attempt to troubleshoot. Quote Link to comment
dnLL Posted January 5, 2020 Author Share Posted January 5, 2020 1 minute ago, BRiT said: And you have no way of seeing what's in /var/log/docker.log.1 ... So this is quite annoying to attempt to troubleshoot. I stopped the docker engine through the webUI... here are the results: root@server:~# df -h /var/log Filesystem Size Used Avail Use% Mounted on tmpfs 256M 2.9M 254M 2% /var/log root@server:~# lsof | grep deleted ipmitail 5272 root 0r REG 0,2 1598 2925 /var/spool/atjobs/a00001018cd2c1 (deleted) tail 5279 root 0r REG 0,2 1598 2925 /var/spool/atjobs/a00001018cd2c1 (deleted) What this shows is... the problem is indeed caused by dockerd. The handles on deleted files are gone, so it was most likely the deleted files. Maybe one of the docker.log actually had stuff in it, I will never know? I still think it's a problem that deleted files take space on /var/log, I will try to find a creative solution to avoid having to stop dockerd while still getting rid of the handles. 1 Quote Link to comment
BRiT Posted January 5, 2020 Share Posted January 5, 2020 It's like /var/log/docker.log.1 was using 163.x Meg or so, if this is the filesize -- 171606437. It aligns to the Size/OFF field of the LSOF output. And you cycling docker and having freed up nearly 164 Meg aligns with this. lsof /var/log/docker.log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME FD - File Descriptor # - The number in front of flag(s) is the file descriptor number of used by the process to associated with the file u - File open with Read and Write permission r - File open with Read permission w - File open with Write permission W - File open with Write permission and with Write Lock on entire file mem - Memory mapped file, usually for share library Quote Link to comment
dnLL Posted January 5, 2020 Author Share Posted January 5, 2020 The issue here is that we move files that are being used by the process (dockerd) so the handles stays there. Deleting it just remove the pointer to the file but the space isn't freed until the handle is fully closed. There is a workaround described here: https://unix.stackexchange.com/questions/68523/find-and-remove-large-files-that-are-open-but-have-been-deleted Now, why is docker.log rotated? To prevent it from taking too much space... but we don't prevent anything by rotating it, and since we delete the old file rather than archiving it, wouldn't it be better to just empty the file rather than move it and delete it? Not sure if this is a dockerd issue or unRAID issue, really. Who handles the log rotation? 1 Quote Link to comment
dnLL Posted January 5, 2020 Author Share Posted January 5, 2020 Traditionally when rotating a log manually you make a copy of it and empty the original, this way the process keeps writing in the original file and there is no handle on the copy. This is weird. Quote Link to comment
dnLL Posted January 5, 2020 Author Share Posted January 5, 2020 I'm thinking docker.log is empty solely because the process is writing in the moved (but deleted) file. Since the file is still opened by dockers, despite not having a pointer on the filesystem is can still be read (and shown in the GUI) as long as you go through the dockerd process. unRAID does have logrotate not only for dockerd. I will take a look at the config and see what I can do but I'm thinking everyone has the same problem here. Quote Link to comment
dnLL Posted January 6, 2020 Author Share Posted January 6, 2020 I don't have access to this sadly: https://access.redhat.com/solutions/3249121 unRAID uses logrotate as shown here: root@server:~# cat /etc/logrotate.d/docker /var/log/docker.log { rotate 5 notifempty missingok size=5M compress delaycompress } Another suggestion coming from the Redhat community: https://bugzilla.redhat.com/show_bug.cgi?id=1477486 Maybe I should report this as a bug for unRAID. Multiple events must happen for this bug to even exist: 1) dockerd not properly closing handles on rotated logs 2) the log rotation must move files rather than copy them 3) having limited /var/log space 4) dockerd must not be restarted for a long time Workaround: using copytruncate option with logrotate. http://man7.org/linux/man-pages/man8/logrotate.8.html I added copytruncate to the config file shown above and tested before/after with logrotate -f and lsof and I can see this fixes the issue completely. I'm gonna add a user script to modify the docker logrotate config file on boot. 1 Quote Link to comment
BRiT Posted January 6, 2020 Share Posted January 6, 2020 Yes, please file a bug report, including details there and linking back to this thread, and also and include the details of using copytruncate in the config file. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.