/var/log almost full of deleted files?


dnLL

Recommended Posts

Here:

root@server:~# df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           256M  167M   90M  65% /var/log

root@server:~# du -sm /var/log
3       /var/log

du doesn't find much and there is no mountmoint under /var/log/* either. I looked it up on the web and people mentionned it usually happens if there are deleted files/handles locked by a process such as:

root@server:~# lsof | grep "/var/log/" | grep deleted
dockerd    6282                       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282                       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  2047 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  2047 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6286 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6286 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6287 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6287 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6288 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6288 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6290 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6290 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6291 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6291 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6295 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6295 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6296 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6296 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6297 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6297 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6298 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6298 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6299 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6299 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6317 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6317 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6318 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6318 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6319 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6319 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6320 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6320 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6321 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6321 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6322 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6322 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6323 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6323 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6324 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6324 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6325 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6325 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6578 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6578 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6579 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  6579 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  8090 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282  8090 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 10337 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 10337 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 12624 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 12624 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 17100 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 17100 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 17643 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 17643 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 19395 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 19395 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 20294 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 20294 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 25049 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 25049 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 28520 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 28520 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 31502 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 31502 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 32693 dockerd         root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
dockerd    6282 32693 dockerd         root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300                       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300                       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  1082 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  1082 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  3031 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  3031 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6301 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6301 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6302 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6302 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6303 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6303 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6304 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6304 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6305 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6305 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6306 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6306 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6307 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6307 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6308 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6308 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6309 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6309 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6310 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6310 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6311 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6311 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6312 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6312 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6314 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6314 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6315 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6315 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6316 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6316 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6976 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6976 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6977 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6977 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6978 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  6978 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  7304 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  7304 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  8412 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  8412 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  9110 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  9110 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  9111 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300  9111 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 10407 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 10407 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 11343 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 11343 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 13038 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 13038 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 13371 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 13371 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 15705 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 15705 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 16646 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 16646 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 20586 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 20586 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 32149 container       root    1w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)
container  6300 32149 container       root    2w      REG               0,28   171606437      25553 /var/log/docker.log.1 (deleted)

Right... so the log rotation works I guess... but the deleted files apparently still take space on /var/log (at least that's what I'm thinking)? I didn't try (yet) to kill/restart the dockerd engine (which would remove the locked handles shown above) since I'd like to find a better way. My server has 200+ days of uptime so yes, rebooting once in a while would obviously prevent the issue but that's not an actual fix, so I'm looking for a better solution.

Link to comment
Simply stop Docker using the Web UI then start it.
That won't prevent it from happening again. Of course I could do a script to do this every once in a while but I can't have Plex randomly reboot when busy, so most likely I'd need to keep doing it manually.

I can't be the only one with this problem.
Link to comment
31 minutes ago, dnLL said:

That won't prevent it from happening again. Of course I could do a script to do this every once in a while but I can't have Plex randomly reboot when busy, so most likely I'd need to keep doing it manually.

I can't be the only one with this problem.

 

What is your docker.log even filled with? 

 

Doing it once will get you by your issue for now. I never said to do it randomly, nor repeatedly. You could try the following, adjust 320m to be whatever increased size you require. for your /var/log/ partition. Once you're satisfied, you can then add it to your /boot/config/go script so it happens automatically upon startup.

 


# resize log partition
mount -o remount,size=320m /var/log

 

I use 192M on mine and here's what mine looks like:

#df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           192M  488K  192M   1% /var/log

Link to comment
30 minutes ago, BRiT said:

 

What is your docker.log even filled with? 

 

Doing it once will get you by your issue for now. I never said to do it randomly, nor repeatedly. You could try the following, adjust 320m to be whatever increased size you require. for your /var/log/ partition. Once you're satisfied, you can then add it to your /boot/config/go script so it happens automatically upon startup.

 

 


# resize log partition
mount -o remount,size=320m /var/log

 

 

I use 192M on mine and here's what mine looks like:

 


#df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           192M  488K  192M   1% /var/log

 

As shown in my original post I have 256MB for mine which is already double the size. Expanding it further only buys time and doesn't solve the issue. I will take a look tonight at what's in the docker logs but 200MB in 7 months ain't that bad overall considering how we use Plex alone. 

Link to comment

The file is actually empty.

root@server:~# ls -lh /var/log/docker.log
-rw-rw-rw- 1 root root 0 Sep 13 04:40 /var/log/docker.log

Something else is taking space... I have no idea how?

root@server:~# du -sm /var/log/*
1       /var/log/apcupsd.events
1       /var/log/apcupsd.events.1
1       /var/log/apcupsd.events.2
1       /var/log/apcupsd.events.3
1       /var/log/apcupsd.events.4
0       /var/log/btmp
1       /var/log/btmp.1
0       /var/log/cron
0       /var/log/debug
1       /var/log/dmesg
0       /var/log/docker.log
0       /var/log/faillog
1       /var/log/lastlog
1       /var/log/libvirt
1       /var/log/maillog
0       /var/log/messages
0       /var/log/nfsd
1       /var/log/nginx
0       /var/log/packages
1       /var/log/pkgtools
0       /var/log/plugins
0       /var/log/removed_packages
0       /var/log/removed_scripts
0       /var/log/samba
0       /var/log/scripts
0       /var/log/secure
0       /var/log/setup
0       /var/log/spooler
0       /var/log/swtpm
1       /var/log/syslog
2       /var/log/syslog.1
2       /var/log/syslog.2
1       /var/log/wtmp

That doesn't make sense at all, df says I have 168MB used.

Link to comment
1 minute ago, BRiT said:

And you have no way of seeing what's in /var/log/docker.log.1 ... So this is quite annoying to attempt to troubleshoot.

I stopped the docker engine through the webUI... here are the results:

 

root@server:~# df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           256M  2.9M  254M   2% /var/log

root@server:~# lsof | grep deleted
ipmitail   5272                       root    0r      REG                0,2        1598       2925 /var/spool/atjobs/a00001018cd2c1 (deleted)
tail       5279                       root    0r      REG                0,2        1598       2925 /var/spool/atjobs/a00001018cd2c1 (deleted)

What this shows is... the problem is indeed caused by dockerd. The handles on deleted files are gone, so it was most likely the deleted files. Maybe one of the docker.log actually had stuff in it, I will never know? I still think it's a problem that deleted files take space on /var/log, I will try to find a creative solution to avoid having to stop dockerd while still getting rid of the handles.

  • Thanks 1
Link to comment

It's like /var/log/docker.log.1 was using 163.x Meg or so, if this is the filesize -- 171606437. It aligns to the Size/OFF field of the LSOF output. And you cycling docker and having freed up nearly 164 Meg aligns with this.

 

lsof /var/log/docker.log
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME

 


FD - File Descriptor

 

# - The number in front of flag(s) is the file descriptor number of used by the process to associated with the file

u - File open with Read and Write permission

r - File open with Read permission

w - File open with Write permission

W - File open with Write permission and with Write Lock on entire file

mem - Memory mapped file, usually for share library

Link to comment

The issue here is that we move files that are being used by the process (dockerd) so the handles stays there. Deleting it just remove the pointer to the file but the space isn't freed until the handle is fully closed.

 

There is a workaround described here:

https://unix.stackexchange.com/questions/68523/find-and-remove-large-files-that-are-open-but-have-been-deleted

 

Now, why is docker.log rotated? To prevent it from taking too much space... but we don't prevent anything by rotating it, and since we delete the old file rather than archiving it, wouldn't it be better to just empty the file rather than move it and delete it? Not sure if this is a dockerd issue or unRAID issue, really. Who handles the log rotation?

  • Thanks 1
Link to comment

I'm thinking docker.log is empty solely because the process is writing in the moved (but deleted) file. Since the file is still opened by dockers, despite not having a pointer on the filesystem is can still be read (and shown in the GUI) as long as you go through the dockerd process.

unRAID does have logrotate not only for dockerd. I will take a look at the config and see what I can do but I'm thinking everyone has the same problem here.

Link to comment

I don't have access to this sadly: https://access.redhat.com/solutions/3249121

 

unRAID uses logrotate as shown here:

root@server:~# cat /etc/logrotate.d/docker
/var/log/docker.log {
  rotate 5
  notifempty
  missingok
  size=5M
  compress
  delaycompress
}

Another suggestion coming from the Redhat community: https://bugzilla.redhat.com/show_bug.cgi?id=1477486

 

Maybe I should report this as a bug for unRAID. Multiple events must happen for this bug to even exist:

1) dockerd not properly closing handles on rotated logs

2) the log rotation must move files rather than copy them

3) having limited /var/log space

4) dockerd must not be restarted for a long time

 

Workaround: using copytruncate option with logrotate.

http://man7.org/linux/man-pages/man8/logrotate.8.html

 

I added copytruncate to the config file shown above and tested before/after with logrotate -f and lsof and I can see this fixes the issue completely. I'm gonna add a user script to modify the docker logrotate config file on boot.

  • Thanks 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.