/Var/Log Filling Up 6.9.2

[Beowu1f]

Hi!

 

My server logs having been filling up very fast. I have been adding additional drives: pre-clearing and re-building parity so I have not been able to reboot for the last few days. So I have expanding the size allocated, now up to 640M! I'd like to figure out the root cause of the issue and resolve rather than continuing to expand file size and rebooting on a regular basis. I've seen some other posts about this but haven't been able to fix so I figured I'd start my own post. I can DM diagnostics, appreciate any help.

 

See below for some more details of the issue.

 

Nov 15 08:13:03 Quaranta nginx: 2021/11/15 08:13:03 [crit] 84532#84532: ngx_slab_alloc() failed: no memory Nov 15 08:13:03 Quaranta nginx: 2021/11/15 08:13:03 [error] 84532#84532: shpool alloc failed Nov 15 08:13:03 Quaranta nginx: 2021/11/15 08:13:03 [error] 84532#84532: nchan: Out of shared memory while allocating message of size 1166. Increase nchan_max_reserved_memory. Nov 15 08:13:03 Quaranta nginx: 2021/11/15 08:13:03 [error] 84532#84532: *2464512 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/cpuload?buffer_length=1 HTTP/1.1", host: "localhost" Nov 15 08:13:03 Quaranta nginx: 2021/11/15 08:13:03 [error] 84532#84532: MEMSTORE:00: can't create shared message for channel /cpuload

 

 

root@Quaranta:~# df -h /var/log

Filesystem      Size  Used Avail Use% Mounted on

tmpfs           640M  539M  102M  85% /var/log

root@Quaranta:~# du -sm /var/log/*

1 /var/log/btmp

0 /var/log/cron

0 /var/log/debug

1 /var/log/diskinfo.log

1 /var/log/dmesg

1 /var/log/docker.log

0 /var/log/faillog

1 /var/log/gitflash

1 /var/log/lastlog

1 /var/log/libvirt

0 /var/log/maillog

0 /var/log/messages

0 /var/log/nfsd

245 /var/log/nginx

0 /var/log/packages

1 /var/log/pkgtools

0 /var/log/plugins

1 /var/log/preclear.disk.log

0 /var/log/pwfail

0 /var/log/removed_packages

0 /var/log/removed_scripts

0 /var/log/removed_uninstall_scripts

1 /var/log/samba

0 /var/log/scripts

0 /var/log/secure

0 /var/log/setup

0 /var/log/spooler

0 /var/log/swtpm

28 /var/log/syslog

167 /var/log/syslog.1

100 /var/log/syslog.2

1 /var/log/vfio-pci

1 /var/log/wtmp

[Squid]

Have you been leaving Main / Dashboard open all this time on any device to monitor the progress?

[Beowu1f]

Yes I have been. In the last few hours it has gone from 85% to 93%. In addition to the UI open I have SSH into the server. I have closed the UI and will see if it continues to expand by tonight. Would it be OK to leave SSH open to monitor instead?

[Beowu1f]

In the last hour with only SSH open (no GUI) the usage has gone from 93% to 96%.

 

12:01pm PT

Filesystem      Size  Used Avail Use% Mounted on

tmpfs           640M  592M   49M  93% /var/log

 

1:18pm PT

Filesystem      Size  Used Avail Use% Mounted on

tmpfs           640M  609M   32M  96% /var/log

[Beowu1f]

Logs are normal after rebuilding party was completed