var/log filling up - error

[TurkeyPerson]

Woke up to this error and common issues suggests posting diagnostic online and temporary fix of restarting:
 

error: Compressing program wrote following message to stderr when compressing log /var/log/nginx/error.log.1:
gzip: stdout: No space left on device
error: failed to compress log /var/log/nginx/error.log.1

Diagnostic is uploaded.

tower-diagnostics-20231221-0742.zip

[JorgeB]

Try booting in safe mode and/or closing any browser windows open to the GUI, only open when you need to use it then close again.

[itimpi]

Something seems to have filled up the /var/log folder.   It might be worth running a command like:

du -sch /var/log/*

To see if you can spit what it is.

[TurkeyPerson]

12 hours ago, itimpi said:

Something seems to have filled up the /var/log folder.   It might be worth running a command like:

du -sch /var/log/*

To see if you can spit what it is.

I already rebooted the server so not sure that output is helpful but good to know if/when it happens again. Thanks.

 

 

12 hours ago, JorgeB said:

Try booting in safe mode and/or closing any browser windows open to the GUI, only open when you need to use it then close again.

Yeah - I'll do this and see if I run into other issues. Thanks.

[TurkeyPerson]

So it's my syslog that's filling up that directory - this is what keeps repeating:
 

Jan 11 04:40:58 Tower nginx: 2024/01/11 04:40:58 [error] 7029#7029: shpool alloc failed
Jan 11 04:40:58 Tower nginx: 2024/01/11 04:40:58 [error] 7029#7029: nchan: Out of shared memory while allocating message of size 14047. Increase nchan_max_reserved_memory.
Jan 11 04:40:58 Tower nginx: 2024/01/11 04:40:58 [error] 7029#7029: *4418775 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.>
Jan 11 04:40:58 Tower nginx: 2024/01/11 04:40:58 [error] 7029#7029: MEMSTORE:00: can't create shared message for channel /devices
Jan 11 04:40:59 Tower nginx: 2024/01/11 04:40:59 [crit] 7029#7029: ngx_slab_alloc() failed: no memory

My usage habits have not changed so I'm not leaving more/less GUI windows than before. In fact, trying to make a greater effort to close them but this has consistently been an issue since it started. Is that really the only possibility? 

Fresh diagnostics taken without a reset attached. 

tower-diagnostics-20240112-0849.zip


Edit: After doing some brief reading, I think this may be caused by NetData. Going to disable it and see if this comes back unless someone has another idea. 

Edited January 12 by TurkeyPerson