/var/log is full

I recently moved my Unraid server to a new system (new MB, CPU, etc) by moving the drives as described in Space Invaders great video on the subject. Everything seems to be working fine. The system will run for several days with no problems and then overnight it will fill up /var/log and while it is still running it will grind to a halt. I looked at /var/logs and it seems to be filling up the syslog and nginx directories. Any idea what could be causing this? I have included a diagnostics I was able to dump from the command line.

It seems I get a bunch of "worker process" errors and it then it turns into nginx Memory Allocation errors. This is a copy of the logs showing both:

2022/04/13 05:46:01 [alert] 7418#7418: worker process 5166 exited on signal 6
ker process: ./nchan-1.2.15/src/store/spool.c:479: spool_fetch_msg: Assertion `spool->msg_status == MSG_INVALID' failed.
2022/04/13 05:46:01 [alert] 7418#7418: worker process 5208 exited on signal 6
ker process: ./nchan-1.2.15/src/store/spool.c:479: spool_fetch_msg: Assertion `spool->msg_status == MSG_INVALID' failed.
2022/04/13 05:46:02 [alert] 7418#7418: worker process 5210 exited on signal 6
2022/04/13 05:46:02 [crit] 5277#5277: ngx_slab_alloc() failed: no memory
2022/04/13 05:46:02 [error] 5277#5277: shpool alloc failed
2022/04/13 05:46:02 [error] 5277#5277: nchan: Out of shared memory while allocating message of size 8902. Increase nchan_max_reserved_memory.
2022/04/13 05:46:02 [error] 5277#5277: *3781670 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
2022/04/13 05:46:02 [error] 5277#5277: MEMSTORE:00: can't create shared message for channel /devices
ker process: ./nchan-1.2.15/src/store/spool.c:479: spool_fetch_msg: Assertion `spool->msg_status == MSG_INVALID' failed.
2022/04/13 05:46:03 [alert] 7418#7418: worker process 5277 exited on signal 6
2022/04/13 05:46:03 [crit] 5322#5322: ngx_slab_alloc() failed: no memory
2022/04/13 05:46:03 [error] 5322#5322: shpool alloc failed
2022/04/13 05:46:03 [error] 5322#5322: nchan: Out of shared memory while allocating message of size 8902. Increase nchan_max_reserved_memory.
2022/04/13 05:46:03 [error] 5322#5322: *3781681 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
2022/04/13 05:46:03 [error] 5322#5322: MEMSTORE:00: can't create shared message for channel /devices
2022/04/13 05:46:03 [crit] 5322#5322: ngx_slab_alloc() failed: no memory
2022/04/13 05:46:03 [error] 5322#5322: shpool alloc failed
2022/04/13 05:46:03 [error] 5322#5322: nchan: Out of shared memory while allocating message of size 8902. Increase nchan_max_reserved_memory.
2022/04/13 05:46:03 [error] 5322#5322: *3781692 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
2022/04/13 05:46:03 [error] 5322#5322: MEMSTORE:00: can't create shared message for channel /devices

wrighthome-diagnostics-20220413-0847.zip

Edited April 13, 20224 yr by TX_Pilot
Update with more details