Nginx Filling /var/log

[bdben]

Hi everyone,

 

I'm quite new to Unraid, having installed the OS just a couple of months ago. I'm using the Fix Common Problems app, which just alerted me that /var/log is at 100% usage. Running df shows me that this is indeed the case, along with /lib/modules and /lib/firmware, though it seems to me that those two are meant to be at 100%, not sure. I've posted my diagnostics here, but what I've found is that syslog.1 and syslog.2 are both massive. They are full of the exact same error from nginx:

Mar 25 15:10:21 Tower nginx: 2020/03/25 15:10:21 [error] 28461#28461: *796641 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Mar 25 15:10:21 Tower nginx: 2020/03/25 15:10:21 [error] 28461#28461: MEMSTORE:00: can't create shared message for channel /disks
Mar 25 15:10:22 Tower nginx: 2020/03/25 15:10:22 [crit] 28461#28461: ngx_slab_alloc() failed: no memory
Mar 25 15:10:22 Tower nginx: 2020/03/25 15:10:22 [error] 28461#28461: shpool alloc failed
Mar 25 15:10:22 Tower nginx: 2020/03/25 15:10:22 [error] 28461#28461: nchan: Out of shared memory while allocating message of size 10001. Increase nchan_max_reserved_memory.

I don't know much about nginx or unix, but it seems clear that nginx is running out of memory. The Unraid dashboard shows memory usage at around 60-75%, so I don't think the system is out of memory, but rather just this one process. I'm still able to access the WebUI, as well as docker containers that live behind a letsencrypt/nginx reverse proxy (though there is nothing new in that container's log, so I don't think it's the same nginx instance that's causing this issue). Additionally, running "top | grep nginx" shows 3 PIDs, each using 0-0.1% memory (if I'm reading it correctly).

 

My next step will be a reboot to see if the problem persists, but without changing anything I'm expecting the logs to just fill up again.

 

Has anyone seen this issue before or know what is causing it? What other troubleshooting steps can I take from here?

tower-diagnostics-20200327-1026.zip

[trurl]

Does it happen in SAFE mode?

 

Very likely unrelated, but your appdata and system shares have files on the array.

[bdben]

Thanks for the reply, I hadn't noticed that before actually. I have both shares set to prefer cache, but I added the cache after building the server. I've heard that the mover won't move files that are in use - is that why some are stuck on the array? How do I fix that?

 

Also regarding SAFE mode, I haven't tested that. I just rebooted normally and haven't seen the error from above occur yet. I also don't have any graphics output on the server, so is there a way to reboot into safe mode from the web ui? 

 

Currently I have ~700 occurrences of this error in syslog:

Mar 27 12:44:22 Tower nginx: 2020/03/27 12:44:22 [alert] 8223#8223: worker process 29904 exited on signal 6
Mar 27 12:44:23 Tower login[29951]: ROOT LOGIN  on '/dev/pts/1'

Just those two lines repeated with different worker process numbers. However it hasn't recurred after the initial boot-up, so maybe that's normal? The log file does not appear to be growing currently.

[trurl]

3 hours ago, bdben said:

mover won't move files that are in use - is that why some are stuck on the array? How do I fix that?

Settings - Docker, disable. Same for VM Manager. Then no appdata or system files will be in use. Run mover.

Mover also won't overwrite duplicate files. Those sometimes have to be cleaned up manually.

 

3 hours ago, bdben said:

is there a way to reboot into safe mode from the web ui? 

You can change the default boot option at Main - Boot Device - Flash - Syslinux Configuration.

[bdben]

Awesome, thanks! In about 5 hours of uptime, syslog has stayed around 280KB so I'm going to keep an eye on it for now. If those error messages come back I'll try safe mode and post my results here.

 

In the meantime, I'll move those system and appdata files off the array.