webUI down - nginx restarted and running 504 timeout

[srfnmnk]

Hi,

 

I woke up this morning to my web ui being down. I restarted nginx and still I had the issue.

 

I've attached the diagnostics for review. I also looked in the nginx logs to see what I could find and found the following. My server had 32G+ of mem available as per free -h command.

 

I rebooted the server and the UI is working again, but I'd like to understand what happened and how to avoid this in the future.

 

Thank you.

 

pumbaa-diagnostics-20231108-1007.zip

[JorgeB]

Try closing any browser windows open to the GUI, only open when you need to use it then close again.

[srfnmnk]

as far as I know, there was only one browser window open from one PC.

[JorgeB]

14 minutes ago, srfnmnk said:

there was only one browser window open from one PC.

Close that one.

[srfnmnk]

ok will try this next time if it happens again. thanks

[srfnmnk]

Ok so today -- I logged in and received this little present. The ui was blank-ish. Looking at the nginx logs I see the following. I attached the full log. Why is nchan_max_reserved_memory running out? How to fix it?

 

gth=1 HTTP/1.1", host: "localhost"
2023/11/09 09:26:26 [error] 19497#19497: MEMSTORE:00: can't create shared message for channel /disks
2023/11/09 09:26:27 [crit] 19497#19497: ngx_slab_alloc() failed: no memory
2023/11/09 09:26:27 [error] 19497#19497: shpool alloc failed
2023/11/09 09:26:27 [error] 19497#19497: nchan: Out of shared memory while allocating message of size 21383. Increase nchan_max_reserved_memory.
2023/11/09 09:26:27 [error] 19497#19497: *550567 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_len
gth=1 HTTP/1.1", host: "localhost"
2023/11/09 09:26:27 [error] 19497#19497: MEMSTORE:00: can't create shared message for channel /disks
2023/11/09 09:26:28 [crit] 19497#19497: ngx_slab_alloc() failed: no memory
2023/11/09 09:26:28 [error] 19497#19497: shpool alloc failed
2023/11/09 09:26:28 [error] 19497#19497: nchan: Out of shared memory while allocating message of size 22385. Increase nchan_max_reserved_memory.
2023/11/09 09:26:28 [error] 19497#19497: *550573 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/update2?buffer_l
ength=1 HTTP/1.1", host: "localhost"
 

 

error.log

[srfnmnk]

I did read through this forum post -- it looks like leaving the browser open can result in this issue. I did leave a browser with it open. Please help us escalate this issue, leaving a browser on a web page overnight should not result in a complete crash of the nginx server.

 

restarting nginx seems to have resolved the issue.

 

Thank you.

[JorgeB]

22 minutes ago, srfnmnk said:

I did leave a browser with it open

I did mention

 

22 hours ago, JorgeB said:

Try closing any browser windows open to the GUI, only open when you need to use it then close again.

 

[srfnmnk]

Understood -- I'm just saying that this is a bug and the community would love to see this resolved. 

[JorgeB]

Not necessarily, it doesn't happen to most people, myself included, seems that having the browser tabs going to sleep may be one of the causes, so worth checking if that's the case.