Out of shared memory errors

[wayner]

I woke up this morning to my logs full of these errors.  The system seems to be working fine, and I only have four dockers running.  Any idea what is causing these? I did a search of the forums and others have had similar issues, but it isn't clear if they fixed the problem.  However it looks like it might be related to having web browser SSH sessions open.

 

Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [crit] 26468#26468: ngx_slab_alloc() failed: no memory
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: shpool alloc failed
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: nchan: Out of shared memory while allocating message of size 234. Increase nchan_max_reserved_memory.
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: *2123245 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/paritymonitor?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: MEMSTORE:00: can't create shared message for channel /paritymonitor
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [crit] 26468#26468: ngx_slab_alloc() failed: no memory
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: shpool alloc failed
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: nchan: Out of shared memory while allocating message of size 237. Increase nchan_max_reserved_memory.
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: *2123246 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/fsState?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: MEMSTORE:00: can't create shared message for channel /fsState
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [crit] 26468#26468: ngx_slab_alloc() failed: no memory
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: shpool alloc failed
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: nchan: Out of shared memory while allocating message of size 234. Increase nchan_max_reserved_memory.
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: *2123247 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/mymonitor?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 16 07:58:16 Portrush nginx: 2023/09/16 07:58:16 [error] 26468#26468: MEMSTORE:00: can't create shared message for channel /mymonitor
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [crit] 26468#26468: ngx_slab_alloc() failed: no memory
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: shpool alloc failed
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: nchan: Out of shared memory while allocating message of size 6081. Increase nchan_max_reserved_memory.
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: *2123249 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: MEMSTORE:00: can't create shared message for channel /devices
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [crit] 26468#26468: ngx_slab_alloc() failed: no memory
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: shpool alloc failed
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: nchan: Out of shared memory while allocating message of size 234. Increase nchan_max_reserved_memory.
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: *2123250 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/arraymonitor?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: MEMSTORE:00: can't create shared message for channel /arraymonitor
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [crit] 26468#26468: ngx_slab_alloc() failed: no memory
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: shpool alloc failed
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: nchan: Out of shared memory while allocating message of size 3687. Increase nchan_max_reserved_memory.
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: *2123253 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/var?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: MEMSTORE:00: can't create shared message for channel /var
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [crit] 26468#26468: ngx_slab_alloc() failed: no memory
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: shpool alloc failed
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: nchan: Out of shared memory while allocating message of size 13833. Increase nchan_max_reserved_memory.
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: *2123254 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 16 07:58:17 Portrush nginx: 2023/09/16 07:58:17 [error] 26468#26468: MEMSTORE:00: can't create shared message for channel /disks

 

portrush-diagnostics-20230916-1142.zip

[caesay]

This is not helpful, but I just wanted to confirm that I'm having the exact same error.

[wayner]

I have tried closing all of the unRAID web UI sessions in browsers to see if this causes the problem to go away.  So far, so good, but that is only for a day and a half.

[wayner]

And I think that this thread is about the same issue:  

 

[Mainfrezzer]

just from my self experience point of view. try to remove the unraid connect plugin in see if the issue still persists.

[wayner]

3 minutes ago, Mainfrezzer said:

just from my self experience point of view. try to remove the unraid connect plugin in see if the issue still persists.

Thanks, I will try that.  Is this something that just happens in 6.12? I never saw this under 6.11.5.  I upgraded a few weeks ago and I keep running into new bugs/issues in 6.12, BTRFS cache corruption, call trace issues, and now this out of memory issue.

[Mainfrezzer]

I cant say for certain but for me its been a source of a few annoyances and it certainly meddles with nginx in some weird way. Had it runing for a long time on 6.11.5 before too.

[wayner]

On 9/18/2023 at 12:14 PM, Mainfrezzer said:

just from my self experience point of view. try to remove the unraid connect plugin in see if the issue still persists.

Now at 3.5 days with running out of shared memory issues, so it looks like this was what was causing my issue.

[wayner]

These errors have reappeared on my system after a few months without them.

 

Yesterday I switched my cache disk from XFS to ZFS and added a second cache drive to the pool in mirrored mode, and I added a Win11VM.  Overnight I started getting these errors again - my log has hundreds or thousands of these entries.  Any idea what is causing these?  I saw a theory in the past that keeping a web browser tab to your server is a potential issue.  This is a snippet of my log:

 

Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: nchan: Out of shared memory while allocating message of size 309. Increase nchan_max_reserved_memory.
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: *3132809 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/parity?buffer_length=1 HTTP/1.1", host: "localhost"
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: MEMSTORE:00: can't create shared message for channel /parity
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [crit] 3106#3106: ngx_slab_alloc() failed: no memory
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: shpool alloc failed
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: nchan: Out of shared memory while allocating message of size 234. Increase nchan_max_reserved_memory.
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: *3132810 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/paritymonitor?buffer_length=1 HTTP/1.1", host: "localhost"
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: MEMSTORE:00: can't create shared message for channel /paritymonitor
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [crit] 3106#3106: ngx_slab_alloc() failed: no memory
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: shpool alloc failed
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: nchan: Out of shared memory while allocating message of size 237. Increase nchan_max_reserved_memory.
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: *3132811 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/fsState?buffer_length=1 HTTP/1.1", host: "localhost"
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: MEMSTORE:00: can't create shared message for channel /fsState
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [crit] 3106#3106: ngx_slab_alloc() failed: no memory
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: shpool alloc failed
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: nchan: Out of shared memory while allocating message of size 234. Increase nchan_max_reserved_memory.
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: *3132812 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/mymonitor?buffer_length=1 HTTP/1.1", host: "localhost"
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: MEMSTORE:00: can't create shared message for channel /mymonitor
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [crit] 3106#3106: ngx_slab_alloc() failed: no memory
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: shpool alloc failed
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: nchan: Out of shared memory while allocating message of size 3688. Increase nchan_max_reserved_memory.
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: *3132813 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/var?buffer_length=1 HTTP/1.1", host: "localhost"
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: MEMSTORE:00: can't create shared message for channel /var
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [crit] 3106#3106: ngx_slab_alloc() failed: no memory
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: shpool alloc failed
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: nchan: Out of shared memory while allocating message of size 14281. Increase nchan_max_reserved_memory.
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: *3132814 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: MEMSTORE:00: can't create shared message for channel /disks
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [crit] 3106#3106: ngx_slab_alloc() failed: no memory
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: shpool alloc failed
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: nchan: Out of shared memory while allocating message of size 316. Increase nchan_max_reserved_memory.
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: *3132816 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/wireguard?buffer_length=1 HTTP/1.1", host: "localhost"
Dec  3 05:15:36 Portrush nginx: 2023/12/03 05:15:36 [error] 3106#3106: MEMSTORE:00: can't create shared message for channel /wireguard