December 10, 20241 yr My /var/log/syslog (and .1, .2) are full of 70MB+ of this error starting about about 22 hours ago: Dec 9 04:40:03 unraid nginx: 2024/12/09 04:40:03 [error] 14669#14669: *9624309 nchan: error publishing message (HTTP status code 507), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost" Dec 9 04:40:03 unraid nginx: 2024/12/09 04:40:03 [crit] 14669#14669: ngx_slab_alloc() failed: no memory Dec 9 04:40:03 unraid nginx: 2024/12/09 04:40:03 [error] 14669#14669: shpool alloc failed Dec 9 04:40:03 unraid nginx: 2024/12/09 04:40:03 [error] 14669#14669: nchan: Out of shared memory while allocating channel /cpuload. Increase nchan_max_reserved_memory. I've seen this come up a lot on the forums without any real resolution. Prior to this, my logs were also filling up with: Dec 8 20:05:39 unraid nginx: 2024/12/08 20:05:39 [alert] 13011#13011: worker process 10895 exited on signal 6 Dec 8 20:05:41 unraid nginx: 2024/12/08 20:05:41 [alert] 13011#13011: worker process 10896 exited on signal 6 Dec 8 20:05:41 unraid nginx: 2024/12/08 20:05:41 [alert] 13011#13011: worker process 10917 exited on signal 6 Dec 8 20:05:41 unraid nginx: 2024/12/08 20:05:41 [alert] 13011#13011: worker process 10918 exited on signal 6 I can delete the logs to free up space so I can see new logs, but that won't change the fact that this is filling up my logs. Has there been any "official" answer about what causes either of these and how to resolve them?
December 11, 20241 yr This sounds like a misconfiguration on data and pathing... The error messages you are encountering with Nginx and Nchan - ?an Nginx extension? on your Unraid system... HTTP Status Code 507: This error typically indicates that the server is out of storage space necessary to complete the request. In your context with Nchan, it suggests an inability to buffer or store messages due to insufficient memory allocation. ngx_slab_alloc() failed: no memory: This critical error message is thrown by Nginx's memory allocator when it fails to allocate memory from the slab pool, which is used for small chunks of memory. It implies that Nginx has run out of available memory for its operations. shpool alloc failed & Out of shared memory: These errors are closely related to the slab allocation failure and indicate that the shared memory pool reserved for Nchan has been exhausted. Increase nchan_max_reserved_memory: This specific message provides a direct hint on how to potentially resolve the issue. nchan_max_reserved_memory is a configuration directive in Nchan that sets the maximum amount of shared memory (in bytes) that Nchan is allowed to use for message buffering and other needs. edit nginx config: http { nchan_max_reserved_memory 128M; # Adjust the value based on your available system memory } What does htop say. You may be out of system resources. But to me this means you have something writing to unraid root in ram and not writing it to a disk. did you modify nginx? on what nexcloud? https://stackoverflow.com/questions/59991038/nginx-nchan-pubsub-how-to-modify-incoming-messages Edited December 11, 20241 yr by bmartino1 Data-tyoo
December 11, 20241 yr Author If I tried to modify the nginx config to fix this, wouldn't it just get overwritten at the next reboot? I haven't modified my Unraid nginx config. Is it possible that some App/Plugin I installed modified it? How would I tell? Skimming through all the files in /etc/nginx (including conf.d) I don't see any mentions of any of the Apps/Plugins I have installed. I don't think they've modified it. It's clear that both a) something is running out of memory, and b) the (ram)disk is out of space. The latter is caused by all the logs from the former, that's no mystery. Why is nginx running out of memory is the question.
December 11, 20241 yr yes if you are editing unraids nginx config it will not survive a reboot unless you used a user script to remove and add the file and restart the nginx service. ^NOT Recommend do at your own risk! Are you running the beat v7? This seems like the outgoin proxy issue. please post a diag file we can check syslog to see if there is a cause efect or what process caused it.
December 11, 20241 yr Author I'm not running the v7 beta. I just updated from 6.12.13 to 6.12.14. I've updated and rebooted since the problem happened, so there's not much in the logs right now, but I'll attach diagnostics anyway. unraid-diagnostics-20241211-1126.zip
December 11, 20241 yr the diag had the previous syslog form last boot which helped in looking more in to the error/issue. You might need to fiel a bug report. as I'm not seing a reason why the is happening, but this appears to be a memory issues within the nginx. The temp solution as i'm not sure if there is a logrotate set to nginx. Default: Quote root@BMM-Unraid:~# cd /etc/logrotate.d/ root@BMM-Unraid:/etc/logrotate.d# ls apcupsd docker libvirtd libvirtd.libxl libvirtd.lxc libvirtd.qemu mcelog nginx ntp rsyslog slim unraid-api vsftpd root@BMM-Unraid:/etc/logrotate.d# cat nginx /var/log/nginx/*.log { su nobody users rotate 10 notifempty size 5M compress delaycompress sharedscripts postrotate /etc/rc.d/rc.nginx rotate endscript } root@BMM-Unraid:/etc/logrotate.d# logrotate /etc/logrotate.d/nginx may need to be edited^ cron run the abve every hour for temp relief... Othersiwse there may be other settign we need to set to unraid nginx config... *as this may be related to the outgoin proxy setting being develped... the errors seem to start here: Quote Dec 10 04:40:02 unraid nginx: 2024/12/10 04:40:02 [crit] 27478#27478: ngx_slab_alloc() failed: no memory Worker processes frequently crash with segmentation faults (segfault at 0 ip 0000000000000000), indicating critical memory access violations. Nchan-specific messages such as nchan: Out of shared memory while allocating channel repeatedly reference channels like /dockerload, /cpuload, /disks, and /shares. with HTTP status code 507 you may need to use tools like htop, free -m, or top to identify memory consumption in real time. to try and find the culprit. Otherwise i think we may need to test the nginx conf ... so letsLocate the Nginx Configuratiom find /etc/nginx -name "*.conf" Common locations include: /etc/nginx/nginx.conf Quote root@BMM-Unraid:/etc/nginx# ls conf.d/ dhparam.pem@ fastcgi.conf fastcgi_params mime.types nginx.conf win-utf root@BMM-Unraid:/etc/nginx# we would need to cat the nginx config and make a user script at first aray start to replace the unraid nginx config with the changes. Defeat backup on v7 rc1: Quote root@BMM-Unraid:/etc/nginx# cat nginx.conf # Unraid user root; worker_processes 1; pid /var/run/nginx.pid; events { worker_connections 1024; # Accept as many connections as possible after getting a notification about a new connection multi_accept on; # Which polling method we should use for multiplexing clients on to threads. If you’re using Linux 2.6+, you should use epoll. use epoll; } # Default error log file # (this is only used when you don't override error_log on a server{} level) # # limetech - here we write log messages to the raw socket where rsyslogd is listening. # Another way to do it is with this line: # error_log syslog:server=localhost,nohostname; # and then also uncomment these two lines in /etc/rsyslog.conf: # #$ModLoad imudp # provides UDP syslog reception # #$UDPServerRun 514 # start a UDP syslog server at standard port 514 # Not sure which method is better... # error_log syslog:server=unix:/dev/log,nohostname; http { ## # Basic Settings ## # Hide nginx version information. server_tokens off; # Define the MIME types for files. include /etc/nginx/mime.types; default_type application/octet-stream; # How long to allow each connection to stay idle; longer values are better # for each individual client, particularly for SSL, but means that worker # connections are tied up longer. (Default: 65) keepalive_timeout 20; # Speed up file transfers by using sendfile() to copy directly # between descriptors rather than using read()/write(). sendfile on; # Tell Nginx not to send out partial frames; this increases throughput # since TCP frames are filled up before being sent out. (adds TCP_CORK) tcp_nopush on; # Tell Nginx to enable the Nagle buffering algorithm for TCP packets, which # collates several smaller packets together into one larger packet, thus saving # bandwidth at the cost of a nearly imperceptible increase to latency. (removes TCP_NODELAY) tcp_nodelay on; # Turn off buffering for proxy requests to be able to support 'flush' during progress updates proxy_buffering off; client_header_timeout 20; client_body_timeout 20; client_body_buffer_size 32k; send_timeout 20; fastcgi_read_timeout 640s; fastcgi_keep_conn on; types_hash_max_size 2048; client_max_body_size 20m; server_names_hash_bucket_size 128; server_name_in_redirect off; port_in_redirect off; ## # Logging Settings ## # Format to use in log files log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; # Default log file # (this is only used when you don't override access_log on a server{} level) # access_log /var/log/nginx/access.log; access_log off; ## # Gzip Settings ## # Disable Gzip compression to support 'flush' progress updates gzip off; ## # SSL Settings ## # Protect against the BEAST attack by preferring RC4-SHA when using SSLv3 and TLS protocols. # Note that TLSv1.1 and TLSv1.2 are immune to the beast attack but only work with OpenSSL v1.0.1 and higher and has limited client support. # SSLv3 removed due to POODLE exploit and CRIME attack # 3DES removed due to SWEET32 attach ssl_protocols TLSv1.3 TLSv1.2; ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!3DES"; ssl_ecdh_curve secp384r1; ssl_prefer_server_ciphers on; ssl_dhparam /etc/nginx/dhparam.pem; # Optimize SSL by caching session parameters for 10 minutes. This cuts down on the number of expensive SSL handshakes. # The handshake is the most CPU-intensive operation, and by default it is re-negotiated on every new/parallel connection. # By enabling a cache (of type "shared between all Nginx workers"), we tell the client to re-use the already negotiated state. # Further optimization can be achieved by raising keepalive_timeout, but that shouldn't be done unless you serve primarily HTTPS. ssl_session_cache shared:SSL:10m; # a 1mb cache can hold about 4000 sessions ssl_session_timeout 10m; ssl_session_tickets off; map $http_upgrade $connection_upgrade { default Upgrade; '' close; } ## # Authentication Settings ## satisfy any; allow 127.0.0.1; allow ::1; allow unix:; deny all; auth_request /auth-request.php; # limit the amount of failed auth requests per IP address limit_req_zone $binary_remote_addr zone=authlimit:1m rate=30r/m; ## # Security Settings ## add_header X-Content-Type-Options 'nosniff'; add_header Referrer-Policy 'same-origin'; ## # set root directory for requests ## # root /usr/local/emhttp; include /etc/nginx/conf.d/servers.conf; } root@BMM-Unraid:/etc/nginx# nano /etc/nginx/nginx.conf Look for the http {} or server {} block and adjust Nchan memory settings: http { ... nchan_max_reserved_memory 64M; # Increase based on available memory nchan_message_timeout 30s; # Consider adding this to clear old messages nchan_message_buffer_length 100; # Buffer size adjustment if needed ... } the restart the service: /etc/rc.d/rc.nginx restart *While this will restart the service. you may lose access to the web UI... run this at your own risk... as on my test system i got a call trace error... It is better to stop then start this service... I have terminal access to the physical machine so login and run the same comamnd to stop then start brought the web ui back #stop start as one command /etc/rc.d/rc.nginx stop && /etc/rc.d/rc.nginx start nginx should have its own error log. we may need to view that as well: tail -f /var/log/nginx/error.log example: root@BMM-Unraid:~# tail -f /var/log/nginx/error.log 2024/12/11 22:53:55 [info] 1625103#1625103: Using 116KiB of shared memory for nchan in /etc/nginx/nginx.conf:160 2024/12/11 22:53:55 [info] 1625103#1625103: Using 131072KiB of shared memory for nchan in /etc/nginx/nginx.conf:160 user scipt to run at boot to do this: #!/bin/bash # Unraid User Script to modify nginx.conf and restart Nginx # Define file paths NGINX_CONF="/etc/nginx/nginx.conf" BACKUP_CONF="/etc/nginx/nginx.conf.bak" # Backup the existing nginx.conf if [ -f "$NGINX_CONF" ]; then cp "$NGINX_CONF" "$BACKUP_CONF" echo "Backup created at $BACKUP_CONF" else echo "Error: $NGINX_CONF not found! Exiting." exit 1 fi # Add the new configuration settings sed -i '/^http {/a \ nchan_max_reserved_memory 64M;\n nchan_message_timeout 30s;\n nchan_message_buffer_length 100;' "$NGINX_CONF" if [ $? -eq 0 ]; then echo "Nginx configuration updated successfully." else echo "Error: Failed to update Nginx configuration." exit 1 fi # Restart Nginx /etc/rc.d/rc.nginx stop && /etc/rc.d/rc.nginx start if [ $? -eq 0 ]; then echo "Nginx restarted successfully." else echo "Error: Failed to restart Nginx. Check configuration." exit 1 fi exit 0 Edited December 12, 20241 yr by bmartino1 Data-typo
December 11, 20241 yr if your not comfortable using a scripted work around that may or may not work. You may need to downgrade back down to 6.12.13 Unraid Web UI > tools > Downgrade OS Edited December 12, 20241 yr by bmartino1 any fix due to error would require a eidt to nginx
December 12, 20241 yr Author I'm not sure why I would need to downgrade. This error was happening on 6.12.13, before I upgraded.
December 13, 20241 yr 9 hours ago, SurprisingEdge said: I'm not sure why I would need to downgrade. This error was happening on 6.12.13, before I upgraded. it was my understanding you had this bug only in the latest updated. in which case downgrading would be your best bet. did you try the nginx edit?
December 13, 20241 yr Author Na, these errors happened before the upgrade. This isn't the first time I've had them either. I have not tried editing the nginx config, that's a can of worms I would really rather not open. So far the only thing I seem to have found that stops them is going to every one of my devices that might have a browser tab open to my Unraid server, and closing all the tabs. It's like there's way too much resource consumption on the server for every browser tab open. I've been limiting how many tabs I have open and haven't had the error since, but I only had like 5 tabs open before so it really doesn't seem like it should be a problem. The server itself has 32GB and is only ever showing like max 30% memory usage. I'm not running VMs, just a bunch of docker containers.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.