April 27, 20251 yr I posted a few days ago about my server becoming unresponsive, but my logs looked OK according to feedback. Last night, I noticed similar sluggishness setting in for the UI -- header would load but drives weren't appearing etc -- and my logs were flooded with "error 4 in nginx[424000+116000] likely on CPU..." Based on a few searches, I've uninstalled the Unraid Connect plugin, but I wanted to check and see if this is a potential hardware issue or if it's definitely software related... As always, any help would be greatly appreciated syslog-192.168.50.233.log
April 27, 20251 yr Community Expert would have preferred a diag fill not just a syslog... I have to asuem intell... My first bet is that this is a intel efficency core over performance core log for nginx... An Nginx error 4, typically displayed as "error 4 segfault", often points to a segmentation fault within the Nginx process, specifically in a Perl-related component are normally always memory related... so lets break this down a bit... Quote error 4 in nginx[424000+116000] likely on CPU X (core Y, socket Z) This line means: error 4 = Memory violation → an invalid memory access (usually a NULL or bad pointer). nginx[424000+116000] = The nginx binary started at memory address 424000, and its size is about 0x116000 bytes (around 1MB+). likely on CPU X = Which CPU and which logical core it crashed on (important because Intel Efficiency Cores and Performance Cores behave differently). On Intel Alder Lake, Raptor Lake, and newer CPUs (like your 12400) — there are two types of cores: P-Cores (Performance) = Strong single-threaded performance. E-Cores (Efficiency) = Lower-power, lower-latency cores, good for background tasks. some known problems with intel... Problem with nginx crashing on E-Cores: If nginx processes are affinitized (stuck) to E-Cores, under heavy load they may: Stall. Be starved for CPU cycles. Suffer memory mismanagement (context switching bugs on less performant E-cores). Some Linux kernel versions and Docker versions still don't properly account for hybrid core types. Memory-related errors can occur if nginx tries to allocate/free memory at high load while being "locked" to an efficiency core. so theroy and guess fixes: Not enough RAM? → unlikely if you have free RAM, but could be docker container limits. Kernel scheduler not properly balancing workloads across P/E cores. also not sure if this is nginx docker, nginx ssl for emhttp or nginx for unrid proxy mangement... need more info... as if its dokcer related we can adjust soem memore options.. --memory=512m --memory-swap=1024m *Extra Parmarters if this is the only error your seeing its most likely a microcode issue with intel user P or E cores... and can be ignored. If you cant access the web ui or experience other crasehes then ther is a misconfiguration and potential issues with nginx on the system...
April 27, 20251 yr Author Just now, bmartino1 said: would have preferred a diag fill not just a syslog... I have to asuem intell... My first bet is that this is a intel efficency core over performance core log for nginx... An Nginx error 4, typically displayed as "error 4 segfault", often points to a segmentation fault within the Nginx process, specifically in a Perl-related component are normally always memory related... so lets break this down a bit... This line means: error 4 = Memory violation → an invalid memory access (usually a NULL or bad pointer). nginx[424000+116000] = The nginx binary started at memory address 424000, and its size is about 0x116000 bytes (around 1MB+). likely on CPU X = Which CPU and which logical core it crashed on (important because Intel Efficiency Cores and Performance Cores behave differently). On Intel Alder Lake, Raptor Lake, and newer CPUs (like your 12400) — there are two types of cores: P-Cores (Performance) = Strong single-threaded performance. E-Cores (Efficiency) = Lower-power, lower-latency cores, good for background tasks. some known problems with intel... Problem with nginx crashing on E-Cores: If nginx processes are affinitized (stuck) to E-Cores, under heavy load they may: Stall. Be starved for CPU cycles. Suffer memory mismanagement (context switching bugs on less performant E-cores). Some Linux kernel versions and Docker versions still don't properly account for hybrid core types. Memory-related errors can occur if nginx tries to allocate/free memory at high load while being "locked" to an efficiency core. so theroy and guess fixes: Not enough RAM? → unlikely if you have free RAM, but could be docker container limits. Kernel scheduler not properly balancing workloads across P/E cores. also not sure if this is nginx docker, nginx ssl for emhttp or nginx for unrid proxy mangement... need more info... as if its dokcer related we can adjust soem memore options.. --memory=512m --memory-swap=1024m *Extra Parmarters if this is the only error your seeing its most likely a microcode issue with intel user P or E cores... and can be ignored. If you cant access the web ui or experience other crasehes then ther is a misconfiguration and potential issues with nginx on the system... I’m running a Xeon 1276v3. Syslog was all I could grab, since I log that to a share
April 27, 20251 yr Community Expert https://www.intel.com/content/www/us/en/products/sku/80915/intel-xeon-processor-e31276-v3-8m-cache-3-60-ghz/specifications.html thank you for confirming the intel cpu type... ok this doesn't apear to be the e/p core issue. does unraid boot at all? web ui unresponsive... Console terminal available? did you edit and code or have scripts that auto run? Please post a diagnostic file... unraid should still boot. at terminal log in and type diagnostic, a zip folder will be on the flash drive.
April 27, 20251 yr Author 7 minutes ago, bmartino1 said: https://www.intel.com/content/www/us/en/products/sku/80915/intel-xeon-processor-e31276-v3-8m-cache-3-60-ghz/specifications.html thank you for confirming the intel cpu type... ok this doesn't apear to be the e/p core issue. does unraid boot at all? web ui unresponsive... Console terminal available? did you edit and code or have scripts that auto run? Please post a diagnostic file... unraid should still boot. at terminal log in and type diagnostic, a zip folder will be on the flash drive. Thanks so much for the assistance, so far. Here is a current syslog I, until just now, had a NVIDIA Max Stream script that autoran at startup. I just deleted that, as I'm not using a GPU. That was my only script at startup. The issue seems to happen at night, usually around 11PM-midnight, though that could be entirely coincidental, and I'm not using the server when it happens. I've just been loading the UI a log, since I had a preclear running tower-diagnostics-20250427-1417.zip
April 27, 20251 yr Community Expert Digging though the diag especial the syslog-previous... I am indeed seeing a nginx crashes around 1 am ish... Quote Apr 27 01:46:05 Tower kernel: nginx[15788]: segfault at 7f8b6b5ea4e0 ip 0000157c34b45f2f sp 00007fffb18fc740 error 4 in nginx[157c34ad4000+97000] Apr 27 01:46:05 Tower kernel: Code: 0f 85 8b 00 00 00 48 8b 45 d0 4c 89 65 d0 48 8b 45 d8 4c 89 65 d8 48 8b 45 e0 4c 89 65 e0 4c 89 65 e8 0f b7 45 90 66 89 45 88 48 8b 45 e8 <48> 8b 38 48 8b 46 08 48 89 45 e8 48 8b 46 10 48 89 45 f0 48 8b 46 18 DetailValue Causenginx process tried to dereference (read) a memory address that was invalid. TimingHappened around 1:46 AM, during system idle or low-activity time. Kernel NoticeClear kernel segfault (not just nginx stopping). Possible TriggersMemory corruption, OOM killer earlier, exhausted file descriptors, broken nginx worker/thread. I think your hitting OOM causing a cascade failure into nginx. if you don't thing this is OOM we may need to increase memory for your nginx by using a script to add worker_rlimit_nofile 65536; nginx usually needs worker_rlimit_nofile set high --I will need to make a script at array start to fix this. we can also try sysctl -w vm.max_map_count=262144 sysctl -w fs.file-max=2097152 Addition system ctl updates. only if OOM is not the issue... if OOM btrfs cache disk install the unraid swapfile plugin and see if swap is used to try and capture the resource hog with htop command... Also notable: There was no clear OOM killer event immediately before this nginx crash, but memory may have still been low or fragmented. Edited April 27, 20251 yr by bmartino1 Data - typo
April 27, 20251 yr Author 4 minutes ago, bmartino1 said: Digging though the diag especial the syslog-previous... I am indeed seeing a nginx crashes around 1 am ish... DetailValue Causenginx process tried to dereference (read) a memory address that was invalid. TimingHappened around 1:46 AM, during system idle or low-activity time. Kernel NoticeClear kernel segfault (not just nginx stopping). Possible TriggersMemory corruption, OOM killer earlier, exhausted file descriptors, broken nginx worker/thread. I think your hitting OOM causing a cascade failure into nginx. if you don't thing this is OOM we may need to increase memory for your nginx by using a script to add worker_rlimit_nofile 65536; nginx usually needs worker_rlimit_nofile set high --I will need to make a script at array start to fix this. we can also try sysctl -w vm.max_map_count=262144 sysctl -w fs.file-max=2097152 Addition system ctl updates. only if OOM is not the issue... if OOM btrfs cache disk install the unraid swapfile plugin and see if swap is used to try and capture the resource hog with htop command... Also notable: There was no clear OOM killer event immediately before this nginx crash, but memory may have still been low or fragmented. OOM = Out of Memory? I have 32GB of RAM. Not sure what on earth my NAS could be doing that would use that memory -- or at least NEED that memory. In regards to what "I think", I have no idea, so I trust your judgement. Assuming it's the system running out of memory, how can I avoid that? Is there a way to monitor/record what is using RAM and how much is being used? Should I implement each of those script lines you proposed in your last? Can I add them via one startup script, or do I need each to be a separate script?
April 27, 20251 yr Community Expert Honesty this seems more like pre ram failure. I would highly recommend running memtest at unraid grub launch. I've recommended some small things only because this is acting like OOM and segfaults are memory / ram issues... I've yet to encounter this personally and went searching for potential caused and fixes. but they are trial and error at this point... SO, at this point its htop monitor and try to capture what happened where to cause nginx which appears to be the emhttp https overlay for unraid in this case. the systemctl can help, the swap plugin will use swap if OOM occurs (more a precaution) making a user script at first arr start to add or edit the existing nginx confg on unraid is trickey due to reboot. a script essential has to "sed/grep" the file insert the line worker_rlimit_nofile 65536; and restart nginx to apply that config... some nginx script was made here for another issues that has been fixed: Edited April 27, 20251 yr by bmartino1 typo
April 27, 20251 yr Author 1 minute ago, bmartino1 said: Honesty this seems more like pre ram failure. I would highly recommend running memtest at unraid grub launch. I've recommend some small things only because this is acting like OOM and segfaults are memory / ram issues... I've yet to encounter this personally and went searching for potential caused and fixes. but they are trial and error at this point... SO, at this point its htop monitor and try to caputre what happend where to cause nginx which appears to be the emhttp https overlay for unraid in this case. the systemctl can help, the sawp plugin will use swap if OOM occurs (more a precaution) making a user script at frist arr start to add or edit the exisitng nginx confg on unraid is trakey do to reboots. a script eseantal has to "sed/grep" teh file insert the line worker_rlimit_nofile 65536; and restart nginx to apply that config... some nginx script was made here for another issues that has been fixed: I was having issues a few months back, which turned out to be PSU (replaced), and a memtest ran for a day or so without issue. I'll run again.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.