MULTIPLE PHP-FPM CHILD EXITED ON SIGNAL 9

[greendragon]

I have tried upgrading from 6.11.5 to 6.12.4 and had to downgrade back because of the webui coming to a halt after 24hours. I would get tons of these PHP warnings in syslog and webui would become unusable with CPU spiking to max. After some time I tried upgrading again to new version 6.12.6 and same exact thing is happening webui stops responding and all of these warnings come into syslog occasionally nginx crashes. Why is this happening in any version i try after 6.11.5. Please let me know how i can fix this it looks like others are experiencing the same thing. Reloading nginx sometimes does not fix this and i have to do full reboot.

[JorgeB]

Try booting in safe mode and/or closing any browser windows open to the GUI, only open when you need to use it then close again.

[greendragon]

Everything runs fine though in 6.11.5 what changed?

[JorgeB]

Could be for example an incompatible plugin with v6.12, it can work with v6.11 fine, hence why you should try safe mode.

[greendragon]

47 minutes ago, JorgeB said:

Could be for example an incompatible plugin with v6.12, it can work with v6.11 fine, hence why you should try safe mode.

Ok booted into safe mode no plugins i will leave it and see what happens

[DontWorryScro]

Following.

Just curious. Say you boot into Safe mode.  then what?  what is the troubleshooting process? 

[JorgeB]

9 minutes ago, DontWorryScro said:

what is the troubleshooting process? 

Use the server normally and see if you have the same issue or not.

[DontWorryScro]

5 minutes ago, JorgeB said:

Use the server normally and see if you have the same issue or not.

unraid runs normally in safe mode?  does it disable all your plugins? anything else?  docker containers still operate ok? 

 

[JorgeB]

5 minutes ago, DontWorryScro said:

unraid runs normally in safe mode?

Yes, except for plugins, VMs and containers run normally.

[greendragon]

Just to give an update, my unraid did run fine with no plugins for over 24 hours without webui crashing. I went through and deleted all plg files under boot folder except community apps. Rebooted normally. Then installed just 2 plugins, appdata backup and user scripts. It ran fine again for over 24 hours then I installed Dynamix Cache Dir and after 15-18 hours my unraid started seeing all of these php errors again and by this morning webui was completely crashed had to reboot. I will now be leaving it running without Dynamix Cache Dir and confirm its this plugin causing the issues.

[JorgeB]

1 hour ago, greendragon said:

I will now be leaving it running without Dynamix Cache Dir and confirm its this plugin causing the issues.

Please post an update when you can confirm if that's really the issue or not, don't remember that plugin causing these issues for other users.

[greendragon]

On 12/15/2023 at 9:05 AM, JorgeB said:

Please post an update when you can confirm if that's really the issue or not, don't remember that plugin causing these issues for other users.

So far 3 days in without Dynamix Cache Dir and no webui warnings/crashes. I wonder if its the previous data of Dynamix Cache Dir from 6.11.5 that was causing it since i know people using this plugin who started unraid at 6.12.x and had no issues. I am going to leave it for a few more days before installing any other plugins. Do you know if its safe to delete the whole Dynamix Cache Dir folder instead of just deleting the plg under /boot/config/plugins? then I could try fresh install.

[JorgeB]

2 hours ago, greendragon said:

Do you know if its safe to delete the whole Dynamix Cache Dir folder

It's safe, but it will also delete the settings.

[DontWorryScro]

On 12/18/2023 at 5:21 AM, greendragon said:

So far 3 days in without Dynamix Cache Dir and no webui warnings/crashes. I wonder if its the previous data of Dynamix Cache Dir from 6.11.5 that was causing it since i know people using this plugin who started unraid at 6.12.x and had no issues. I am going to leave it for a few more days before installing any other plugins. Do you know if its safe to delete the whole Dynamix Cache Dir folder instead of just deleting the plg under /boot/config/plugins? then I could try fresh install.

 

Any developments?  I had gone straight to deleting that plugin to test and sure enough the errors kept popping for me.  So for me it has to be something else.

[DontWorryScro]

this is really becoming too much a pain to me to be worth it.  i regret finally jumping from 6.11.5 which ran flawless for me and I will most likely be going back.

[spydermixx]

Just to chime in with my experience, I've been having the same errors since upgrading from 6.11 but have never had Dynamix Cache Dir installed. I've managed to narrow my issues down to mostly occurring after or during Appdata Backups has run. I'm not saying it's specifically that plugin that's the cause of the issue, just that I can successfully cause enough php errors to completely tank Unraid even with no WebGUI windows open by triggering a manual backup operation.

Edited December 28, 2023 by spydermixx

[Gromit83]

UPDATE:

I removed the unnassigned drives plugin and did the AMD CPU power setting thingy for Ryzen. Seems to be stable now.

UPDATE2: Since I got asked about the AMD power thingy Link: AMD power setting thingy for Ryzen

 

ORIGINAL:

I'm having these errors after 6.12.5. It ran stable since before 6.12.4 I think. Thought it was my NVME adapter so fleshed out 250 Euros for a QNAP one. Guess I could have saved that money.

 

This error also brings down the cache drives and makes the GUI mostly non responsive. The server also won't power down with the power button. A hard reset tho makes everything work for 2-3 days and then just comes crashing down.

 

Turned on logging to the flash drive, but the log has nothing of interest really. Only this. Making me believe it's some pluginerror as described above here.

 

Here are the plugins I use:
 

Quote

ca.update.applications.plg    7.26 KB    2023-10-06 19:04    flash
community.applications.plg    107 KB    2024-02-10 01:00    flash
dynamix.system.autofan.plg    4.28 KB    2023-04-21 13:14    flash
dynamix.system.temp.plg    5.63 KB    2023-03-03 12:43    flash
dynamix.unraid.net.plg    25.7 KB    2024-01-13 20:12    flash
fix.common.problems.plg    21.4 KB    2024-01-21 01:00    flash
folder.view.plg    8.02 KB    2024-01-24 20:56    flash
gpustat.plg    9.27 KB    2023-12-10 01:00    flash
it87-driver.plg    3.89 KB    2023-04-21 13:12    flash
NerdTools.plg    9.47 KB    2024-02-07 16:45    flash
nvidia-driver.plg    17.2 KB    2024-01-22 01:00    flash
unassigned.devices-plus.plg    5.58 KB    2023-12-02 01:55    flash
unassigned.devices.plg    105 KB    2024-02-10 16:27    flash
unbalanced.plg    3.22 KB    2024-01-27 22:16    flash
user.scripts.plg    7.45 KB    2023-10-06 19:04    flash

 

Server log from flashdrive.

 

Quote

Feb 10 17:58:36 SERVER php-fpm[5507]: [WARNING] [pool www] child 25204 exited on signal 9 (SIGKILL) after 54.524903 seconds from start
Feb 10 17:59:02 SERVER php-fpm[5507]: [WARNING] [pool www] child 25205 exited on signal 9 (SIGKILL) after 80.675137 seconds from start
Feb 10 17:59:17 SERVER php-fpm[5507]: [WARNING] [pool www] child 25705 exited on signal 9 (SIGKILL) after 40.972817 seconds from start
Feb 10 17:59:22 SERVER php-fpm[5507]: [WARNING] [pool www] child 25728 exited on signal 9 (SIGKILL) after 39.427096 seconds from start
Feb 10 17:59:31 SERVER php-fpm[5507]: [WARNING] [pool www] child 25729 exited on signal 9 (SIGKILL) after 47.915070 seconds from start
Feb 10 17:59:33 SERVER php-fpm[5507]: [WARNING] [pool www] child 25952 exited on signal 9 (SIGKILL) after 15.759806 seconds from start
Feb 10 18:14:57 SERVER php-fpm[5507]: [WARNING] [pool www] child 26023 exited on signal 9 (SIGKILL) after 348.896421 seconds from start
Feb 10 18:16:55 SERVER php-fpm[5507]: [WARNING] [pool www] child 26562 exited on signal 9 (SIGKILL) after 988.126817 seconds from start
Feb 10 18:16:56 SERVER nginx: 2024/02/10 18:16:55 [error] 5616#5616: *597668 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.86.131, server: , request: "POST /plugins/dynamix.system.temp/include/SystemTemp.php HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.86.210", referrer: "http://192.168.86.210/Main"
Feb 10 18:16:56 SERVER nginx: 2024/02/10 18:16:55 [error] 5616#5616: *597668 auth request unexpected status: 504 while sending to client, client: 192.168.86.131, server: , request: "POST /plugins/dynamix.system.temp/include/SystemTemp.php HTTP/1.1", host: "192.168.86.210", referrer: "http://192.168.86.210/Main"
Feb 10 18:16:56 SERVER nginx: 2024/02/10 18:16:55 [error] 5616#5616: *597527 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.86.131, server: , request: "POST /plugins/unassigned.devices/UnassignedDevices.php HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.86.210", referrer: "http://192.168.86.210/Main"
Feb 10 18:16:56 SERVER nginx: 2024/02/10 18:16:55 [error] 5616#5616: *597527 auth request unexpected status: 504 while sending to client, client: 192.168.86.131, server: , request: "POST /plugins/unassigned.devices/UnassignedDevices.php HTTP/1.1", host: "192.168.86.210", referrer: "http://192.168.86.210/Main"
Feb 10 18:16:56 SERVER nginx: 2024/02/10 18:16:55 [error] 5616#5616: *597667 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.86.131, server: , request: "POST /plugins/unassigned.devices/UnassignedDevices.php HTTP/1.1", subrequest: "/auth-request.php", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.86.210", referrer: "http://192.168.86.210/Main"
Feb 10 18:16:56 SERVER nginx: 2024/02/10 18:16:55 [error] 5616#5616: *597667 auth request unexpected status: 504 while sending to client, client: 192.168.86.131, server: , request: "POST /plugins/unassigned.devices/UnassignedDevices.php HTTP/1.1", host: "192.168.86.210", referrer: "http://192.168.86.210/Main"

 

Edited March 6 by Gromit83
Update with new information2