Skip to content
View in the app

A better way to browse. Learn more.

Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

Caddy docker suddenly started causing zombie php-fpm processes to appear, eventually takes CPU to 100% and causes thermal issues

Featured Replies

Hello,

I'm unsure of where to post this since the issue isn't caused by unraid or caddy-cloudflare alone, but both of them together. It also started suddenly, with no updates or config changes to unraid or caddy-cloudflare.

Basically, around 36 hours ago my server started having a CPU temp alarm. I ssh'd in and ran htop and saw that there was at least 7 php-fpm pool processes using 100% of the CPU. I found this post which had docker as the cause. I stopped docker and waited about an hour and no processes were spawned, likely meaning a container is the primary issue. I went through all of my 45+ containers one by one, waiting good time between launching the next and eventually determined that caddy-cloudflare was the cause. I noticed that I'm under a cooldown for requesting certs but other than that there is not any relevant logging to caddy-cloudflare or the syslog.

My current setup for caddy utilizes caddy-cloudflare and compose manager. It happens with the previous and latest versions of caddy built with the cloudflare module. I'm using a few custom networks, br0 is the bridge and then I have two docker networks utilized by caddy.

Here's what I've tried so far..

  • I tried deleting the caddy_data and caddy_config volumes which delayed these processes from appearing for around 30 minutes.

  • I removed the DNS challenge config from caddy and switched to self signed internal certs. At first, this seemed to actually work! I went a whole 40 minutes with no issues and went and did something else. When I came back though, the zombies were there again.

  • I tried adding limits to the caddy stack to see if that was the cause of the issue. Nope, zombies still spawned and memory usage doesn't ever go above 8%.

  • Rebooting several times.

  • Updating caddy-cloudflare by building it locally, using the latest version of caddy-builder.

  • Disabling Home Assistant integrations which utilize caddy's reverse proxy. Time will tell if this resolves the issue, I'm still waiting to see that.

Unclear if related, but one of my drives seemed to die around the same time as this issue. I have a replacement on the way, but the timing is at least suspicious. I'll update this post if replacing the emulated drive resolves this issue.

In the meantime, I've created a script which monitors and kills these zombie php-fpm processes, as killing the processes seems to not cause other problems elsewhere and temporarily resolves the issue.

I've considered deleting the docker image, but setting up all of my services (even if sort of easy to restore under Apps) along with my custom networks would be rough. I've also thought about updating to the 7.2 beta, but it doesn't seem to have any changes related to it that would fix this issue. If disabling HA integrations and replacing the drive doesn't fix this issue, I'll first try deleting the docker image and building from scratch. If this still doesn't resolve it, I'll try updating to beta.

If anyone else has any suggestions, I appreciate it.

Edited by tekcor
Added details about updating caddy-cloudflare

  • Author

I haven't been able to quite pin this down yet. With the cloudflare TLS section of caddy-cloudflare's Caddyfile disabled and relevant home assistant integrations disabled, it didn't happen for over 24 hours. I wanted to see if maybe it was the DNS after all (which I tested previously and determined was not the case) so I re-enabled it, and within about an hour it started happening again. Perhaps I didn't wait long enough (40 minutes instead of 55) and it's been cloudflare's TLS all along. Will update further if I find out, running one more time with cloudflare TLS enabled to be sure. Mostly just updating in case someone else runs into this bizarre, frustrating issue.

  • Author

I re-enabled everything except for the HA integrations and it hasn't happened again for two days according to my monitor script. So I'm not sure what's causing this, but I suppose I can live with it for now since my script cleans it up when it happens. Very strange either way.

  • Author

I went four days without encountering this issue, which is a new record. However, suspiciously, it started happening again when I opened the unraid webui this morning to resume a parity check. It seems that it may be tied to both caddy-cloudflare running while visiting the webui.

  • 5 months later...
  • Author

I'm not sure what fixed this, but I haven't encountered this issue for months. Every now and then I'll still get a php-fpm process logged but it quickly goes away without my monitor script cleaning it up. I'm back to running pretty much everything I was before, so it remains a mystery as to what was causing this problem.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

Account

Navigation

Search

Search

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.