Upgraded from 7.2.4 straight to 7.3.1

Upgrade went smoothly. System rebooted, was able to spin up drives and started docker containers. All seemed fine. Went to bed and woke up to the server not being accessible on local network. Web GUI would not load, not responding to ping. Had to perform unclean shutdown. Upon reboot, things began working again. But the cycle began again, went to bed and server was not responsive the next morning.

Considering downgrading back to 7.2.4 but wanted to post here first. The Downgrade Unraid OS page recommended posting here before attempting that so here I am.

There are some other issues as well since the upgrade. Tailscale plugin does not work. At least one container will not start properly which basically kills a stack of other containers. Luckily Pi-hole and Plex still work assuming the server is running properly.

rocinante-diagnostics-20260605-2046.zip

Edited June 6Jun 6 by MACGoof

Enable the syslog server and post that if it crashes again.

Hopefully this captured it all. I am in the process of converting an old RasPi into a Syslog server for what that is worth.

syslog-previous

Jun 6 17:27:41 Rocinante php-fpm[8273]: [WARNING] [pool www] child 329491 exited on signal 9 (SIGKILL) after 476.489707 seconds from start

Jun 6 17:35:04 Rocinante php-fpm[8273]: [WARNING] [pool www] child 333974 exited on signal 9 (SIGKILL) after 276.942640 seconds from start

Jun 6 17:44:05 Rocinante php-fpm[8273]: [WARNING] [pool www] child 336161 exited on signal 9 (SIGKILL) after 644.283148 seconds from start

Jun 6 17:50:42 Rocinante php-fpm[8273]: [WARNING] [pool www] child 341492 exited on signal 9 (SIGKILL) after 555.157480 seconds from start

Jun 6 18:25:33 Rocinante php-fpm[8273]: [WARNING] [pool www] child 346740 exited on signal 9 (SIGKILL) after 589.301096 seconds from start

In my experience, these errors can be the result of the server being close to exhausting the memory, GUI can become extremely slow, like 1 minute to open the dashboard, try limiting the memory for VMs/docker services, or adding a little more RAM.

It could also be one or more containers hogging the CPU, try pinning only some cores to them, and leave cores 0/1 available for Unraid.

Also, recommend trying a couple of other things, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, and enable this:

https://docs.unraid.net/unraid-os/release-notes/7.0.0/#excessive-flash-drive-activity-slows-the-system-down

16 hours ago, JorgeB said:

Jun 6 17:27:41 Rocinante php-fpm[8273]: [WARNING] [pool www] child 329491 exited on signal 9 (SIGKILL) after 476.489707 seconds from start

Jun 6 17:35:04 Rocinante php-fpm[8273]: [WARNING] [pool www] child 333974 exited on signal 9 (SIGKILL) after 276.942640 seconds from start

Jun 6 17:44:05 Rocinante php-fpm[8273]: [WARNING] [pool www] child 336161 exited on signal 9 (SIGKILL) after 644.283148 seconds from start

Jun 6 17:50:42 Rocinante php-fpm[8273]: [WARNING] [pool www] child 341492 exited on signal 9 (SIGKILL) after 555.157480 seconds from start

Jun 6 18:25:33 Rocinante php-fpm[8273]: [WARNING] [pool www] child 346740 exited on signal 9 (SIGKILL) after 589.301096 seconds from start

In my experience, these errors can be the result of the server being close to exhausting the memory, GUI can become extremely slow, like 1 minute to open the dashboard, try limiting the memory for VMs/docker services, or adding a little more RAM.

It could also be one or more containers hogging the CPU, try pinning only some cores to them, and leave cores 0/1 available for Unraid.

Also, recommend trying a couple of other things, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, and enable this:

https://docs.unraid.net/unraid-os/release-notes/7.0.0/#excessive-flash-drive-activity-slows-the-system-down

Thanks a ton for your help. Changed Global Share Settings to max as well as the step in your link (touch /boot/config/fastusr). Will see how it goes. As for CPU and RAM resources, I'd be surprised if those were exhausted. Granted I've only got two out of over a dozen dockers running but it is still crashing with just the two.

Took your advice though and pinned CPUs. I set CPUs 2-11 for Docker and left 0-1 for the system. Does this look reasonable? Never messed with these settings before. Didn't do anything with CPU isolation fwiw. Thanks again.

Woke up to the system not responsive again. Though after a few minutes the GUI actually loads and can SSH to the box. Now we have logs prior to shutdown.

syslog-Rocinante.log

The server is running out of RAM, you have a rogue process (starter.sh) using approximately 61GB

That is likely running inside a container.

Thanks for finding that. I did update container configs to limit RAM usage.

The most recent crash was CPU resources depleted. Logs for that attached. Have reconfigured CPU pinning to be more restrictive since then. Also configured max RAM settings for containers as suggested. It should be noted that I have downgraded back to 7.2.4 but there has been no improvement. Considering updating BIOS for my MB as there are updates available. System had been stable prior to attempted update to Unraid 7.3.1 so not sure.

syslog-Rocinante (1).log

Edited June 10Jun 10 by MACGoof

Again, that starter.sh process ran the server out of RAM, it was using around 58GB of RAM

Out of memory: Killed process 733603 (start.sh) total-vm:61144552kB, anon-rss:61136512kB

This process was running inside a Docker container with this long ID string.

task_memcg=/docker/52668f00206d0cc53fecee251d6f428be8eeff903b6501bb172433553cf27258

Turns out a container that was shut down (and NOT set to auto-start) turned itself on and was causing all of this. Odd behavior and is what had me a bit confused.

Thanks for your patience and assistance.