I have been tooting through some troubleshooting with chatgpt to get some initial troubleshooting done. I did a full reset of facial recognition with immich, and I think that's what initially led to instability and crashing. My system as 32GB DDR5 RAM, 2GB reserved for a Homeassistant VM. That's off. I stopped all the immich dockers (four of them: server, machine_learning, redis, postgres) and I'll leave those off for now. I did turn on mirror logs to flash drive, looks like log_previous is in cluded in this attached diagnostics zip. Crafty 4 (minecraft server) is also a memory hog, as well as the seafile-elasticsearch docker. Those are all stopped now as well.

If this is immich slowly gobbling up all the memory and crashing the server, well that shouldn't be happening. I can start to chase that down over at the immich forums, or here.

Thanks in advance!

mayorgoodway-diagnostics-20260921-1046.zip

Edited September 21Sep 21 by rutherford

The below is AI slop, but I'm working outside of what I'm capable of here. I hope some of this helps someone. It's not an answer, it's where I'm at right now.

chatgpt AI status update

Update / more troubleshooting:

I've brought Immich back online and have been reproducing the facial-recognition rebuild with considerably more instrumentation. I also put hard Docker memory limits on the Immich containers so that, hopefully, I can observe the failure without allowing one container to consume the entire host.

So far this is pointing much more strongly toward memory growth in immich_server during the facial-recognition rebuild.

immich_machine_learning does not appear to be running away. Its cgroup memory peaked around 2.6 GB and has since fallen toward ~1 GB while facial recognition continues processing. There have been no ML cgroup OOM events.

immich_server is a very different story. I initially limited it to 6 GB. It reached that ceiling and the cgroup recorded 1,106 memory.max events, although there were no OOM/OOM-kill events. I raised the limit to 10 GB and subsequently 12 GB so I could continue observing it without immediately killing the workload.

Once given additional headroom, immich_server continued growing steadily rather than immediately leveling off. Over one measured ~4.5-minute interval its cgroup memory increased from roughly 6.3 GB to 8.0 GB. A few minutes later it was around 9.6 GB. Importantly, most of the increase is anonymous/application memory rather than just reclaimable file cache — at ~9.6 GB cgroup usage, approximately 7.1 GB was anonymous memory.

At the same time, the host itself remains responsive. It still had roughly 14 GB MemAvailable, no swap, CPU was mostly idle, and I/O wait was around 6–8%. The NVMe cache devices have not shown anything resembling 100% utilization or huge latency, there are no persistent D-state processes in the spot checks, and I've seen no OOM, hung-task, BTRFS error, NVMe reset/timeout, or I/O-error messages in the live kernel log.

Facial Detection and Facial Recognition job concurrency are both set to 1.

This makes my current working hypothesis that the original "whole server grinds to a halt" behavior is caused by immich_server continually increasing its memory footprint during a large facial-recognition rebuild. Without a Docker memory limit, it may eventually consume enough of this 32 GB machine to put the host into severe reclaim/thrashing. That would also explain why I wasn't necessarily seeing a clean OOM kill before the machine became effectively unusable.

I don't want to label this a confirmed Immich memory leak yet because I'm still letting the rebuild run to see whether the memory eventually plateaus. For now I've left immich_server capped at 12 GB and immich_machine_learning at 10 GB, and the Unraid server is still snappy while the rebuild continues.

I'm continuously logging host memory/reclaim stats, vmstat, iostat, Docker stats, D-state/top processes, kernel messages, ML/server cgroup memory (including anon/file breakdown and OOM/max events), and the Immich server/ML logs. So if it deteriorates again I should have much better evidence than I did during the previous crashes.

Your previous log does capture a substantial stall: multiple Unraid services, including the WebGUI and Docker, were blocked waiting for filesystem reads. There is no recorded OOM-kill decision that identifies the cause. The diagnostics’ memory snapshot was collected after reboot, so it cannot show memory availability during that stall.

Your later measurements make immich_server memory growth worth investigating. Facial recognition runs between the server container and PostgreSQL, so relatively low machine-learning container usage is consistent with that workload.

Also, the 1,106 max events indicate attempts to exceed the container’s memory limit—not 1,106 OOM kills. The separate oom and oom_kill counters matter here.

Keep the memory limits in place for now, with room for Unraid, PostgreSQL, Redis, and other applications. Avoid continually raising the server limit.

Please provide the exact Immich version, approximate library/face count, and timestamped server logs and memory readings from your current test. Does memory eventually level off and the rebuild finish, or does it keep growing until it reaches the limit?

If the host slows again, collect fresh diagnostics before rebooting, if possible. The memory and I/O measurements from that same period will help distinguish memory pressure from a separate read stall.

The facial recognition wrapped up over night. Immich version 3.2.2, unraid 7.3.2.

from the immichdb queries:

face_count is 273k, person_count is 10k

We are stretching a bit past the limits of my skill here! I've attached three files that I hope answer your request there Jorge "timestamped server logs and memory readings from your current test."

I also got a related, and possibly helpful, response over at Immich Discord:

Stang70Fastback — 9/18/26, 7:41 PM

Is it normal for Immich Server to be using such a crazy amount of RAM? I though maybe it was still working on stuff since I did just upload like 19,000 images and videos, but the administrative panel shows NO active jobs, and CPU usage appears to be minimal...

mertalev - 9/18/26, 9:14 PM
i think the node.js version in the latest release has a memory leak that keeps image files loaded for ml in memory. you can see how the ml container received nearly as much data as the server is using. we’ve updated node.js to a fixed version but haven’t made a release for it yet. as a workaround, restarting the container would free the excess memory

jeppeoland — 9/19/26, 3:44 AM

Well, for me, after updating to 3.2.2, I have lots of broken images/thumbnails. Server memory climbs to full in like a minute. And I get lots of HTTP 500's from the server as well. Rebooting doesn't fix.

oom-memory-windows.tsv first-oom-immich-server.log oom-events.txt

oom-events.txt confirms four container-scoped OOM events at 21:07, 21:52, 22:44, and 23:33. Each event killed the immich process with about 12.3 GB of anonymous memory.

The TSV shows the same pattern. Memory reaches the 12 GB limit, drops below 1 GB after the process restarts, and then grows again. Therefore, immich_server did not level off during this rebuild. It repeatedly reached the limit and restarted.

The memory limit appears to have protected the host. The facial-recognition rebuild still completed despite those restarts. I would keep the limit in place and leave sufficient memory for Unraid, PostgreSQL, Redis, and your other containers. Do not keep raising the limit.

This is strong evidence of an Immich 3.2.2 server-side memory problem during the rebuild. It does not identify the exact software defect. The Discord explanation is plausible, and Immich issue #31488 (https://github.com/immich-app/immich/issues/31488) reports a similar per-asset immich_server memory-growth pattern. However, that report concerns a different job, so it is not a confirmed match.

I suggest reporting this upstream to Immich with these three files, version 3.2.2, and the counts of approximately 273,000 faces and 10,000 people. Until Immich releases a confirmed fix, restart immich_server after large machine-learning jobs if its memory remains high.

This also makes memory pressure a more plausible explanation for the original whole-server slowdown. However, the earlier log showed many services blocked on filesystem reads and did not record a host OOM.

So we cannot conclusively join the two events yet. If the host slows again while the 12 GB limit is active, please collect diagnostics before rebooting.