Michael_P

It happened back on the 13th, so whatever you were doing with one of your docker containers ran it OOM and it killed your postgres container. Reboot to clear the log so FCP stops warning you, then if it happens again investigate which container is behaving badly and fix its config or limit the RAM its allowed to use.

Looks like back on the 1st kasm ran it OOM, then on the 20th xvnc did it too - maybe also related to kasm. If it keeps happening, check its config so it's not writing to RAM or limit the container. Reboot the server to clear the log or FCP will continue to warn you since it's still in the log

The server ran OOM because of a misbehaving docker, looks like whatever was spawning all of those Chrome processes - also, your VPN username and password are exposed in the diagnostics, you should remove it from your post

1- This is the wrong forum for this 2- You get what you pay for on AliExpress 3- If you want a disk shelf, get a disk shelf, don't bother with trying to jimmy a server chassis into your setup if you're trying to get out of the ghetto

Looks like emby ran it OOM doing a media scan, you'll need to reboot to clear the log then if it happens again limit the memory allowed to the container

First issue is maybe you have a browser window open to the dashboard, if left open long enough can crash the gui The OOM may be related to whatever container is running dotnet - if you run ps auxf and save the output to a file, then sanitize any user names and passwords for running docker containers from the file, I can try to narrow it down example: ps auxf > ps.txt

Did you leave a browser open with the dashboard for a long period of time?

Could still be frigate, since its doing ffmpeg work in RAM too - but since there was only one stream going, I'd still point to the mapped location being wrong

Never used it so wouldn't be able to tell you how to configure it to disk, but here's where it was doing its work. It could still be frigate, but I see this in the log so it looks like the conversions were done in RAM /tmp/unmanic/unmanic_file_conversion-ungnp-1733949847/Yesterday (2019) 1080p.BluRay.1400MB.DD5.1.x264-GalaxyRG-ungnp-1733949847-WORKING-3-1.mkv /tmp/unmanic/unmanic_file_conversion-gogqt-1733949847/X-Men Apocalypse 2016 1080p BluRay x264 DTS-JYK-gogqt-1733949847-WORKING-3-1.mkv /tmp/unmanic/unmanic_file_conversion-tflpx-1733949847/X-Men.The.Last.Stand.2006.720p.BrRip.x264.YIFY-tflpx-1733949847-WORKING-3-1.mp4 /tmp/unmanic/unmanic_file_conversion-bjlas-1733949847/X-Men (2000)-bjlas-1733949847-WORKING-3-1.mp4

Looks like this time you were converting movies, which were going to RAM during the process

Check the power cables and remove any splitters

Keep in mind you need every drive in the array to rebuild a failed drive, so dodgy drives in the array aren't a good idea, either

I wouldn't trust it with anything but backups

Keep good backups. For me, any number >0 gets a drive tossed. Murphy's law says that number's gonna increase right when you need that drive to re-build another one.

Probably Immich You also have a container flapping, check the docker page in advanced view which container has the lowest uptime or using a port that's already in use Nov 30 01:26:52 Server kernel: docker0: port 13(veth2b88528) entered blocking state Nov 30 01:26:52 Server kernel: docker0: port 13(veth2b88528) entered disabled state Nov 30 01:26:55 Server kernel: docker0: port 13(veth2b88528) entered blocking state Nov 30 01:26:55 Server kernel: docker0: port 13(veth2b88528) entered forwarding state

I filed a bug report back in April, didn't get any traction until the last beta release. If the system has a docker container that passes username and password in its run command, it's also included in ps.txt. I would agree that they should remove all diagnostics files forum posts to be safe.

Dunno, I don't use Frigate. Worth a shot, tho - and if it doesn't work just map whatever the cache location it has in its settings to a disk.

To add - your EX900 nvme is about 400 TB over its rated endurance, might be time to replace it.

Emby is pulling down a large portion of the resources, but you also have a lot errors on one of your drives: Nov 27 11:39:03 Tower kernel: BTRFS info (device nvme1n1p1): read error corrected: ino 20006314 off 40960 (dev /dev/sdt1 sector 172007976) Nov 27 11:39:12 Tower kernel: BTRFS warning (device nvme1n1p1): csum failed root 5 ino 20006316 off 4096 csum 0x8941f998 expected csum 0xdedc4706 mirror 2

To avoid running OOM, make sure any caches are mapped to a disk - for the high CPU usage, not much you can do if you don't use a GPU to offload the transcode

Looks like Emby is causing the problem

That's the thing about flaky power supplies, they're flaky. Check all of the connections, and make sure your UPS is UPSing. If it keeps happening, odds are it's the PSU (as already said, it's a prime suspect for random reboots).

If you don't have that mapped to a disk, it's writing to RAM.

Happened on the 17th and looks like it stopped your VM, may have been frigate that ran it OOM - you'll need to reboot to clear the log so FCP stops warning you. If it happens again, check the configs for your containers so they're not writing to RAM

From my perspective, it's been very reliable - in the sense it's never outright crashed (knock on wood). Even as hardware (thumbdrives and cache drives on two occasions) was failing it just chugged along, and when the broken things were replaced, it went right back to doing what it does.