Michael_P

You'll still need to reboot or FCP will warn you since it's in the log

Yep, it's hitting the limit on the container Memory cgroup out of memory: Killed process 2086114 (Plex Media Serv)

You might try a different PSU, too, just to rule out flaky power

Clipable - still a bad idea tho if you want to open it up to the general public

Map the /tmp directory to a disk to see what it's storing there, maybe that'll help you figure out whatever it's doing. The other option is to limit the RAM available to the container

Rule out the bad RAM first or it will just keep sending bad data

Anything after 2021 will have the 170TB endurance "limit"

I get ya, but with passing memtest and if it's 'out of the blue', the drive wear looks like a definite point of interest. No shade at PNY, but I consider them to be a tier 2 OEM, and if the drive says its reached 100% of its lifespan I'd say it's time to replace - and the 33% drive might have the OG TBW factor instead of their 'updated' one for that series, it's likely really at 100% too

nvme0 is Data Units Read: 1,624,580,180 [831 TB] Data Units Written: 284,023,005 [145 TB] nvme1 is Data Units Read: 492,647,517 [252 TB] Data Units Written: 234,470,619 [120 TB] Rated writes for those are ~170 or so according to my google-fu, could be getting flaky

Maybe, but those drives have a ton of writes and are both probably end of life

Without the previous logs it's not possible to tell if the OOM is related to the crashes, but it's probably not. In this case Jellyfin was probably transcoding and ran the host OOM. You can re-boot to clear the log so FCP stop warning you about it, and if it happens again you can reconfigure it to transcode to disk or limit the RAM allowed to the container.

This is the likely answer even if you're offloading ML.

Looks like on the 15th of Nov and on the 14th of Dec a python process ran away, if you enable the advanced view on the docker tab you can see if 9c26809c57e9 matches any of your containers - it's the one that spawned the process. Once you've fixed it, reboot to clear the log so FCP stops warning you.

Looks like whatever you or your server was doing back on the 11th around 1752 had a java process run away with all of the RAM. If you were doing something at that time (transcodes, immich. whatever is creating all of those docker-proxy processes, etc) you can ignore it, and if it keeps happening you can investigate which one needs to be put on a diet. You can reboot to clear the log so FCP stops warning you.

Looks like you have a pretty low RAM system, from a modern standpoint, so whatever the system was doing around 2340 to around 2350 on the 7th ran itself OOM (which on the last event the kernel killed docker to save itself). You can add more RAM to alleviate the problem, or limit the RAM available to your services. You will need to reboot to clear the log so FCP stop warning you, too.

This

Your system didn't OOM, your nvidia card fell over and FCP saw 'out of memory' and generated the warning - try updating the driver and system BIOS. Then, you can reboot to clear the log so FCP stops warning you Dec 1 20:15:57 227CS kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359 Dec 1 20:15:57 227CS kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:354 Dec 1 20:15:57 227CS kernel: NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from pRmApi->Alloc(pRmApi, device->session->handle, isSystemMemory ? device->handle : device->subhandle, &physHandle, isSystemMemory ? NV01_MEMORY_SYSTEM : NV01_MEMORY_LOCAL_USER, &memAllocParams, sizeof(memAllocParams)) @ nv_gpu_ops.c:4922

Probably, since it was einstein that was being killed repeatedly back on the 29th and 30th. Bad memory wouldn't cause an OOM, only using too much RAM will. Fix whatever the container is doing, or limit the memory available to the container itself, and reboot to clear the log.

Sort of, while it was the nvidia card falling over, it didn't cause your system to go OOM (it would take and awful lot of log to fill up available RAM and knock the host over). AI is great, but it only ever gets it sort of correct. FCP saw the string 'out of memory' in the log and spit out the warning. Nov 30 05:53:56 Tower kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359You need to update the GPU driver and/or motherboard BIOS - if that fails, set it to use a lower PCIE generation in BIOS to see if the errors stop

Looks like plex pushed it over, limit the memory allowed to the container or reconfigure to cache to disk - reboot to clear the log so FCP stop warning you You should also try to figure out what's going on here, it's flooding the log Nov 30 09:09:46 hoth avahi-daemon[2760]: Registering new address record for fdf9:beec:fb34:6c40:XXXX:XXXX:XXXX:f1f9 on br0.*. Nov 30 09:16:46 hoth avahi-daemon[2760]: Withdrawing address record for fdf9:beec:fb34:6c40:XXXX:XXXX:XXXX:f1f9 on br0. Nov 30 09:17:05 hoth avahi-daemon[2760]: Registering new address record for fdf9:beec:fb34:6c40:XXXX:XXXX:XXXX:f1f9 on br0.*. Nov 30 09:26:51 hoth avahi-daemon[2760]: Withdrawing address record for fdf9:beec:fb34:6c40:XXXX:XXXX:XXXX:f1f9 on br0. Nov 30 09:28:06 hoth avahi-daemon[2760]: Registering new address record for fdf9:beec:fb34:6c40:XXXX:XXXX:XXXX:f1f9 on br0.*. Nov 30 09:35:28 hoth avahi-daemon[2760]: Withdrawing address record for fdf9:beec:fb34:6c40:XXXX:XXXX:XXXX:f1f9 on br0. Nov 30 09:36:40 hoth avahi-daemon[2760]: Registering new address record for fdf9:beec:fb34:6c40:XXXX:XXXX:XXXX:f1f9 on br0.*.

Looks like maybe you're immich container is pushing your server over the edge, you'll need to add more RAM to the server or limit the amount of RAM your services and VM is allowed. Reboot to clear the log so FCP stops warning you

You've got it right - make sure it's actually transcoding to disk, and if all else fails you can limit the memory allowed to the container. It could be choking on a particular file, too - judging by how it spikes rapidly (I've had that happen with Plex as well). So you can investigate that angle by narrowing down any recent additions to the library

RIP

Sounds like you're recruiting a mule lol. Pro tip: never carry a package internationally given to you by a stranger.

Looks like both events happened back on the 3rd, and it killed your VM both times. If your Plex service is transcoding to RAM, that may have had something to do with it. Reboot to clear the logs, and if it happens again you can try putting some of your services on a diet