Michael_P

No, when the system doesn't have anymore RAM to allocate to a process it invokes OOM and the reaper goes and tries to find the process it thinks is running away and kills it to save the system from crashing

FCP will continue to warn you until you reboot the server and the log is cleared - if it sees OOM in the log, it sends an alert

I don't use it, but see if there's a upnp setting in there and disable it if your network isn't using it. Google is your friend there

MakeMKV was doing something at the time, so makes sense. Yesterday, it ran OOM yesterday too at 22:17 but never finished logging what it killed.

It ran OOM at 07:15 and whatever is trying to open port 51820 is failing. Python was doing something at the time which ran the host OOM Edited to add, MakeMKV was doing something too which tipped it over

When the host ran OOM, it killed your VM because it was the largest consumer at the time. If your VM is set up properly and leaving enough RAM for the rest of the system, I'd say it was probably immich that ran it OOM. Reboot the system to clear the log, if it happens again in the future, limit the RAM available to the container

CRCs are always connection related, so I'd try replacing the cables again, making sure not to tie them all in a bundle. How hot is the PCIe card getting?

Good to hear Does this mean we can expect security enhancements to make this less of a home-centric OS?

Strong possibility

If the plex scheduled maintenance was scheduled around the same time, that may be a clue, too - the media scanner can run away if it chokes on a file and run the host OOM

Qbittorrent is using quite a bit of RAM, but nothing else is jumping out at me. Reboot to clear the log, and if it happens again start by checking the config for it to make sure it's not storing files in RAM. The next step would be limiting RAM available to the containers.

You have a lot going on in there, and it's killing different things each time it runs OOM, sonar, qbittorent, python3 - might have something to do with your tdarr instance, but can't be sure with those logs save the output of ps auxf > ps.txt and have a look at what container is spawning what - if you scrub any usernames and passwords from that file, you can post it here

When a sector can't be written to for whatever reason, it's marked as pending, if it fails again on the next attempt then it will reallocate it to one of the spares. It has nothing to do with the slot, unless you were shaking the drive while in the slot while it was being written to... The drive is bad, but it's your data so go nuts. Keep good backups, because if you need that drive to restore another one, cross your fingers, if another sector goes bad your parity is lost.

Reallocated sectors are internal drive errors, changing slots wouldn't matter. if you use that drive in the array, prepare for hurt feelings. WD drives are really good at telling you when they should be replaced - listen to them.

Have you tried the sleep plugin? In conjunction with your board's RTC wake it should work for you. But, you should note that turning stuff on and off a lot could actually do more harm to your components than just leaving it running.

WD says that drive's warranty expired 6 years ago, so it's pretty old

Not sure if you're trying to convince yourself or what, but if a new drive has a reallocated sector it's defective and should be returned to the retailer. Even on a used drive, it's not to be trusted - and if still under warranty should be returned to the manufacturer as it's an indicator for drive failure (and why it's tracked by SMART). Roll the dice if you like, but have good backups.

Anything greater than 0 is cause for return.

Looks like back on the 24th, Plex media scanner ran away and was killed. If it happens again, it's likely choking on a corrupt file so you should investigate further, but for now it's safe to reboot to clear the log so FCP stops warning you.

Jan 25 15:00:19 Gandalf kernel: Tasks state (memory values in pages): Jan 25 15:00:19 Gandalf kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Jan 25 15:00:19 Gandalf kernel: [ 15417] 0 15417 110 13 24576 0 0 s6-svscan Jan 25 15:00:19 Gandalf kernel: [ 16050] 0 16050 55 6 24576 0 0 s6-supervise Jan 25 15:00:19 Gandalf kernel: [ 16060] 0 16060 52 1 24576 0 0 s6-linux-init-s Jan 25 15:00:19 Gandalf kernel: [ 16157] 0 16157 55 6 24576 0 0 s6-supervise Jan 25 15:00:19 Gandalf kernel: [ 16158] 0 16158 55 5 24576 0 0 s6-supervise Jan 25 15:00:19 Gandalf kernel: [ 16159] 0 16159 57 5 36864 0 0 s6-supervise Jan 25 15:00:19 Gandalf kernel: [ 16160] 0 16160 58 7 36864 0 0 s6-supervise Jan 25 15:00:19 Gandalf kernel: [ 16168] 0 16168 52 6 24576 0 0 s6-ipcserverd Jan 25 15:00:19 Gandalf kernel: [ 31279] 0 31279 407 15 36864 0 0 busybox Jan 25 15:00:19 Gandalf kernel: [ 31538] 99 31538 217065 193497 1847296 0 0 python3 Jan 25 15:00:19 Gandalf kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0-1,oom_memcg=/docker/ab76d1381c7334fc7cd737076fe1fa4f0fdc64862fe2ae1fdb92920d53cb96c0,task_memcg=/docker/ab76d1381c7334fc7cd737076fe1fa4f0fdc64862fe2ae1fdb92920d53cb96c0,task=python3,pid=31538,uid=99 Jan 25 15:00:19 Gandalf kernel: Memory cgroup out of memory: Killed process 31538 (python3) total-vm:868260kB, anon-rss:773988kB, file-rss:0kB, shmem-rss:0kB, UID:99 pgtables:1804kB oom_score_adj:0 Jan 25 15:00:40 Gandalf kernel: kworker/u97:1 invoked oom-killer: gfp_mask=0xc00(GFP_NOIO), order=0, oom_score_adj=0 It was a running process at the time, you can probably narrow it down if you run "ps auxf" from the terminal and see what's spawning what

On the 25th whatever one of your containers was doing with Python (busybox perhaps?) ran the system OOM - reboot to clear the log so FCP stops warning you, and if it happens again try limiting the amount of RAM available to the container

The drive needs to be replaced, it's always fatal D: 184 Hex: 0xB8 Attribute name: End-to-End error / IOEDC Better: Lower Critical: Yes Description: This attribute is a part of Hewlett-Packard's SMART IV technology, as well as part of other vendors' IO Error Detection and Correction schemas, and it contains a count of parity errors which occur in the data path to the media via the drive's cache RAM

Also make sure you're not passing the GPU to more than one VM and not also using it for Docker containers or the host itself.

Whatever docker container was running Python back on the 18th around 16:30 ran the server OOM - if you remember doing something at the time, then you can narrow it down to which one. Else, check the result of ps auxf or run ps auxf > ps.txt making sure to sanitize any user names and passwords it may contain, then you can check which container is running what. Since it only happened once, it shouldn't be too much to worry about - if it happens again you can limit the memory to the container. Remember to reboot the server to clear the log so FCP stops warning you

Looks like the system ran OOM on December 9th, then again on the 19th - both times it killed your VM, doesn't necessarily mean it was the issue but if you were messing around with it on those days then it's a start. Could also be Immich, and the system is just killing your VM because it has the largest amount allocated, so keep an eye on that too. In the meantime, reboot to clear the log so FCP stops warning you, then if it happens again you can investigate further.