Michael_P

When you are able to get in you can run ps -auxf and that will tell you which container is kicking off the ffmpeg processes

No

FCP just sees the "Out of Memory" from your log, it's not the host running OOM it's your nVidia card falling over. Try updating drivers/BIOS see if that fixes it or drop it down to gen 4/3

Looks like it was at boot, and doesn't appear to be anything to worry about - I've seen a few people have MCEs at boot with ryzen cpus, check for an updated bios see if it stops.

Back on the 5th thru around the 8th one of your containers was hitting its memory limit and was being killed, you can toggle advanced view on the docker tab to find whatever container id this matches to figure out which one it is: d922e3e766ff If you've already fixed the issue, you can reboot to clear the log to have FCP stop warning you. Also, NUT appears to be spamming your log, you should fix whatever it's complaining about Oct 22 17:44:46 T upsmon[3194985]: Poll UPS [[email protected]] failed - Data stale Oct 22 17:44:47 T rc.nut: Writing NUT configuration... Oct 22 17:44:48 T usbhid-ups[3194949]: Reconnecting. If you saw "nut_libusb_get_interrupt: Input/Output Error" or similar message in the log above, try setting "pollonly" flag in "ups.conf" options section for this driver!

Tdarr hit its limit and was killed, not tailscale Oct 21 18:54:44 Tower kernel: Memory cgroup out of memory: Killed process 66443 (Tdarr_Server) total-vm:15349868kB, anon-rss:14818028kB, file-rss:35480kB, shmem-rss:0kB, UID:99 pgtables:99960kB oom_score_adj:0

For storage file names, I'd ditch the encode data unless you really need it for something and just keep the imdb if you're worried about Plex or whatever matching

Looks like maybe sonarr was doing work that ran the host OOM, you can reboot to clear the log so it stops warning you and if it happens again, check its config or limit the RAM to the container. You also have problems with really long filenames unable to be copied/moved so you should trim them down to a more reasonable length that the filesystem can manage. Instance: Sep 21 06:00:07 Tower shfs: copy_file: /mnt/cache/media/Movies/The Devil Comes at Night (2023)/The Devil Comes at Night (2023) [imdb-tt23571868][WEBDL-1080p][8bit][h264][EAC3 5.1]-RABiDS.mkv /mnt/disk23/media/Movies/The Devil Comes at Night (2023)/The Devil Comes at Night (2023) [imdb-tt23571868][WEBDL-1080p][8bit][h264][EAC3 5.1]-RABiDS.mkv.partial (5) Input/output error

Looks like it to me. Keep it to a MAX of 4 per connector and stay far far away from the SATA splitters and any other molded style splitter connectors. I added my own additional connectors to the PSU feed lines, they punch down just like ethernet jacks. In my experience, Toshiba drives are very sensitive to power fluctuation.

I don't see anything crazy, qbittorrent is using a lot of CPU but not too out there, try to run it again next time it acts up

And you haven't change the default zfs cache amount? (1/8th system ram) what does ps -auxf look like?

If they were all transcoding to RAM, then that'd do it. You can try caching to disk instead for a while and/or limiting memory to the container. FWIW I have my plex container limited to 4G since if it finds a file it doesn't like during a scan it will spike to infinite RAM usage in a couple of seconds.

The slowdowns are likely because you're very close to running OOM, that's where these are coming from and will make the server unresponsive: Oct 9 20:46:14 unServerD php-fpm[12266]: [WARNING] [pool www] child 3078784 exited on signal 9 (SIGKILL) after 163.772122 seconds from start Oct 9 20:46:28 unServerD php-fpm[12266]: [WARNING] [pool www] child 3080607 exited on signal 9 (SIGKILL) after 76.074364 seconds from start Oct 9 20:46:38 unServerD php-fpm[12266]: [WARNING] [pool www] child 3081694 exited on signal 9 (SIGKILL) after 24.022172 seconds from start Oct 9 20:46:40 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082525 exited on signal 9 (SIGKILL) after 12.009674 seconds from start Oct 9 20:46:50 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082530 exited on signal 9 (SIGKILL) after 12.010067 seconds from start Oct 9 20:46:52 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082531 exited on signal 9 (SIGKILL) after 12.011153 seconds from start Oct 9 20:47:16 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082557 exited on signal 9 (SIGKILL) after 26.023875 seconds from start Oct 9 20:47:18 unServerD php-fpm[12266]: [WARNING] [pool www] child 3082633 exited on signal 9 (SIGKILL) after 26.023854 seconds from start It also looks like plex is choking on one of your files, could be corrupt or just because of the OOM issues Oct 9 21:06:19 unServerD kernel: Plex Transcoder[3143683]: segfault at 18 ip 0000150df840ea23 sp 00007ffc332bb280 error 4 in libavcodec.so.60[4f1a23,150df80a0000+406000] likely on CPU 17 (core 11, socket 1)

You have a lot of ffmpeg transcodes probably from frigate. Change it's cache location so that it's mapped to a disk share instead of RAM

With 96G of RAM you don't need swap, I run an absolute ton of services and a couple of chunky VMs on 32

Try leaving it off for a bit and see if it stops going OOM, I've seen these processes in a couple OOM reports in the last week and both times it was nginx proxy manager (its node process in particular) Oct 9 08:07:39 unServerD kernel: [ 22401] 0 22401 320314 8283 7968 315 0 1306624 0 0 node Oct 9 08:07:39 unServerD kernel: [ 22447] 0 22447 13121 1127 961 166 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22448] 0 22448 13121 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22449] 0 22449 13188 1484 961 523 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22450] 0 22450 13153 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22451] 0 22451 13155 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22452] 0 22452 13112 1141 1041 100 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22453] 0 22453 13112 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22454] 0 22454 13078 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22455] 0 22455 13078 1015 961 54 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22456] 0 22456 13011 1463 961 502 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22457] 0 22457 12978 1321 961 360 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22458] 0 22458 13011 1123 961 162 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22459] 0 22459 13011 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22460] 0 22460 13011 965 961 4 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22461] 0 22461 13011 1067 961 106 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22462] 0 22462 13012 961 961 0 0 81920 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22463] 0 22463 13012 1126 961 165 0 81920 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22464] 0 22464 13011 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22465] 0 22465 13064 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22466] 0 22466 13054 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22467] 0 22467 13012 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22468] 0 22468 13011 1146 961 185 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22469] 0 22469 13011 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22470] 0 22470 13011 1314 961 353 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22471] 0 22471 13011 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22472] 0 22472 13011 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22473] 0 22473 13011 1234 1041 193 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22474] 0 22474 13088 1049 961 88 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22475] 0 22475 13088 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22476] 0 22476 13012 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22477] 0 22477 13046 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22478] 0 22478 13120 1582 961 621 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22479] 0 22479 13087 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22480] 0 22480 13046 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22481] 0 22481 13046 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22482] 0 22482 13087 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22483] 0 22483 13046 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22484] 0 22484 13089 1041 1041 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22485] 0 22485 13045 1237 1041 196 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22486] 0 22486 13121 961 961 0 0 86016 0 0 nginx Oct 9 08:07:39 unServerD kernel: [ 22487] 0 22487 12720 1555 835 720 0 81920 0 0 nginx

Just for shits and giggles, put a memory limit on your nginx proxy manager and see if that gets killed before the whole host goes oom - start with 1GB

Seeing as how the kernel was reaping different processes, that suggests it wasn't something running away and instead was just the sever being over-utilized - instead of trying to swap it out, just configure either your tdarr or plex to transcode to disk since that's where it will end up anyway with the swap file.

Looks like you have tdarr and plex transcodes going at the same time, if they're both set to transcode to RAM try running it without tdarr for a bit and see if it stops running oom

Do you have something bound to port 80?

Probably not related Go to the docker tab and toggle the advanced view and see which container has this ID (or close to it) 7d3934da0c4f - it may be nginx proxy manager as another user recently had the same node process running way going OOM and being killed. I can't offer any advice as to why it's doing that, I have been running it for years and have never seen it do that, best I can advise is to re-install it. You will need to restart the sever to clear the log You should also try to figure out what's going on with these, it's flooding your log Oct 3 14:37:35 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is reachable again Oct 3 14:47:46 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is unreachable Oct 3 14:47:46 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is reachable again Oct 3 14:47:55 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is unreachable Oct 3 14:48:40 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is reachable again Oct 3 14:48:48 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is unreachable Oct 3 14:49:22 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is reachable again Oct 3 14:49:30 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is unreachable Oct 3 14:49:30 Tower dhcpcd[1976]: br0: fe80::be24:11ff:fe4c:4bc3 is reachable agai

The parity errors don't tell you if a drive is bad or not, either parity or array, the errors are the number of differences from what the parity drive has calculated for the value of a particular sector. The errors can be from bad RAM, cables, random corruption or other hardware issues, or usually power disruption while writing. You'll need to interrogate your disks to make sure they're OK hardware wise, then re-sync parity - hoping whatever was in those 87 sectors was just out of sync or not important if the data was actually corrupted. This is a good reminder that parity is for redundancy, not data protection. It has no idea what's on your disks, and can only rebuild a disk based on whatever sector info it's got stored.

No, parity doesn't contain any file information, it only knows it was expecting a 1 and got a 0 from a particular sector on the disk, not what part of which file occupied that sector (it won't know which is the correct bit, the 1 on parity or the 0 from the array disk). If you've had an unclean shutdown, you can just run a correcting parity check to sync parity again, chance for corruption is lower. But, if it just suddenly started showing errors, you need to figure out why (bad disk, bad cable, bad RAM) before you continue.

Unless you have checksums, you have no way of knowing which files are 'broken'. If you're showing errors, it's because the zeros and ones on the disk don't match what the parity calculation is returning.

Looks like maybe the RAM going bad, run memtest to see if it finds any errors