July 15, 20241 yr Hello everyone, the fix common problems plugin just detected out of memory errors on my server. I'm not sure where to start digging... The only 'problem' that occurs quite regulary is that my jellyfin docker-container stops. In the log file of the unraid server the only 'out of memory' error entry I can find, seems to stop an ffmpeg process. The server has 32GB of Ram with about 8GB in use in 'normal' operation. Maybe you guys can give me a hint. Diagnostics are attached! Thank you very much in advance. Jaytie diagnostics-20240715-1910.zip
July 15, 20241 yr Community Expert 3 minutes ago, Jaytie said: In the log file of the unraid server the only 'out of memory' error entry I can find, seems to stop an ffmpeg process. Do you have jellyfin transcoding to RAM?
July 15, 20241 yr Author 50 minutes ago, Michael_P said: Do you have jellyfin transcoding to RAM? Thank you for your answer. I didn't configure it this way. I just tested it and it transcodes in the 'transcodes' folder unter appdata. I also limited the the amount of ram the container is allowed to use to 8GB.
July 15, 20241 yr Community Expert 3 hours ago, Jaytie said: Thank you for your answer. I didn't configure it this way. I just tested it and it transcodes in the 'transcodes' folder unter appdata. I also limited the the amount of ram the container is allowed to use to 8GB. Well whatever it was doing, it ran the host OOM back on the 2nd. FCP will continue to complain about it until you reboot the log is cleared. Since it doesn't look like a re-occurring issue, I'd just reboot and ignore it - and if it happens again, verify the config has the necessary work directories mapped to disk
July 16, 20241 yr Author Solution 10 hours ago, Michael_P said: Well whatever it was doing, it ran the host OOM back on the 2nd. I tried to reconstruct what happened on the 2nd of July and was successful. Back then I tested the encoding performance with multiple simultanious streams in Jellyfin. I could just recreate the OOM log entry. But what was running out of memory is the container which is limited to 8GB by me. Can you confirm that FCP also alerts me, when a container is running oom and not only the host system?
July 16, 20241 yr Community Expert 6 minutes ago, Jaytie said: But what was running out of memory is the container which is limited to 8GB by me The system ran OOM and the reaper killed the process which it believed was the culprit. Either your limit wasn't respected, or it's not working correctly. 9 minutes ago, Jaytie said: Can you confirm that FCP also alerts me, when a container is running oom and not only the host system? FCP just scans the log and alerts when it sees 'OOM' - and it will continue to alert until the log is cleared on reboot
July 16, 20241 yr Author 25 minutes ago, Michael_P said: The system ran OOM and the reaper killed the process which it believed was the culprit. Either your limit wasn't respected, or it's not working correctly. On my test a few minutes ago the same log entry appeared right at the moment, when the 8GB memory limit of the Docker-Container was reached. The host systems memory had around 50% usage at that time.
July 16, 20241 yr Community Expert 27 minutes ago, Jaytie said: On my test a few minutes ago the same log entry appeared right at the moment, when the 8GB memory limit of the Docker-Container was reached. The host systems memory had around 50% usage at that time. If Docker is enforcing the limit, it won't (or at least shouldn't) show in the system log. If the OOM reaper is invoked, it will
July 16, 20241 yr Author 32 minutes ago, Michael_P said: If Docker is enforcing the limit, it won't (or at least shouldn't) show in the system log. Strange, that seems to be the case. Thats the log entry when the docker memory limit is reached: Jul 16 11:01:14 Homebase kernel: Memory cgroup out of memory: Killed process 11551 (ffmpeg) total-vm:3747300kB, anon-rss:1277744kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:4276kB oom_score_adj:0 Jul 16 11:01:56 Homebase kernel: av:hevc:df0 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0 Jul 16 11:01:56 Homebase kernel: CPU: 7 PID: 16827 Comm: av:hevc:df0 Tainted: P O 6.1.74-Unraid #1 Jul 16 11:01:56 Homebase kernel: Hardware name: Gigabyte Technology Co., Ltd. B760M DS3H DDR4/B760M DS3H DDR4, BIOS F16 12/14/2023 Jul 16 11:01:56 Homebase kernel: Call Trace: Jul 16 11:01:56 Homebase kernel: <TASK> Jul 16 11:01:56 Homebase kernel: dump_stack_lvl+0x44/0x5c Jul 16 11:01:56 Homebase kernel: dump_header+0x4a/0x211 Jul 16 11:01:56 Homebase kernel: oom_kill_process+0x80/0x111 Jul 16 11:01:56 Homebase kernel: out_of_memory+0x3b3/0x3e5 Jul 16 11:01:56 Homebase kernel: mem_cgroup_out_of_memory+0x7c/0xb2 Jul 16 11:01:56 Homebase kernel: try_charge_memcg+0x44a/0x5ad Jul 16 11:01:56 Homebase kernel: charge_memcg+0x31/0x79 Jul 16 11:01:56 Homebase kernel: __mem_cgroup_charge+0x29/0x41 Jul 16 11:01:56 Homebase kernel: __handle_mm_fault+0x881/0xcf9 Jul 16 11:01:56 Homebase kernel: ? raw_spin_rq_unlock_irq+0x5/0x10 Jul 16 11:01:56 Homebase kernel: handle_mm_fault+0x13d/0x20f Jul 16 11:01:56 Homebase kernel: do_user_addr_fault+0x2c3/0x48d Jul 16 11:01:56 Homebase kernel: exc_page_fault+0xfb/0x11d Jul 16 11:01:56 Homebase kernel: asm_exc_page_fault+0x22/0x30 Jul 16 11:01:56 Homebase kernel: RIP: 0033:0x14c2df9d424a Jul 16 11:01:56 Homebase kernel: Code: c5 fe 7f 07 c5 fe 7f 47 20 c5 fe 7f 47 40 c5 fe 7f 47 60 c5 f8 77 c3 66 0f 1f 84 00 00 00 00 00 40 0f b6 c6 48 89 d1 48 89 fa <f3> aa 48 89 d0 c5 f8 77 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 Jul 16 11:01:56 Homebase kernel: RSP: 002b:000014c2ce9ad8b8 EFLAGS: 00010206 Jul 16 11:01:56 Homebase kernel: RAX: 0000000000000000 RBX: 000014c2b408c800 RCX: 0000000000188c40 Jul 16 11:01:56 Homebase kernel: RDX: 000014c2b9119040 RSI: 0000000000000000 RDI: 000014c2b957f000 Jul 16 11:01:56 Homebase kernel: RBP: 00000000005eec00 R08: 0000000000000007 R09: 000014c2b408c820 Jul 16 11:01:56 Homebase kernel: R10: a539d7c265516a54 R11: 0000000000000090 R12: 0000000000000000 Jul 16 11:01:56 Homebase kernel: R13: 000014c2d4123908 R14: 0000000000000006 R15: 0000000000000000 Jul 16 11:01:56 Homebase kernel: </TASK> Jul 16 11:01:56 Homebase kernel: memory: usage 8388608kB, limit 8388608kB, failcnt 1505365 Jul 16 11:01:56 Homebase kernel: swap: usage 0kB, limit 9007199254740988kB, failcnt 0 Jul 16 11:01:56 Homebase kernel: Memory cgroup stats for /docker/e1ec82a8a67d06905519c219e24704d5e72978f69734455d13218e181a2a75e9: Jul 16 11:01:56 Homebase kernel: anon 4071133184 Jul 16 11:01:56 Homebase kernel: file 4457185280 Jul 16 11:01:56 Homebase kernel: kernel 61616128 Jul 16 11:01:56 Homebase kernel: kernel_stack 6701056 Jul 16 11:01:56 Homebase kernel: pagetables 18108416 Jul 16 11:01:56 Homebase kernel: sec_pagetables 0 Jul 16 11:01:56 Homebase kernel: percpu 4648 Jul 16 11:01:56 Homebase kernel: sock 0 Jul 16 11:01:56 Homebase kernel: vmalloc 8192 Jul 16 11:01:56 Homebase kernel: shmem 4457156608 Jul 16 11:01:56 Homebase kernel: file_mapped 101199872 Jul 16 11:01:56 Homebase kernel: file_dirty 0 Jul 16 11:01:56 Homebase kernel: file_writeback 0 Jul 16 11:01:56 Homebase kernel: swapcached 0 Jul 16 11:01:56 Homebase kernel: anon_thp 27262976 Jul 16 11:01:56 Homebase kernel: file_thp 0 Jul 16 11:01:56 Homebase kernel: shmem_thp 25165824 Jul 16 11:01:56 Homebase kernel: inactive_anon 4152025088 Jul 16 11:01:56 Homebase kernel: active_anon 20316160 Jul 16 11:01:56 Homebase kernel: inactive_file 24576 Jul 16 11:01:56 Homebase kernel: active_file 0 Jul 16 11:01:56 Homebase kernel: unevictable 4355948544 Jul 16 11:01:56 Homebase kernel: slab_reclaimable 28472680 Jul 16 11:01:56 Homebase kernel: slab_unreclaimable 8133008 Jul 16 11:01:56 Homebase kernel: slab 36605688 Jul 16 11:01:56 Homebase kernel: workingset_refault_anon 0 Jul 16 11:01:56 Homebase kernel: workingset_refault_file 418588 Jul 16 11:01:56 Homebase kernel: workingset_activate_anon 0 Jul 16 11:01:56 Homebase kernel: workingset_activate_file 37906 Jul 16 11:01:56 Homebase kernel: workingset_restore_anon 0 Jul 16 11:01:56 Homebase kernel: workingset_restore_file 11170 Jul 16 11:01:56 Homebase kernel: workingset_nodereclaim 0 Jul 16 11:01:56 Homebase kernel: pgscan 2603050 Jul 16 11:01:56 Homebase kernel: pgsteal 2224813 Jul 16 11:01:56 Homebase kernel: pgscan_kswapd 547864 Jul 16 11:01:56 Homebase kernel: pgscan_direct 2055186 Jul 16 11:01:56 Homebase kernel: pgsteal_kswapd 518175 Jul 16 11:01:56 Homebase kernel: pgsteal_direct 1706638 Jul 16 11:01:56 Homebase kernel: pgfault 13461787 Jul 16 11:01:56 Homebase kernel: pgmajfault 9769 Jul 16 11:01:56 Homebase kernel: pgrefill 286105 Jul 16 11:01:56 Homebase kernel: pgactivate 198791 Jul 16 11:01:56 Homebase kernel: pgdeactivate 213058 Jul 16 11:01:56 Homebase kernel: pglazyfree 0 Jul 16 11:01:56 Homebase kernel: pglazyfreed 0 Jul 16 11:01:56 Homebase kernel: Tasks state (memory values in pages): Jul 16 11:01:56 Homebase kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Jul 16 11:01:56 Homebase kernel: [ 24834] 0 24834 9410567 131038 2027520 0 0 jellyfin Jul 16 11:01:56 Homebase kernel: [ 4530] 0 4530 842285 111564 2748416 0 0 ffmpeg Jul 16 11:01:56 Homebase kernel: [ 7112] 0 7112 1023772 292197 4243456 0 0 ffmpeg Jul 16 11:01:56 Homebase kernel: [ 10540] 0 10540 668568 133634 1998848 0 0 ffmpeg Jul 16 11:01:56 Homebase kernel: [ 14498] 0 14498 711444 109988 2699264 0 0 ffmpeg Jul 16 11:01:56 Homebase kernel: [ 14718] 0 14718 770254 204448 3362816 0 0 ffmpeg Jul 16 11:01:56 Homebase kernel: [ 16731] 0 16731 221706 42837 1118208 0 0 ffmpeg Jul 16 11:01:56 Homebase kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=e1ec82a8a67d06905519c219e24704d5e72978f69734455d13218e181a2a75e9,mems_allowed=0,oom_memcg=/docker/e1ec82a8a67d06905519c219e24704d5e72978f69734455d13218e181a2a75e9,task_memcg=/docker/e1ec82a8a67d06905519c219e24704d5e72978f69734455d13218e181a2a75e9,task=ffmpeg,pid=7112,uid=0 Jul 16 11:01:56 Homebase kernel: Memory cgroup out of memory: Killed process 7112 (ffmpeg) total-vm:4095088kB, anon-rss:1166848kB, file-rss:2284kB, shmem-rss:0kB, UID:0 pgtables:4144kB oom_score_adj:0 Jul 16 11:02:00 Homebase kernel: ffmpeg invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jul 16 11:02:00 Homebase kernel: CPU: 9 PID: 16883 Comm: ffmpeg Tainted: P O 6.1.74-Unraid #1 Jul 16 11:02:00 Homebase kernel: Hardware name: Gigabyte Technology Co., Ltd. B760M DS3H DDR4/B760M DS3H DDR4, BIOS F16 12/14/2023
July 16, 20241 yr Community Expert Still looks like the kernel killing it because it thinks it's about to crash due to running OOM In any case, you know what caused it - and if you want force it to use docker's killer, add the --oom-kill-disable option - but use with care as the kernel will start killing system processes if it goes OOM again
July 17, 20241 yr Author On 7/16/2024 at 1:16 PM, Michael_P said: Still looks like the kernel killing it because it thinks it's about to crash due to running OOM One addition: After restarting my jellyfin container I stumbled across this warning message, that I didn't recognize until now: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Is this normal behaviour if I use --memory=16G ?
July 17, 20241 yr Community Expert 2 hours ago, Jaytie said: One addition: After restarting my jellyfin container I stumbled across this warning message, that I didn't recognize until now: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. Is this normal behaviour if I use --memory=16G ? Safe to ignore
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.