BrandonG777 Posted March 21 Share Posted March 21 (edited) Getting some out of memory errors after a handful of recent changes to one of my systems. Changes I can think of are switching to a new PC, setting up Frigate container, migrated from Plex to Emby, and upgrading to 6.12.8 unRAID. Server runs fine for days/weeks and then eventually hard locks. I originally thought it was the addition of a defective USB Coral device causing this but removing that I'm still having this issue. z-diagnostics-20240321-0518.zip Edited March 21 by BrandonG777 Quote Link to comment
Solution Michael_P Posted March 21 Solution Share Posted March 21 31 minutes ago, BrandonG777 said: Frigate container, Check your Frigate config, ffmpeg ran it OOM which implies it's transcoding to RAM. A few users have had the same issue over the past month or so Quote Link to comment
BrandonG777 Posted March 21 Author Share Posted March 21 5 minutes ago, Michael_P said: Check your Frigate config, ffmpeg ran it OOM which implies it's transcoding to RAM. A few users have had the same issue over the past month or so Is that related to these extra parameters? --shm-size=256mb --mount type=tmpfs,target=/tmp/cache,tmpfs-size=100000000 I do have a dedicated temp SSD for stuff like this. I'm thinking I should just map that to /tmp/cache? However, I'm sure the SSD isnt as fast as system memory and I have 64GB (which currently reports as 46% utilized). For performance reasons, should I try to adjust this shm to work? Quote Link to comment
Michael_P Posted March 21 Share Posted March 21 5 minutes ago, BrandonG777 said: Is that related to these extra parameters? No idea, I don't run Frigate - maybe try its support thread 6 minutes ago, BrandonG777 said: However, I'm sure the SSD isnt as fast as system memory Shouldn't matter for this use case, wear would be the concern - probably another question in their support thread 1 Quote Link to comment
BrandonG777 Posted March 22 Author Share Posted March 22 23 hours ago, Michael_P said: Check your Frigate config, ffmpeg ran it OOM which implies it's transcoding to RAM. A few users have had the same issue over the past month or so Would you mind sharing what you looked at to determine that it's Frigate causing the OOM errors? I've dug through the logs but perhaps I'm overlooking it or just looking in the wrong place? Quote Link to comment
Michael_P Posted March 22 Share Posted March 22 25 minutes ago, BrandonG777 said: Would you mind sharing what you looked at to determine that it's Frigate causing the OOM errors? I've dug through the logs but perhaps I'm overlooking it or just looking in the wrong place? Lots of Frigate and ffmpeg processes running, and then the reaper killing ffmpeg for using ~48GB of RAM - all signs that point to Frigate being the issue Mar 20 15:21:27 z kernel: [ 24893] 0 24893 64344 10672 450560 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25262] 0 25262 64006 10635 446464 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25548] 0 25548 83510 13786 598016 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25610] 0 25610 64344 10672 446464 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25632] 0 25632 64006 10127 442368 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25642] 0 25642 64007 10636 442368 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25653] 0 25653 31705 2258 155648 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 31960] 0 31960 1633221 73281 1323008 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31962] 0 31962 1635247 75319 1339392 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31963] 0 31963 1631413 71403 1306624 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31966] 0 31966 1635433 75509 1339392 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31968] 0 31968 1634820 74102 1335296 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31971] 0 31971 1635217 74896 1339392 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31973] 0 31973 1635260 75360 1343488 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31980] 0 31980 1194088 67613 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 31986] 0 31986 1194088 67710 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 31993] 0 31993 1194257 68060 1056768 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 32004] 0 32004 1194088 67412 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 32013] 0 32013 1218086 67613 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 32021] 0 32021 1221128 67406 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 32029] 0 32029 1194088 67951 1052672 0 0 frigate.capture Mar 20 15:21:27 z kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/docker/2e75c9f1047141b9b35bf3cc90663194f3be0ffc64a30482e4361cc762487692,task=ffmpeg,pid=10143,uid=0 Mar 20 15:21:27 z kernel: Out of memory: Killed process 10143 (ffmpeg) total-vm:48965636kB, anon-rss:43494820kB, file-rss:78340kB, shmem-rss:18804kB, UID:0 pgtables:85608kB oom_score_adj:0 Quote Link to comment
BrandonG777 Posted March 22 Author Share Posted March 22 Lots of Frigate and ffmpeg processes running, and then the reaper killing ffmpeg for using ~48GB of RAM - all signs that point to Frigate being the issue Mar 20 15:21:27 z kernel: [ 24893] 0 24893 64344 10672 450560 0 0 ffmpegMar 20 15:21:27 z kernel: [ 25262] 0 25262 64006 10635 446464 0 0 ffmpegMar 20 15:21:27 z kernel: [ 25548] 0 25548 83510 13786 598016 0 0 ffmpegMar 20 15:21:27 z kernel: [ 25610] 0 25610 64344 10672 446464 0 0 ffmpegMar 20 15:21:27 z kernel: [ 25632] 0 25632 64006 10127 442368 0 0 ffmpegMar 20 15:21:27 z kernel: [ 25642] 0 25642 64007 10636 442368 0 0 ffmpegMar 20 15:21:27 z kernel: [ 25653] 0 25653 31705 2258 155648 0 0 ffmpegMar 20 15:21:27 z kernel: [ 31960] 0 31960 1633221 73281 1323008 0 0 frigate.processMar 20 15:21:27 z kernel: [ 31962] 0 31962 1635247 75319 1339392 0 0 frigate.processMar 20 15:21:27 z kernel: [ 31963] 0 31963 1631413 71403 1306624 0 0 frigate.processMar 20 15:21:27 z kernel: [ 31966] 0 31966 1635433 75509 1339392 0 0 frigate.processMar 20 15:21:27 z kernel: [ 31968] 0 31968 1634820 74102 1335296 0 0 frigate.processMar 20 15:21:27 z kernel: [ 31971] 0 31971 1635217 74896 1339392 0 0 frigate.processMar 20 15:21:27 z kernel: [ 31973] 0 31973 1635260 75360 1343488 0 0 frigate.processMar 20 15:21:27 z kernel: [ 31980] 0 31980 1194088 67613 1048576 0 0 frigate.captureMar 20 15:21:27 z kernel: [ 31986] 0 31986 1194088 67710 1048576 0 0 frigate.captureMar 20 15:21:27 z kernel: [ 31993] 0 31993 1194257 68060 1056768 0 0 frigate.captureMar 20 15:21:27 z kernel: [ 32004] 0 32004 1194088 67412 1048576 0 0 frigate.captureMar 20 15:21:27 z kernel: [ 32013] 0 32013 1218086 67613 1048576 0 0 frigate.captureMar 20 15:21:27 z kernel: [ 32021] 0 32021 1221128 67406 1048576 0 0 frigate.captureMar 20 15:21:27 z kernel: [ 32029] 0 32029 1194088 67951 1052672 0 0 frigate.captureMar 20 15:21:27 z kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/docker/2e75c9f1047141b9b35bf3cc90663194f3be0ffc64a30482e4361cc762487692,task=ffmpeg,pid=10143,uid=0Mar 20 15:21:27 z kernel: Out of memory: Killed process 10143 (ffmpeg) total-vm:48965636kB, anon-rss:43494820kB, file-rss:78340kB, shmem-rss:18804kB, UID:0 pgtables:85608kB oom_score_adj:0 Thank you but I wasn’t questioning your diagnosis but rather what logs/files you looked at to determine that so that I might possibly be able to diagnose these issues myself and stop relying on others for things I feel like I should know how to do. Sent from my iPhone using Tapatalk Quote Link to comment
Michael_P Posted March 22 Share Posted March 22 5 minutes ago, BrandonG777 said: Thank you but I wasn’t questioning your diagnosis but rather what logs/files you looked at to determine that so that I might possibly be able to diagnose these issues myself and stop relying on others for things I feel like I should know how to do. No worries - first place to look is which process the reaper kills, it will (usually) kill the process using the most RAM at the time the system runs out. From that you can work backwards to see what started that process. Not 100%, but most of the time it's enough to figure it out Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.