Michael_P
-
Posts
669 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by Michael_P
-
-
25 minutes ago, BrandonG777 said:
Would you mind sharing what you looked at to determine that it's Frigate causing the OOM errors? I've dug through the logs but perhaps I'm overlooking it or just looking in the wrong place?Lots of Frigate and ffmpeg processes running, and then the reaper killing ffmpeg for using ~48GB of RAM - all signs that point to Frigate being the issue
Mar 20 15:21:27 z kernel: [ 24893] 0 24893 64344 10672 450560 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25262] 0 25262 64006 10635 446464 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25548] 0 25548 83510 13786 598016 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25610] 0 25610 64344 10672 446464 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25632] 0 25632 64006 10127 442368 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25642] 0 25642 64007 10636 442368 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 25653] 0 25653 31705 2258 155648 0 0 ffmpeg Mar 20 15:21:27 z kernel: [ 31960] 0 31960 1633221 73281 1323008 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31962] 0 31962 1635247 75319 1339392 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31963] 0 31963 1631413 71403 1306624 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31966] 0 31966 1635433 75509 1339392 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31968] 0 31968 1634820 74102 1335296 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31971] 0 31971 1635217 74896 1339392 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31973] 0 31973 1635260 75360 1343488 0 0 frigate.process Mar 20 15:21:27 z kernel: [ 31980] 0 31980 1194088 67613 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 31986] 0 31986 1194088 67710 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 31993] 0 31993 1194257 68060 1056768 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 32004] 0 32004 1194088 67412 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 32013] 0 32013 1218086 67613 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 32021] 0 32021 1221128 67406 1048576 0 0 frigate.capture Mar 20 15:21:27 z kernel: [ 32029] 0 32029 1194088 67951 1052672 0 0 frigate.capture Mar 20 15:21:27 z kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/docker/2e75c9f1047141b9b35bf3cc90663194f3be0ffc64a30482e4361cc762487692,task=ffmpeg,pid=10143,uid=0 Mar 20 15:21:27 z kernel: Out of memory: Killed process 10143 (ffmpeg) total-vm:48965636kB, anon-rss:43494820kB, file-rss:78340kB, shmem-rss:18804kB, UID:0 pgtables:85608kB oom_score_adj:0 -
5 minutes ago, BrandonG777 said:
Is that related to these extra parameters?
No idea, I don't run Frigate - maybe try its support thread
6 minutes ago, BrandonG777 said:However, I'm sure the SSD isnt as fast as system memory
Shouldn't matter for this use case, wear would be the concern - probably another question in their support thread
-
1
-
-
31 minutes ago, BrandonG777 said:
Frigate container,
Check your Frigate config, ffmpeg ran it OOM which implies it's transcoding to RAM. A few users have had the same issue over the past month or so
-
31 minutes ago, OriginalOne said:
Is it RAM or disk space that is running out?
And how can a docker container take down the host? Is there some method to avoid this?
Whatever Immich is doing is running the system out of RAM - left unrestricted, any container can use whatever system resources it has access to. You can limit the container's available RAM in the advanced settings, but that won't solve whatever the container is doing to run itself out of RAM.
As for why Immich is chugging down all that RAM, I can't help you there - you can ask in the container's help thread or try to get help on Immich's site
-
1 hour ago, Smokie said:
Can anyone even point me in the right direction?
Depends on how critical the data on the server is. If you don't have the capacity to back up the entire system, then figure out what data you can't afford to lose.
I use this script in conjunction with the appdata backup plugin
-
1
-
-
52 minutes ago, LugoCloud said:
How can I get new key for my USB or should I wait until I get a good quality USB
https://docs.unraid.net/unraid-os/manual/changing-the-flash-device/
-
16 minutes ago, LugoCloud said:
Tried a different USB now but is a very old (that worked)
DOn't have any other that I can use at this time
what USB model you suggest for me to get I can order Amazone
Thank youCant help you there, but I'd suggest a reputable brand from a reputable seller - there's a lot of shadiness in the world of flash drives. From what I remember, a 16/32GB USB 2.0 device is what's usually what's recommended but don't quote me on that
-
1
-
-
4 minutes ago, LugoCloud said:
Getting error 110
try another port, or retry creating the USB
-
16 hours ago, LugoCloud said:
Pardon my ignorance, but I don't understand
No worries
I'm not familiar with Frigate so I can't tell you where exactly to go, but in its settings or config file there should be storage folders and working folders, make sure that wherever ffmpeg is set to place its work files (which briefly looking at the docs appears it may be /tmp/cache) is either limited or mapped to a disk folder
-
13 hours ago, Miss_Sissy said:
"the standard" (without citations) for critical data.
It's well enough known to anyone that 'competently runs NAS and network', but you know that. Here's the citation for those that don't (straight from CISA):
https://www.cisa.gov/sites/default/files/publications/data_backup_options.pdf
-
1
-
-
21 minutes ago, LugoCloud said:
any suggestions will be greatly appreciated
Make sure it's not configured to transcode to a directory that isn't mapped to a drive
-
2 hours ago, LugoCloud said:
I'm using Frigate and Immich can one of this be the problem
Here's the container that was running the process, see if any part of the string matches either of the container IDs (advanced view on the docker tab) - Frigate is the most likely as there were quite a few processes running at the time (via coral/cuda-EvtHandlr), and on the 10th it was Frigate capture that invoked the reaper (each time ffmpeg ran away with the RAM)
Most likely Frigate: 42895499d47e566044b88aaad34d27527654e862a99663365824e302465beb4d Whatever you're using for GPU offload: 130189b80344b2fc2eb03b176497c5eba0aa27c18268a83afec3ba62771debc9 -
14 hours ago, Miss_Sissy said:
Protection against drive failures is something best accomplished through RAID, not backups. For example, on our primary NAS, up to two of the six discs can fail without the NAS going offline or losing any data. On our secondary NAS, which has less valuable data, one of the five drives can fail without data loss.
14 hours ago, Miss_Sissy said:Backups are for dealing with a catastrophic data loss, whether through flood, fire, malware, hackers, or some other act of God, Satan, or the Flying Spaghetti Monster.
unRaid or RAID(1,5,10,whatever) does absolutely nothing to protect data. RAID is for maintaining uptime - period.
It doesn't take a catastrophe to cause data loss, a simple press of the delete key is sufficient. That's why an actual, tested, backup strategy is critical for critical data. 3 copies, 2 on different media, and 1 offsite is the standard.
-
15 hours ago, OriginalOne said:
Any idea what caused this outage?
Check your immich config, looks like maybe it's the issue
QuoteMar 11 19:23:48 HomeServer kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/79f5b372d9378a6b1c32fae3726e83bcebc827e27c400d7074e684f5566b4bf8,task=gunicorn,pid=25424,uid=0
Mar 11 19:23:48 HomeServer kernel: Out of memory: Killed process 25424 (gunicorn) total-vm:45869304kB, anon-rss:14620764kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:36692kB oom_score_adj:0
-
12 hours ago, mihcox said:
OUT OF MEMORY ERRORS detected
Nothing reported in the logs, I'm not sure what FCP checks for but it may be flagging this line
Mar 6 15:52:31 PrecisionRig emhttpd: cmd: /usr/local/emhttp/plugins/user.scripts/showLog.php Out of Memory Check -
5 hours ago, ryphez said:
I'm running Frigate as a docker for alerting and have Blue Iris in Windows VM. I assume this would be more likely a Frigate issue as the VM would have the resources separately, right?
It looks like the Frigate container running an ffmpeg process - the VM's processes wouldn't be separately visible (they'd all be contained in the VM)
Mar 4 04:03:19 Andromeda kernel: [ 6380] 0 6380 293668 24347 483328 0 0 frigate.logger Mar 4 04:03:19 Andromeda kernel: [ 6430] 0 6430 395035 33985 782336 0 0 frigate.recordi Mar 4 04:03:19 Andromeda kernel: [ 6449] 0 6449 3564 1318 61440 0 0 python3 Mar 4 04:03:19 Andromeda kernel: [ 6451] 0 6451 398660 28977 581632 0 0 frigate.detecto Mar 4 04:03:19 Andromeda kernel: [ 6455] 0 6455 496288 27344 638976 0 0 frigate.output Mar 4 04:03:19 Andromeda kernel: [ 6459] 0 6459 31705 2264 159744 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: [ 6495] 0 6495 31705 1764 163840 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: [ 6510] 0 6510 31705 1763 163840 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: [ 6541] 0 6541 31705 1763 167936 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: [ 6561] 0 6561 31705 1752 155648 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: [ 7173] 0 7173 537950 39200 753664 0 0 frigate.process Mar 4 04:03:19 Andromeda kernel: [ 7174] 0 7174 537886 39269 753664 0 0 frigate.process Mar 4 04:03:19 Andromeda kernel: [ 7178] 0 7178 540778 42136 774144 0 0 frigate.process Mar 4 04:03:19 Andromeda kernel: [ 7179] 0 7179 542982 44344 794624 0 0 frigate.process Mar 4 04:03:19 Andromeda kernel: [ 7180] 0 7180 437282 33519 602112 0 0 frigate.capture Mar 4 04:03:19 Andromeda kernel: [ 7181] 0 7181 432781 33462 622592 0 0 frigate.capture Mar 4 04:03:19 Andromeda kernel: [ 7182] 0 7182 433060 35446 638976 0 0 frigate.capture Mar 4 04:03:19 Andromeda kernel: [ 7183] 0 7183 443746 35576 638976 0 0 frigate.capture Mar 4 04:03:19 Andromeda kernel: [ 7226] 0 7226 123878 11583 421888 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: [ 7272] 0 7272 33524 3110 192512 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: [ 7312] 0 7312 33272 4360 184320 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: [ 7321] 0 7321 32862 3458 196608 0 0 ffmpeg Mar 4 04:03:19 Andromeda kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/9c13b2c56bcbd99b2cd6b66f1f8009072a6cef3c76db3819560af169514f1c2b,task=ffmpeg,pid=25112,uid=0 Mar 4 04:03:19 Andromeda kernel: Out of memory: Killed process 25112 (ffmpeg) total-vm:35154720kB, anon-rss:34703828kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:68236kB oom_score_adj:05 hours ago, ryphez said:Thought wouldnt docker just not start if there was a shared port?
Dunno, just a guess based on all the errors in the log
Mar 4 04:21:02 Andromeda kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth622dd44: link becomes ready Mar 4 04:21:02 Andromeda kernel: br-fb0d039ca56f: port 28(veth622dd44) entered blocking state Mar 4 04:21:02 Andromeda kernel: br-fb0d039ca56f: port 28(veth622dd44) entered forwarding state Mar 4 04:21:04 Andromeda kernel: br-fb0d039ca56f: port 28(veth622dd44) entered disabled state Mar 4 04:21:04 Andromeda kernel: veth70b83b4: renamed from eth0 Mar 4 04:21:04 Andromeda kernel: br-fb0d039ca56f: port 28(veth622dd44) entered disabled state Mar 4 04:21:04 Andromeda kernel: device veth622dd44 left promiscuous mode Mar 4 04:21:04 Andromeda kernel: br-fb0d039ca56f: port 28(veth622dd44) entered disabled state Mar 4 04:22:04 Andromeda kernel: br-fb0d039ca56f: port 28(veth1e7adde) entered blocking state Mar 4 04:22:04 Andromeda kernel: br-fb0d039ca56f: port 28(veth1e7adde) entered disabled state Mar 4 04:22:04 Andromeda kernel: device veth1e7adde entered promiscuous mode Mar 4 04:22:04 Andromeda kernel: eth0: renamed from vethb1e8c0f -
1 hour ago, ryphez said:
had my unraid instance lock up last night and it seems to have self recovered but likely related to this out of memory error. Attached
Looks like your NVR isn't configured correctly, the ffmpeg process ran the host out of RAM
You also have a lot of network errors, check your docker containers to make sure you're not sharing ports or something
-
14 hours ago, JudasD said:
mprime was manually run earlier in that day. This was my attempt at trying to test my memory, etc. The OOM instance i am reporting was well after mprime was executed. I do not have any other scripts running at the same time as the "daily" schedule. If i move the daily schedule, the OOM issue follows the execution time. It is the fix common problems plugin that reports the issue because it is part of the daily script.
Do you have any other logs? The only instances in your attached diagnostics were mprime related.
-
On 2/29/2024 at 12:53 PM, JudasD said:
Thank you for any assistance you can provide
mprime is probably your culprit. As soon as it started it killed the system, your VM was just the first thing the reaper saw - then after that it killed the mprime process because it was using 132GB of RAM
Feb 28 21:50:13 Server kernel: process '/downloads/prime95/mprime' started with executable stack Feb 28 21:50:32 Server kernel: mprime invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0Feb 28 21:50:40 Server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=55a7482a9a2d896c5ce337890b0711b65664b2c735e2fb64aabe326efab7bb4e,mems_allowed=0-1,global_oom,task_memcg=/,task=mprime,pid=31853,uid=0 Feb 28 21:50:40 Server kernel: Out of memory: Killed process 31853 (mprime) total-vm:132210072kB, anon-rss:128343568kB, file-rss:0kB, shmem-rss:2352kB, UID:0 pgtables:251728kB oom_score_adj:0 Feb 28 21:50:44 Server kernel: oom_reaper: reaped process 31853 (mprime), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB -
4 hours ago, cloudyhome said:
Any help is appreciated. I would really like to fix this rather sooner than late
Looks like maybe your NVR isn't configured correctly, ffmpeg looks like it was transcoding to RAM and ran the system OOM.
Mar 2 01:16:28 Tower kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/d4f9dee74e2b127f4e102ce05fada9f34016069af611ac5c91f41d49f669731c,task=ffmpeg,pid=8558,uid=0 Mar 2 01:16:28 Tower kernel: Out of memory: Killed process 8558 (ffmpeg) total-vm:56654772kB, anon-rss:56268620kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:110316kB oom_score_adj:0 Mar 2 01:16:31 Tower kernel: oom_reaper: reaped process 8558 (ffmpeg), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB -
You can add either molex or SATA using punchdown/pass through connectors to the existing lines. Much better than trying to add too many drives to a single connector using a splitter. You can buy them for like a dollar on ebay.
-
6 minutes ago, Daver1 said:
I like this! not sure how to add it and those who my post are aimed at may not either..
In the advanced tab/custom nginx configuration..
-
Add this to NGINX in the advanced tab/custom nginx configuration to allow for large file transfers in nextcloud
proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_buffering off; proxy_max_temp_file_size 16384m; client_max_body_size 0;-
2
-
-
Single rail power is all well and good, but it's more important to limit the number of drives per connector. And don't, use molded style connector splitters. Use only quality individually wired and pinned connectors.
-
1
-
1
-
Out of Memory Errors
in General Support
Posted
No worries - first place to look is which process the reaper kills, it will (usually) kill the process using the most RAM at the time the system runs out. From that you can work backwards to see what started that process.
Not 100%, but most of the time it's enough to figure it out