DrSpaldo

Hi @MrK, yes, I do think that was either the primary cause or a big part of it. Although I have no concrete proof. I know originally when I was deep into the issue, I found out that the --memory line doesn't always work with Unraid. I did a quick search earlier and didn't find the exact post where I remember seeing it though... Here are a few to look through that may help - Unraid OOM Issues · blakeblackshear/frigate Edit: check this post to see if you are using the fuse file system for your Unraid Frigate container - FUSE post - Frigate Github

No worries, thanks @bucketphobia@itimpi

@bucketphobiaI've turned on the permit exclusing shares, however, I don't see how to turn on the exclusive access part anywhere. Although, I do remember that I did change the default for my dockers to point directly to the drive and not use fuse... /mnt/cachetwo/appdata/opera rather than /mnt/user/appdata/opera

Thanks @bucketphobia, I will also turn that on. Fortunately mine has died down heaps since I made the above other changes, but who knows what the actual root cause is. Seems to be so many people in similar (but not exactly) the same situation

I would love to say that I figured out exactly what was causing the issue, but, I didn't. I made so many changes and in the end I am not sure if it was one of them or the combination of them. I am not getting the same CPU max out or freeze/crashes that I was before. It is back to normal. Here are some of the things that I did: - Daily script to delete docker locks - increase tmpfs run size script on boot - create 4G tmpfs plex ram scratch script on boot - use the swapfile plugin - change the VM dirty background ratio - Replace the MB & CPU - Replace the SATA card - Pinned CPU cores / docker isolation - Move some of the things running on Unraid to my Proxmox server; this included the Frigate container (my CCTV) & the Windows VM I would say that I noticed some differences from the top down, but, the main one was moving Frigate & the VM to the Proxmox server. I am sorry this isn't a good answer, but I just don't know. When you start searching for similar issues, many many people are having them as well, just in slightly different ways. The Unraid staff are not really helping (only sometimes, then they just give up). Fortunately I had JorgeB trying to help in this instance.

For those having issues with MagicMirror after the most recent update (constant reboots, essentially not starting up) It looks like for me it was the plugin/module MMM-MyCalendar breaks it. I have disabled it for now and there has been a issue created below. Let's hope it is looked at soon... Release 2.25.0 of MagicMirror breaks the module · Issue #40 · jclarke0000/MMM-MyCalendar (github.com)

Hi @Giorgioare you able to expand on how you did this? I am thinking of doing the same...

Hi, I just left it at the default (60)

Is there a reason the swap file would fill up if I still had heaps of ram free? Today I had about 30 gigs RAM free and the swap file was sitting at around 7 gig (91%)

I've switched that (Frigate container) over to my Proxmox server. So I will see how it goes...

Thanks for the reply @yayitazale. I will try some of these changes on my system and see if it makes a difference.

Add me to the list. I have been having issues on and off since June. It started happening after updating Unraid from 6.9.2. I have since updated a few times with no great change, I am now running 6.12.3 I can't remember all the changes that I have made to the Unraid system itself along with the Frigate container. But it seems to have limited it recently to one crash/system freeze per week. Here is part of my journey

Ah ok, interesting. My Ubuntu VM is actually still running without issue. So maybe not that!

@Mbeco although it hasn't 100% fixed the issue (and I changed so many things in the process, they could also take part) but one of the top 3 potential issues that I feel it may be was the VM's I was running. I am running a Windows 10 VM & MacOS VM. I cannot tell you which was the most likely culprit but I lean towards Windows. For me, what I did was move both of the VM's over to another Proxmox server I had. Obviously this is not a solution, but, like you, I didn't have the time for weeks of downtime like people above have suggested. Because to truly work out what is causing the issue, you would need massive amounts of server downtime as well as your own time sitting there watching. If you can, I suggest turning off one of the VM's, or better still if you have spare hardware, move them over to a Proxmox instance and just see if that helps your situation. If it doesn't, then I can make suggestions on what I did for my docker troubleshooting & changes along the way... PS. I see you are using PhotoPrism. You should give a very nice project Immich a go. Although in early development, I have been very impressed so far. There is an Unraid community app as well.

Thanks for the reply & info @primeval_god

Absolutely. I have had no issues with Unraid for sometime, it started happening around the time I updated from one of the older versions but I can’t 100% confirm that. I wish I remembered everything I changed, because it was so much. I’m much more stable now, just maxing out once a week during my appdata/CA backup day. tomorrow if I get some time, I will go through all the things I changed to see if that helps you. How much RAM in total do you have an how many dockers / VM’s? How would they look for their system utilisation compared to your reasources?

What swap file size would you recommend if I had 48 gigs total ram? Does the swap file size count as a total across the entire system or is the number inputted per docker?

@MbecoI feel like you are going on the same crazy ride has I have been

I've noticed that during my weekly docker/appdata backup it looks like Unraid cannot stop the Frigate container and/or gives OOM errors. Any ideas of what may lead to this or what I can do to prevent this from happening? Jul 30 11:44:01 SpaldoUnraid kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=f6bd84541f24695692d01c50b3d4f9190ec11f453d2e440fc68c2e13a3192478,mems_allowed=0,global_oom,task_memcg=/docker/f6bd84541f24695692d01c50b3d4f9190ec11f453d2e440fc68c2e13a3192478,task=frigate.capture,pid=378,uid=0 Jul 30 11:44:01 SpaldoUnraid kernel: Out of memory: Killed process 378 (frigate.capture) total-vm:33863704kB, anon-rss:31910812kB, file-rss:0kB, shmem-rss:504kB, UID:0 pgtables:62780kB oom_score_adj:0 Jul 30 11:44:02 SpaldoUnraid kernel: oom_reaper: reaped process 378 (frigate.capture), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB The above is some extracts from the syslog around the time that the machine was coming back to life after being frozen for a few hours. Of note, I could not connect to the Unraid machine via webgui or SSH. However, the MPV stream I have running from Frigate continued to work, regardless of issues I was having with connecting to Unraid itself...

Its my CCTV NVR, so it has to be on

In this instance, I am not sure. I just couldn't connect to the server in any way for 4 hours or so. I could not connect to see what the server was doing

@JorgeBI've noticed that it has not been happening as often, almost not at random like before. Except it has happened a couple times when I am doing my weekly docker/appdata backup. This week I have switched from the old CA version to the new one, unfortunately it happened again this week. Looking at the logs it seems to be another OOM error with my Frigate docker. See attached logs. Any ideas what I can do to prevent this from happening? Out of interest though, I usually run a camera stream from Frigate to my desktop, the entire time the stream did not stop while it should have been doing the backup, which means the container didn't stop? I couldn't get into the webui or SSH in during this time, only ping. spaldounraid-diagnostics-20230730-1453.zip

Thanks for the post @JorgeB. I've installed the swap file plugin. What would you recommend for monitoring the ram usage as well as the fragmented RAM?

Thanks for the post @samsausages. I will setup a user script for solution 1 every 24 hours and on first array start for solution 3. As for solution 2, I will just have to see if I can work out what dockers may be causing issues, I have a few..

So this morning I had the issue again. CPU at max and I couldn't do anything. However, I decided to just leave the server running as I didn't need it today. Between 6.30am and 2.45pm I could not use the server at all, couldn't load the webui or even connect via SSH. After that though, all back to 'normal'. I am just not sure how I can work out exactly what was happening at that time and what I can do to stop it from happening? spaldounraid-diagnostics-20230709-1609.zip