1 hour ago, bucketphobia said:

Hey @DrSpaldo, I was having a very similar issue, but slightly different - the cause of mine was a dockerd log process.

 

The solution to mine was to ensure that "Permit exclusive shares" is on, and that "Exclusive Access" is indeed active on my cache pool.

 

Hope this helps.

Thanks @bucketphobia, I will also turn that on. Fortunately mine has died down heaps since I made the above other changes, but who knows what the actual root cause is. Seems to be so many people in similar (but not exactly) the same situation

@bucketphobiaI've turned on the permit exclusing shares, however, I don't see how to turn on the exclusive access part anywhere. 

 

Although, I do remember that I did change the default for my dockers to point directly to the drive and not use fuse... /mnt/cachetwo/appdata/opera rather than /mnt/user/appdata/opera

1 hour ago, DrSpaldo said:

@bucketphobiaI've turned on the permit exclusing shares, however, I don't see how to turn on the exclusive access part anywhere. 

 

Although, I do remember that I did change the default for my dockers to point directly to the drive and not use fuse... /mnt/cachetwo/appdata/opera rather than /mnt/user/appdata/opera

Once you have enabled the global option it should be automatic if a share is set up to have all its files to be on a single pool/drive and that is how they are actually currently placed   If it is not enabling then you have probably not met that criteria.

10 hours ago, DrSpaldo said:

@bucketphobiaI've turned on the permit exclusing shares, however, I don't see how to turn on the exclusive access part anywhere. 

 

Although, I do remember that I did change the default for my dockers to point directly to the drive and not use fuse... /mnt/cachetwo/appdata/opera rather than /mnt/user/appdata/opera

 

If it says no - you likely need to (1): disable secondary storage; (2): make sure that nothing on the share is on other drives

No worries, thanks @bucketphobia@itimpi

Do you think the issues have gone after removing frigate from the server?

When playing about with frigate (2 cams go2rtc streams and HW encoding enabled intel with coral). Only seems to happen when I am messing about with settings trying to get go2rtc in to home assistant etc.

Nov  5 13:00:58 Tower kernel: ffmpeg invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov  5 13:00:58 Tower kernel: CPU: 3 PID: 7505 Comm: ffmpeg Tainted: P           O       6.1.49-Unraid #1

 

It says invoking oom-killer but I guess it doesn't work?

 

It makes the server unreachable. I have to power cycle it.

 

I have added the "--memory=3G" line to the frigate docker and installed the swap file plugin.

4 hours ago, MrK said:

Do you think the issues have gone after removing frigate from the server?

When playing about with frigate (2 cams go2rtc streams and HW encoding enabled intel with coral). Only seems to happen when I am messing about with settings trying to get go2rtc in to home assistant etc.

Nov  5 13:00:58 Tower kernel: ffmpeg invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
Nov  5 13:00:58 Tower kernel: CPU: 3 PID: 7505 Comm: ffmpeg Tainted: P           O       6.1.49-Unraid #1

 

It says invoking oom-killer but I guess it doesn't work?

 

It makes the server unreachable. I have to power cycle it.

 

I have added the "--memory=3G" line to the frigate docker and installed the swap file plugin.

 

Hi @MrK, yes, I do think that was either the primary cause or a big part of it. Although I have no concrete proof. 

 

I know originally when I was deep into the issue, I found out that the --memory line doesn't always work with Unraid. I did a quick search earlier and didn't find the exact post where I remember seeing it though... Here are a few to look through that may help - Unraid OOM Issues · blakeblackshear/frigate

 

Edit: check this post to see if you are using the fuse file system for your Unraid Frigate container - FUSE post - Frigate Github

Edited November 6, 20232 yr by DrSpaldo

For those that also may come and find this thread. I have just upgraded a friends server from 6.9.2 to 6.12.10 and now having crashing issues. It is lasting max of 1 day then the server goes totally unresponsive, even on SSH. It does ping back though. 

 

At this stage, I am trying all the suggestions above, however, I cannot take Frigate off this server. So it will be interesting to find out if there is a way to keep Unraid running with Frigate as well. I try to keep updating this thread with the outcome/progress

Apologies for the necro... have you ever figured out what is causing this?

My server started doing this as well.

 

It starts about 11 days after a reboot.

While this was happening, I managed to turn off the docker service as well as the sole VM I had (Home Assistant), but the "lsof" command spam continued.

The server never crashes, but becomes sluggish, uses a lot of power (CPU almost maxed out) and continues like that indefinitely. HDD usage is also almost maxed out.

 

Rebooting the server fixes it... until it starts happening again.

 

Currently using Unraid 7.0.0.

 

34 minutes ago, war4peace said:

Apologies for the necro... have you ever figured out what is causing this?

My server started doing this as well.

 

It starts about 11 days after a reboot.

While this was happening, I managed to turn off the docker service as well as the sole VM I had (Home Assistant), but the "lsof" command spam continued.

The server never crashes, but becomes sluggish, uses a lot of power (CPU almost maxed out) and continues like that indefinitely. HDD usage is also almost maxed out.

 

Rebooting the server fixes it... until it starts happening again.

 

Currently using Unraid 7.0.0.

 

Simple answer would be “I think so”. But with Unraid, who knows. The community support is pretty good but the actual devs don’t really do much here. If you search similar topics, there are so many and all you hear is crickets.

 

My suggestion would be taking as much as possible off the fuse file system. So for example, in my Frigate docker, instead of having the storage as /mnt/user/share I now have /mnt/disk4/media/frigate

 

The limitation is that then the storage has to be on that drive. 
 

I changed so many things, I can’t totally be sure if that’s what fixed it, but hence the “I think so” response. I’d try that first….

Thank you, it took this much time for my issue to resurface, and it turns out it was not the lsof thing all along.

Or... maybe it also was, but in combination with other things.

 

I found this topic referring exactly to my problem, but of course it sat unanswered.

 

Sigh...