Jump to content
Pducharme

[support] pducharme's Dockers support thread

624 posts in this topic Last Reply

Recommended Posts

18 hours ago, tobytl said:

I'm seeing the same behavior as @IamSpartacus, including the OOM reaping with any memory limit set (up to 8G).

 

Beyond that... Welcome to the madness that is linux memory management...

 

For instance, in my case the issues start well before any errors ever crop up in any logs. After running the UFV container for a few hours, the Plex container starts getting weird, transcodes will be fine, but "Direct Play" content will suddenly disappear from memory while streaming which throws an error on the client about the server or connection not being fast enough to stream. A few hours after that, I'll start seeing unresponsive containers, they are running and the process inside them is running (and working), just not responding to input (webui/commandline/etc). And then after about a day of running the UFV container, the dreaded "unable to fork" and "resource temporarily unavailable" messages start popping up all over the place. All of this while the host still thinks only about 50% of ram is being used. Fun.

 

Anyway, I'm currently testing with an enforced low JVM memory heap for the container to see if that helps. By default the unifi-video script sets a heap size of 1/4 of your system memory (8G in my case), but only 1548M on their nvr appliance. But, it also has an ENV override variable, so I added a variable to the docker template for "JVM_MX" set to a value of "1548M", and it worked in setting the heap size at least. So far a couple of hours in, and it's not using a ton of memory yet, and the early symptoms haven't started. Needs a much longer testing period though.

Oh wow, I thought I was going mad when my unraid server constantly stopped responding since the weekend.

I was wracking my brains, as one of my kids turned off my UPS which coincided at the same time the docker for Unifi Video updates that people are talking about in this thread. I also recently installed a new graphics card and set up emby/plex dockers to use it for transcoding (so I wasn't sure if something I did there caused it as well.)

I was getting the "unable to fork" messages constantly.

I posted in general support, didn't get a reply, then started trying different troubleshooting steps, etc.

I finally had time today to properly troubleshoot and found the thread in top for this docker was hammering my RAM. Couldn't agree more about your statement about linux memory management. Couldn't understand why unraid was saying I had x memory, but I kept getting errors hinting at something was hogging all the RAM.

Anyway, I can be at ease now I know the cause... I was about to give up.

Share this post


Link to post
19 hours ago, tobytl said:

Anyway, I'm currently testing with an enforced low JVM memory heap for the container to see if that helps. By default the unifi-video script sets a heap size of 1/4 of your system memory (8G in my case), but only 1548M on their nvr appliance. But, it also has an ENV override variable, so I added a variable to the docker template for "JVM_MX" set to a value of "1548M", and it worked in setting the heap size at least. So far a couple of hours in, and it's not using a ton of memory yet, and the early symptoms haven't started. Needs a much longer testing period though.

Please keep us posted on this -- I'm very curious for long-term results, if it's an easy fix adding one more variable the template can be updated easily for this! :)

Share this post


Link to post

New to Unraid and Unifi Video, have been up and running fine for 9 months. Yesterday my USB crashed and have to restore from backup, Since then Unifi Vid will not work properly, it launches fine but keeps using a 172.17.0.4 (Docker0) address and wont see all my cameras which are on my LAN at 192.168.13.0/24. I have removed the docker and reinstalled but it always wants to use the (Docker0) bridge. If I set the docker network type fm bridge to host I get both IP's. Im at a loss to where the real problem is. Any guidance would be appreciated.

 

Share this post


Link to post

After a full 24 hours, it's still stable. I'm not going to call it a "fix", as I don't think it addresses any root problems, but it's a pretty good mitigation. So far at least, completely stable. I'll leave it running on the main server.

 

Meanwhile, I'm going to play with getting a "wiredtiger" implementation working in the docker container on a test system to see if that affects anything. Don't think it will, and it's a nightmare to in-place upgrade the DB reliably, so I bow to the container dev in that endeavor. Good luck on whenever that becomes necessary (assuming UBNT don't figure out a way to do that nicely.... not holding my breath though).

Share this post


Link to post
20 hours ago, tobytl said:

After a full 24 hours, it's still stable. I'm not going to call it a "fix", as I don't think it addresses any root problems, but it's a pretty good mitigation. So far at least, completely stable. I'll leave it running on the main server.

 

Meanwhile, I'm going to play with getting a "wiredtiger" implementation working in the docker container on a test system to see if that affects anything. Don't think it will, and it's a nightmare to in-place upgrade the DB reliably, so I bow to the container dev in that endeavor. Good luck on whenever that becomes necessary (assuming UBNT don't figure out a way to do that nicely.... not holding my breath though).

As personally someone who has not been having issues -- I'm adding it anyway, a quick extra variable to help reduce any potential issues is well worth it. Set your same settings of 1548M to JVM_MX as shown

 

image.png.d6cb2845f4224e883f531b8c6ef9b2aa.png

Share this post


Link to post
On 2/18/2019 at 4:49 PM, Pducharme said:

i'm pretty tired of the Ubiquiti Networks new releases breaking everything.  It use to be straitforward, you install latest .deb, then it is updated.  Now, there is issues with MongoDB versions, etc.  Also, I only have 2 camera, one is not supported anymore (the original UVC-micro) and the new Micro G3.  I created this docker to have a quick setup for NVR, but the updates breaks it almost at every release.  I'm tempted to buy the CKGen2+ and just call it a day.    Is the latest Camera are also Wi-Fi ?  I can't get Ethernet everywhere I want a camera.   Never saw Protect either, is it good? 

I still can't get your Unraid Unifi Video docker to work right since the last update over a week ago.   Can you please just let us know if your going to fix it and release another update?  If your going to drop support for it ill just buy a CKGen2+.   All my cameras have been offline for a week.

I have been using this docker since the beginning and you have done great work it so far.

Whatever you decide is fine but please let everyone know.

Share this post


Link to post

Trying to get Docker Hub building and tagging issues sorted out w/ @Pducharme so you guys can at least run an older version easily.

But I have to admit, I have *no idea* what is going wrong. My own personal setup is working fine.

Anyone want to collaborate?

Share this post


Link to post
1 hour ago, suprjet44 said:

I still can't get your Unraid Unifi Video docker to work right since the last update over a week ago.   Can you please just let us know if your going to fix it and release another update?  If your going to drop support for it ill just buy a CKGen2+.   All my cameras have been offline for a week.

I have been using this docker since the beginning and you have done great work it so far.

Whatever you decide is fine but please let everyone know.

I was able to get mine working again and so were others by first delete the docker container entirely.

Second I deleted the appdata folder for unifivideo as well. 

 

Then I installed a fresh copy of the container and let it build its own new appdata folder and all the files within it. 

 

I then restored my video recordings and restarted the container. I also added the JVM_MX variable as noted above too and RAM usage has been small and the container has been stable for me. 

Share this post


Link to post
Posted (edited)

I am going to test the JVM_MX here to, its now using 27% of RAM (total ram is 64Gb!) with is to much for me, my desktop is running in this server too...

hope the JVM_MX wil work here. it just has 1 camera...

Edited by sjaak

Share this post


Link to post

I have been having a similar problem. I have been getting the OOM error message with memory usage of 73% ( 16 Gb of ram installed ). I found that unifi-video was eating up 26% of my memory. This had happened after i turned on the "enhanced motion algorithm". I turned that off on the four cameras i have and restarted the docker. My memory usage went down a bit but still is high. Memory usage went from 73% down to 51% and slowly came back up to 61% after the docker was running for a few mins. 

 

Just and interesting bit that i noticed and didn't see anybody else mention. i will be following the thread to see if anybody has a fix for the high mem usage.

Share this post


Link to post
4 hours ago, mediaman12 said:

I have been having a similar problem. I have been getting the OOM error message with memory usage of 73% ( 16 Gb of ram installed ). I found that unifi-video was eating up 26% of my memory. This had happened after i turned on the "enhanced motion algorithm". I turned that off on the four cameras i have and restarted the docker. My memory usage went down a bit but still is high. Memory usage went from 73% down to 51% and slowly came back up to 61% after the docker was running for a few mins. 

 

Just and interesting bit that i noticed and didn't see anybody else mention. i will be following the thread to see if anybody has a fix for the high mem usage.

The JVM_MX variable addresses the memory usage problem. The unifi-video launcher sets the java heap to 25% of your memory by default, which is a tad excessive. They set it to 1548M on their own hardware, so that's probably safe. Edit your UniFi-Video docker container, click the "Add another Path, Port, Variable, Label or Device" at the bottom, and set it up as in the image provided by @CorneliousJD below:

On 2/27/2019 at 1:26 PM, CorneliousJD said:

As personally someone who has not been having issues -- I'm adding it anyway, a quick extra variable to help reduce any potential issues is well worth it. Set your same settings of 1548M to JVM_MX as shown

 

image.png.d6cb2845f4224e883f531b8c6ef9b2aa.png

 

Share this post


Link to post

here too! after 3 days, unifi-video wont use more then 4% RAM, and it stable.

Share this post


Link to post
12 hours ago, suprjet44 said:

Unifi Video still keeps saying its Updating and thats it.

Any ideas?

Having this issue also! Anyone find a fix?

Share this post


Link to post

Been posted a few times already, those of us with working containers had to fully delete the container and appdata (take a backup first) and then let install a new one with fresh appdata, restore from backup, then restore recordings, and then set the JVM variable above too so it doesn't consume RAM through the roof. 

 

Been stable for me since doing both those things. 

Share this post


Link to post

Thanks for the docker but it's really frustrating me lately...

Anytime the docker has to be restarted for whatever reason (restart server, power outage etc.) UniFi-Video docker stops working. The only error in the unraid log is always the same saying it can't find system.properties and resorts to a default file I suppose. The webGUI pulls up the login window but the credentials won't work. The only solution is to reinstall and restore from backup. I only use the default docker settings. I have a another Windows PC I use for home automation and I'm wondering if it would just be easier to install UniFi-Video on that, but I would like to keep the docker if I can just get it to work right.

 

 

(unifi-video) WARNING!!!! system.properties cannot be found..restoring from : /usr/lib/unifi-video/etc/system.propetties

 

Docker Mappings:

/var/lib/unifi-video = /mnt/cache/appdata/unifi-video/

/var/lib/unifi-video/videos = /mnt/user/UnfiVideoShareForStorage

 

Share this post


Link to post

@bluesky509, it sounds like you're not passing in an outside folder for the config to get saved in and so every time you start up, it looks like a fresh install. But the paths you've shown clearly indicate you are, unless you've passed them in that order which would be the *opposite* of correct. You need to do some troubleshooting.

The first thing I'd do is look in that outside folder while it is running, is the file there? If it isn't, you've found your issue and need to figure out what is wrong w/ your path mapping. You could `docker exec` into your container and see where the file *really* is and what you've mounted into it.

Share this post


Link to post
17 hours ago, fryfrog said:

@bluesky509, it sounds like you're not passing in an outside folder for the config to get saved in and so every time you start up, it looks like a fresh install. But the paths you've shown clearly indicate you are, unless you've passed them in that order which would be the *opposite* of correct. You need to do some troubleshooting.

The first thing I'd do is look in that outside folder while it is running, is the file there? If it isn't, you've found your issue and need to figure out what is wrong w/ your path mapping. You could `docker exec` into your container and see where the file *really* is and what you've mounted into it.

Not sure what you mean by "passing them in that order".

Would me not having a cache disk (it's in future plans) affect anything? I don't normally use /mnt/cache for that reason. All files are currently where they are supposed to be. The system.properties file does exist inside /mnt/cache/app... 

In that past I've tried using custom path but the issue issue would occur.

 

I "docker exec" into the docker and /mnt is empty. /var/cache/unifi-video has hls folder (empty) and exports folder (2 videos inside). I didn't see any other files elsewhere.

 

Share this post


Link to post
-v /data/unifi-video/data:/var/lib/unifi-video \
-v /data/unifi-video/videos:/var/lib/unifi-video/videos \

^ Docker paths are passed in like that, w/ the *outside* path first and the *inside* path second. Your post doesn't actually show the exact way you're passing them in, just path = path. And they're *reversed* from how they should be if you were reading them right from the `-v` argument.

It looks in a specific place for that file and clearly isn't finding it. Because the insides of a Docker container get nuked every time they're updated (or even restarted if you use `--rm` like I do), you lose your config every time. I think you've got your setup wrong.

Can you show your run command? When you `docker exec` into it, you should be looking at `/var/lib/unifi-video` and `/var/lib/unifi-video/videos`.

Share this post


Link to post
2 hours ago, bluesky509 said:

Would me not having a cache disk (it's in future plans) affect anything?

Yes, because without a cache disk, the path /mnt/cache is in RAM, and will be lost on reboot.

2 hours ago, bluesky509 said:

The system.properties file does exist inside /mnt/cache/app...

Until the system is rebooted, then /mnt/cache is gone along with all its contents until something writes to that path.

Share this post


Link to post
3 hours ago, fryfrog said:

-v /data/unifi-video/data:/var/lib/unifi-video \
-v /data/unifi-video/videos:/var/lib/unifi-video/videos \

 ^ Docker paths are passed in like that, w/ the *outside* path first and the *inside* path second. Your post doesn't actually show the exact way you're passing them in, just path = path. And they're *reversed* from how they should be if you were reading them right from the `-v` argument.

It looks in a specific place for that file and clearly isn't finding it. Because the insides of a Docker container get nuked every time they're updated (or even restarted if you use `--rm` like I do), you lose your config every time. I think you've got your setup wrong.

Can you show your run command? When you `docker exec` into it, you should be looking at `/var/lib/unifi-video` and `/var/lib/unifi-video/videos`.

Sorry, i'm still new to this... How do I execute a "run command" exactly?

 

Share this post


Link to post
1 hour ago, jonathanm said:

Yes, because without a cache disk, the path /mnt/cache is in RAM, and will be lost on reboot.

Until the system is rebooted, then /mnt/cache is gone along with all its contents until something writes to that path.

i figured it was something along those lines. Now just need to fix.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.