[support] pducharme's Dockers support thread

[stormshaker]

I hear your pain. I installed my first Protect setup last month, and it’s been good. I like the UI and the iOS app. I like that it has a battery for graceful shutdowns and that the Unifi Network controller is built-in. It’s fast to connect to from remote. Downside is no storage redundancy, and I couldn’t get it to power over USB (tip, just buy the correct Ubiquiti POE injector, it’s cheaper than a QC USB charger, anyway). It’s not that expensive, considering maintenance should be much easier (in fact, it can keep itself completely to date automatically, unlike the Network controller).

 

UBNT have a single wireless camera model, but it’s for indoor use only.

[dapice]

hi, i just updated this docker. after completion, ive had this ecreen when i open the container. it had been an hour and no progress. id this normal? or is this a bad update?

 

[g00fy]

20 minutes ago, dapice said:

hi, i just updated this docker. after completion, ive had this ecreen when i open the container. it had been an hour and no progress. id this normal? or is this a bad update?

 

Having the same problem, it's been like this all day, tried restarting the docker a few times but still no joy...

[CorneliousJD]

Read the posts above. We've all had the same issue. I got around it by taking my latest backup from the apodata of this container. It has auto backups, then deleting appdata and container. Then restoring from backup when the new container is in initial setup.

[spyd4r]

i've tried a clean install after ditching my appdata on beta, testing and latest branches.. and none will start up from scratch on a fresh install.

[IamSpartacus]

So based on the latest comments on this thread I'm going to assume it's NOT safe to update my Unifi Video container.

[RyanDownyJr]

Unsure if it will help much, but my install is also hosed as expecting. Error log being generate by Log under Dockers Menu is:

 

Feb 19 09:55:49 Tower nginx: 2019/02/19 09:55:49 [error] 7603#7603: *3148358 connect() to unix:/var/tmp/UniFi-Video.sock failed (111: Connection refused) while connecting to upstream, client: 172.27.232.2, server: , request: "GET /dockerterminal/UniFi-Video/ws HTTP/1.1", upstream: "http://unix:/var/tmp/UniFi-Video.sock:/ws", host: "192.168.101.3:1447"

 

I'm getting blasted with this every 3 to 5 seconds when the docker is running. Hope this helps!

 

Downy

[suprjet44]

Im getting the same loading screen forever also.  

Is there going to be another update released that fixes this?  or should i try to follow what CorneliousJD said?

15 hours ago, CorneliousJD said:

Read the posts above. We've all had the same issue. I got around it by taking my latest backup from the apodata of this container. It has auto backups, then deleting appdata and container. Then restoring from backup when the new container is in initial setup.

 

[spyd4r]

i was able to get a beta tag operational by editing /usr/lib/unifi-video/conf/mongodv3.6+.conf and changing it to engine: wiredTiger. i did this during a fresh install and then restored a backup.. it seems to have worked.. although in settings it still thinks its a 3.9.12 controller.

[suprjet44]

I was able to revert back to 3.9.11 but not i cant get to any of my cameras and it wont adopt them.  This has been such a pain.

Do you think there will be an update that will fix all of this?

[Andiroo2]

On 2/18/2019 at 9:35 PM, CorneliousJD said:

Read the posts above. We've all had the same issue. I got around it by taking my latest backup from the apodata of this container. It has auto backups, then deleting appdata and container. Then restoring from backup when the new container is in initial setup.

Confirming that this worked for me as well.  After deleting and reinstalling the docker, it took a while to apply permissions to my recordings, and then the old recordings don't show in the new console, but i'm back online again at least.  I'll look around to see if I can get the old recordings to appear again, but I'm guessing that they can just be deleted from storage.

[CorneliousJD]

9 minutes ago, Andiroo2 said:

Confirming that this worked for me as well.  After deleting and reinstalling the docker, it took a while to apply permissions to my recordings, and then the old recordings don't show in the new console, but i'm back online again at least.  I'll look around to see if I can get the old recordings to appear again, but I'm guessing that they can just be deleted from storage.

Someone else above told me how to do this, so they should get credit, not me! But... as long as you pointed your recording directory to the same as before, you just need to scan for them. -- I was able to get all my recordings back this way!

 

 

 

[Andiroo2]

3 hours ago, CorneliousJD said:

Someone else above told me how to do this, so they should get credit, not me! But... as long as you pointed your recording directory to the same as before, you just need to scan for them. -- I was able to get all my recordings back this way!

Worked for me too, but I made sure to check off "Restore Recordings" first.  When I clicked "Analyze" without it checked, it warned me that it may delete missing recordings it finds.

[Andiroo2]

3 hours ago, Andiroo2 said:

Confirming that this worked for me as well.  After deleting and reinstalling the docker, it took a while to apply permissions to my recordings, and then the old recordings don't show in the new console, but i'm back online again at least.  I'll look around to see if I can get the old recordings to appear again, but I'm guessing that they can just be deleted from storage.

Further to this, after restoring the backup and restoring my old recordings, I noticed that the version number in settings was still 3.9.x, but in the main web portal it was 3.10.1.  After restarting the docker, I got the dreaded "Updating Unifi-Video" page again and my heart sank...luckily it sped through the different steps and I was running again in less than a minute with both pages showing 3.10.1.

[CorneliousJD]

3 hours ago, Andiroo2 said:

Further to this, after restoring the backup and restoring my old recordings, I noticed that the version number in settings was still 3.9.x, but in the main web portal it was 3.10.1.  After restarting the docker, I got the dreaded "Updating Unifi-Video" page again and my heart sank...luckily it sped through the different steps and I was running again in less than a minute with both pages showing 3.10.1.

You know, same thing happened to me actually. I didn't think much of it at the time, though maybe some weird browser cache was causing it, but it's all showing 3.10.1 now and haven't had issues with it since. fingers crossed!

[IamSpartacus]

I'm guessing this is why everyone who upgraded had issues?

 

Quote

To ensure a smooth upgrade path, NVR appliances must upgrade to 3.9.12 before upgrading to 3.10.0+.  Then, upgrading an NVR from 3.9.12 to 3.10.0+ must be done via the web UI update.  Subsequent updates can then be performed via apt, manual, etc. unless otherwise noted in the release notes.  Since the soft release requires a manual install, here is the safe way to download and install it (this also works if you already attempted an install via dpkg, you're basically just reinstalling with the proper JRE):

https://community.ubnt.com/t5/UniFi-Video-Blog/UniFi-Video-3-10-1-Full-Release/ba-p/2674717

[fryfrog]

I've been passively watching the problems while on vacation, but it doesn't seem like any specific cause has been nailed down.

I run from my own copy of the repo where I try to test before pushing to the main one and mine is working fine, though I did have problems w/ the featureVersion of the mongodb when we advanced it to 3.6 and 4.0. But after stepping that up correctly, my container is happily running mongodb on the inside.

The install is *very* old though, so might be using the old form of the db instead of the new? That is what I'll look into when I have the time. And a scratch setup, which I can't see any reason it shouldn't.

Anyone good at mongodb and unifi-video want to collaborate on solving this? :/

[IamSpartacus]

Looks like there is a major memory leak when upgrading as well.  Looks like a fresh install is the only thing that fixes it according to the forums.  As of right now with my container having been running for about 12 hours now since upgrading and it's using just about 14GB of RAM.  Not good.

 

I've since limited my containers RAM usage limit to 4GB.

Edited February 22, 2019 by IamSpartacus

[Andiroo2]

On 2/22/2019 at 12:32 AM, IamSpartacus said:

Looks like there is a major memory leak when upgrading as well.  

Can confirm this. 

On 2/22/2019 at 12:32 AM, IamSpartacus said:

I've since limited my containers RAM usage limit to 4GB.

I tried this and I still get crashes. Not just my UniFi-Video docker, but it hangs my whole unRAID server. 

[IamSpartacus]

Does anyone know if it's possible to downgrade back to 3.9?  My 3.10.x container is unsable because of the memory leak as it keeps hitting my defined 4GB ram limit and then crashing.  It's unusable at this point.

[CorneliousJD]

What process are you using to check RAM usage of the container? I'm seeing only about 12% total system RAM usage with container uptime of a few DAYS so I'm not sure if I'm hitting an issue here or not? I'd like to look deeper into it. 

[IamSpartacus]

5 minutes ago, CorneliousJD said:

What process are you using to check RAM usage of the container? I'm seeing only about 12% total system RAM usage with container uptime of a few DAYS so I'm not sure if I'm hitting an issue here or not? I'd like to look deeper into it. 

 

I'm just showing the RAM constantly creeping up towards 4GB in Advanced Docker settings.  Then I keep seeing this in my syslog:

 

Feb 25 09:38:31 SPE-UNRAID01 kernel: Task in /docker/55b2e60f9f27f0028b26ff6f2afd34d31c045f1660d64d530ff2622a795f96cc killed as a result of limit of /docker/55b2e60f9f27f0028b26ff6f2afd34d31c045f1660d64d530ff2622a795f96cc
Feb 25 09:38:31 SPE-UNRAID01 kernel: memory: usage 4194304kB, limit 4194304kB, failcnt 234151298
Feb 25 09:38:31 SPE-UNRAID01 kernel: memory+swap: usage 4194304kB, limit 8388608kB, failcnt 0
Feb 25 09:38:31 SPE-UNRAID01 kernel: kmem: usage 47156kB, limit 9007199254740988kB, failcnt 0
Feb 25 09:38:31 SPE-UNRAID01 kernel: Memory cgroup stats for /docker/55b2e60f9f27f0028b26ff6f2afd34d31c045f1660d64d530ff2622a795f96cc: cache:12772KB rss:4133200KB rss_huge:1951744KB shmem:3480KB mapped_file:396KB dirty:396KB writeback:132KB swap:0KB inactive_anon:3928KB active_anon:4134452KB inactive_file:4896KB active_file:1180KB unevictable:0KB
Feb 25 09:38:31 SPE-UNRAID01 kernel: Tasks state (memory values in pages):
Feb 25 09:38:31 SPE-UNRAID01 kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 25 09:38:31 SPE-UNRAID01 kernel: [  32436]     0 32436     5840       81    81920        0             0 run.sh
Feb 25 09:38:31 SPE-UNRAID01 kernel: [  32578]     0 32578     4314       42    73728        0             0 jsvc
Feb 25 09:38:31 SPE-UNRAID01 kernel: [   8499]    99  8499  7912541  1030559 10633216        0             0 jsvc
Feb 25 09:38:31 SPE-UNRAID01 kernel: Memory cgroup out of memory: Kill process 8499 (jsvc) score 985 or sacrifice child
Feb 25 09:38:31 SPE-UNRAID01 kernel: Killed process 8499 (jsvc) total-vm:31650164kB, anon-rss:4123796kB, file-rss:0kB, shmem-rss:0kB

 

It's been doing this for a few days and only reboots fix it (until it reaches the RAM limit again).

 

For now I've rolled back to 3.9.  Lots of people complaining about memory leaks in 3.10 on the Ubiquiti forums.

Edited February 25, 2019 by IamSpartacus

[CorneliousJD]

2 minutes ago, IamSpartacus said:

 

I'm just showing the RAM constantly creeping up towards 4GB in Advanced Docker settings.  Then I keep seeing this in my syslog:

 

It's been doing this for a few days and only reboots fix it (until it reaches the RAM limit again).

 

For now I've rolled back to 3.9.  Lots of people complaining about memory leaks in 3.10 on the Ubiquiti forums.

This is all i see when I show advanced docker view, it doesn't tell me how much it's using up though.

I don't have a hard limit set on any containers that i know of, so I'm not sure how I can see how much it's actually using?

 

I don't seem to be running into any issues on 3.10 though yet after I did a fresh install of it.

 

[IamSpartacus]

9 minutes ago, CorneliousJD said:

This is all i see when I show advanced docker view, it doesn't tell me how much it's using up though.

I don't have a hard limit set on any containers that i know of, so I'm not sure how I can see how much it's actually using?

 

I don't seem to be running into any issues on 3.10 though yet after I did a fresh install of it.

 

 

I see this.  Might be new view features in 6.7RC.

 

Edited February 25, 2019 by IamSpartacus

[tobytl]

I'm seeing the same behavior as @IamSpartacus, including the OOM reaping with any memory limit set (up to 8G).

 

Beyond that... Welcome to the madness that is linux memory management...

 

For instance, in my case the issues start well before any errors ever crop up in any logs. After running the UFV container for a few hours, the Plex container starts getting weird, transcodes will be fine, but "Direct Play" content will suddenly disappear from memory while streaming which throws an error on the client about the server or connection not being fast enough to stream. A few hours after that, I'll start seeing unresponsive containers, they are running and the process inside them is running (and working), just not responding to input (webui/commandline/etc). And then after about a day of running the UFV container, the dreaded "unable to fork" and "resource temporarily unavailable" messages start popping up all over the place. All of this while the host still thinks only about 50% of ram is being used. Fun.

 

Anyway, I'm currently testing with an enforced low JVM memory heap for the container to see if that helps. By default the unifi-video script sets a heap size of 1/4 of your system memory (8G in my case), but only 1548M on their nvr appliance. But, it also has an ENV override variable, so I added a variable to the docker template for "JVM_MX" set to a value of "1548M", and it worked in setting the heap size at least. So far a couple of hours in, and it's not using a ton of memory yet, and the early symptoms haven't started. Needs a much longer testing period though.