[support] dlandon - Zoneminder 1.36


Recommended Posts

1 hour ago, ThreeFN said:

If it's still growing, My next best guess is that ZM config may have an error in it.  Have you changed this?

32548086_Screenshot2021-01-08154947.png.23d5c94bcdee8d718a37c56d52ba1e4a.png

 

Does the rate of growth of the container match your expected rate of recorded video?

 

Your config and run command looks correct to me. 

 

/mnt/disks/cctv/zoneminder has files in it, but does it have the LATEST files and ALL the files?  If the cctv disk wasn't mounted at one point when the docker started some files might go into the image instead, I'm pretty sure I goofed and broke things that way one time.

 

 

 

No it doesn't match. Videos of each camera at day time are 1.3GB per 30 mins for each camera. the docker image is growing like 1.5GB per day or something like that.

 

This are the latest events:

image.thumb.png.0d0375225ec25b12889adec125d78974.png

 

And this is an SCP session I did to verify the files are there:

image.thumb.png.927422d6ae501c42d148f7e6469daf64.png

image.png.d212f6e621a8ea6461d7909fc7264ad5.png

 

Also these are the storage options on zoneminder, I haven't mess with that:

image.thumb.png.8218c83386e76318dd087b8c32433594.png

 

I'm thinking maybe these are the logs or the database that zoneminder is saving on the docker image instead of the appdata folder or something like that?

 

EDIT: I think it's something related to temporary files or something on the docker, just restarted the zoneminder docker and it's down to 1.86GB

Edited by BlueRyuu
Link to comment
13 minutes ago, BlueRyuu said:

No it doesn't match. Videos of each camera at day time are 1.3GB per 30 mins for each camera. the docker image is growing like 1.5GB per day or something like that.

 

This are the latest events:

image.thumb.png.0d0375225ec25b12889adec125d78974.png

 

And this is an SCP session I did to verify the files are there:

image.thumb.png.927422d6ae501c42d148f7e6469daf64.png

image.png.d212f6e621a8ea6461d7909fc7264ad5.png

 

Also these are the storage options on zoneminder, I haven't mess with that:

image.thumb.png.8218c83386e76318dd087b8c32433594.png

 

I'm thinking maybe these are the logs or the database that zoneminder is saving on the docker image instead of the appdata folder or something like that?

 

EDIT: I think it's something related to temporary files or something on the docker, just restarted the zoneminder docker and it's down to 1.86GB

If it 'went back down' it would seem that it was temp files, perhaps package updates or something.

 

Your database should be stashed in appdata so that shouldn't grow the image unless it's mirrored over for some reason.

 

It does seem odd that your 1 day change originally was 'exactly' doubling in size, from 3.31GB to 6.62GB which seems more than coincidental.

 

For reference I'm running 3.06GiB image for my 222GiB of video at the moment.

Link to comment
1 hour ago, ThreeFN said:

If it 'went back down' it would seem that it was temp files, perhaps package updates or something.

 

Your database should be stashed in appdata so that shouldn't grow the image unless it's mirrored over for some reason.

 

It does seem odd that your 1 day change originally was 'exactly' doubling in size, from 3.31GB to 6.62GB which seems more than coincidental.

 

For reference I'm running 3.06GiB image for my 222GiB of video at the moment.

Wow, didn't notice it was exactly the double in size. Yep, that's weird. Well, gonna keep monitoring it for now.

 

Thanks for the help.

Link to comment

@dlandon

 

I'm saddened you're dropping ES/ML from your container but definitely understand your view and desire to simplify what you're supporting.  Thanks for all you've done! Is it possible those that want to keep ES/ML within this container to do so?  I can't really do a VM because within unraid the GPU can't be shared across VM and Docker containers and used at the same time.

Link to comment
23 minutes ago, repomanz said:

@dlandon

 

I'm saddened you're dropping ES/ML from your container but definitely understand your view and desire to simplify what you're supporting.  Thanks for all you've done! Is it possible those that want to keep ES/ML within this container to do so?  I can't really do a VM because within unraid the GPU can't be shared across VM and Docker containers and used at the same time.

The Docker will stay as it is, so yes you can just stay with what you have.  The new Docker with ES only is a new Docker called zonemnder.unraid.  When Zoneminder version 1.36 comes out, you will be behind though.

 

The ML part of ES is very detailed and difficult to handle in a Docker. A VM or stand alone computer is a much better solution.

Link to comment
42 minutes ago, dlandon said:

The Docker will stay as it is, so yes you can just stay with what you have.  The new Docker with ES only is a new Docker called zonemnder.unraid.  When Zoneminder version 1.36 comes out, you will be behind though.

 

The ML part of ES is very detailed and difficult to handle in a Docker. A VM or stand alone computer is a much better solution.

So I understand fully; your container will have zm and es?  What about hooks or is that part of the ML install?

Edited by repomanz
Link to comment
17 hours ago, ThreeFN said:

ES does support remote ML server as described here.

 

Which if you go down the rabit hole, you get pliablepixels mlapi software, which does have a containerized version someone has made (and may have GPU support?).  It may be possible even now to glue this all together.

 

Obviously experimentation must ensue.

 

The more I dig through stuff the more I tend to agree with dlandon that this container is doing a lot of 'init install' stuff that is more OS/VM behavior than docker pull/start behavior and I don't fault wanting to kick that to the curb.

 

Having said that ZM is 'almost exclusively made useful' by ES's ML stuff for false positive rejection, so no ML = no ZM for me.  So at the moment it looks like the options are spin up a VM and get a well supported installation (pliablepixels documentation), or investigate the aforementioned remote ML docker.  My preference is for the latter because at the moment, containerization is about the only way in unraid to spread a GPU amongst multiple work loads (eg plex & ML) unless/until vGPU/SRIOV/etc support is added and VMs & docker can share.

 

I guess the other solution would be to move ML processing to a Google Coral device, and give that device to the ZM VM.  Or even go the route of a TPU & mlapi VM remoted to the ZM docker.  The benchmarks seem to indicate the TPU is good but maybe hamstrung a bit by it's usb 2.0 (or slower) interface to get the data.  Not sure if the M.2 versions would be any faster, if TPU can't saturate 2.0 that seem like the chip is the bottleneck and not it's interface...

 

Hell of a device for the cost, size, and power draw though...

 

Dlandon, I'm guessing you'll be dropping GPU support entirely from the docker?  Like even for ffmpeg use directly by ZM and not for ML (h.264 decode etc)?  Or is that something that doesn't require your direct support (give container gpu, ffmpeg sorts it out) and will work on its own?

@threefn This is almost exactly the same I would have wrote. We´re in the same boat for sure :)

Edited by Sic79
Link to comment
The time has come for me to make some changes to the Zoneminder Docker.  It has become a real support burden for me and I just don't have the time for all the support issues.  For those of you using ES machine learning (hook processing), you will need to move to building a VM for Zoneminder.  This Docker is acting more like a VM than a Docker and that is not how Dockers are supposed to work.
 
There will be no more updates to this Docker.  I will be building a new Zoneminder Docker that will have basic ES notifications of events and that is all.
Thanks for all your work on this, completely understand. It's been great to use.

Sent from my Pixel 3a using Tapatalk

Link to comment
7 hours ago, Sic79 said:

@threefn This is almost exactly the same I would have wrote. We´re in the same boat for sure :)

Unfortunately I spoke too soon.  If I had read the FAQ with more comprehension, the current mlapi implementation still requires most if not all of the ML libraries be installed on the ZM install, so that isn't a solution just yet.  The mlapi docker hasn't been updated in 7 months but you will probably have to setup/build the docker yourself because of the compile requirements of opencv so I don't think one click install app version of a ML docker is in the cards for most unraid users.  A VM following the instructions would therefore be 'easier' for most users.

 

And to continue to rain on the parade, it looks like the Google Coral TPU isn't really an interchangeable processor like Yolo and cpu/gpu, the coral runs different models and has a different performance profile (faster but less successful detections) and is more used as a separate suit of detectors, ,like object vs face vs alpr etc.  I'm not entirely sure what all objects tpu can detect to be honest, it may or may not be the same list as the yolo list.

 

So It's going to have to be a separate GPU and a VM for the foreseeable future if one wants 'max performance', at a TPU if you're curious, and wait for pliablepixels to continue development on the MLAPI advancing to a less kludgy (his words) implementation.

Link to comment
1 hour ago, ThreeFN said:

Unfortunately I spoke too soon.  If I had read the FAQ with more comprehension, the current mlapi implementation still requires most if not all of the ML libraries be installed on the ZM install, so that isn't a solution just yet.  The mlapi docker hasn't been updated in 7 months but you will probably have to setup/build the docker yourself because of the compile requirements of opencv so I don't think one click install app version of a ML docker is in the cards for most unraid users.  A VM following the instructions would therefore be 'easier' for most users.

 

And to continue to rain on the parade, it looks like the Google Coral TPU isn't really an interchangeable processor like Yolo and cpu/gpu, the coral runs different models and has a different performance profile (faster but less successful detections) and is more used as a separate suit of detectors, ,like object vs face vs alpr etc.  I'm not entirely sure what all objects tpu can detect to be honest, it may or may not be the same list as the yolo list.

 

So It's going to have to be a separate GPU and a VM for the foreseeable future if one wants 'max performance', at a TPU if you're curious, and wait for pliablepixels to continue development on the MLAPI advancing to a less kludgy (his words) implementation.

I believe you're correct here; looking through dlandon's container based on a fresh install (this morning) it appears even eventserver items have been stripped.  Therefore within this container; I am unable to configure a remote ML server as that information / configuration exists within the objectconfig.ini which is no longer there.

further, the mlapi project looks to have dependencies with zm itself, for example in the documentation states you need to run: python3 mlapi_adduser.py but it errors because the ./db path is empty.  I can only assume that is zoneminder's db path which doesn't exist on a separate vm while being used by dlandon's container.

I may have to move everything to a separate VM to get going again.  Side note; are there any ubuntu 20 containers out there I can install on unraid? I really wanted the GPU configuration of this installation but I can't hook both in containers and vm's at the same time.

Link to comment
22 hours ago, BlueRyuu said:

Hello.

Is it normal that the docker container image is growing too fast?

Yesterday it was 3.31GB and today it is  6.62GB.

image.png.64a0ad2e793b22220d5e372a08bfdb77.png

 

I'm saving the videos to an Unassigned device (Data Path).

At this rate it will fill up my docker image in less than a week :(

sorry to jump in the conversation but how you create videos ? My alarm only generate multiple jpg. I need to manual export via webui to create a video file.

 

Thank you in advance!

Link to comment

Are you all beginning to see what a beast ES/ML can be and why it was so hard for me to work it out in a Docker?  Every time I thought I had it going properly, things in ES would change and I had to start over.  I know that the Docker was a pretty quick and easy way to get the ES/ML going, but apparently very few here thought my efforts were valuable enough to support my work.

Link to comment
Unfortunately I spoke too soon.  If I had read the FAQ with more comprehension, the current mlapi implementation still requires most if not all of the ML libraries be installed on the ZM install, so that isn't a solution just yet.  The mlapi docker hasn't been updated in 7 months but you will probably have to setup/build the docker yourself because of the compile requirements of opencv so I don't think one click install app version of a ML docker is in the cards for most unraid users.  A VM following the instructions would therefore be 'easier' for most users.
 
And to continue to rain on the parade, it looks like the Google Coral TPU isn't really an interchangeable processor like Yolo and cpu/gpu, the coral runs different models and has a different performance profile (faster but less successful detections) and is more used as a separate suit of detectors, ,like object vs face vs alpr etc.  I'm not entirely sure what all objects tpu can detect to be honest, it may or may not be the same list as the yolo list.
 
So It's going to have to be a separate GPU and a VM for the foreseeable future if one wants 'max performance', at a TPU if you're curious, and wait for pliablepixels to continue development on the MLAPI advancing to a less kludgy (his words) implementation.

So sad it had to be like this since it worked so well, but sure I can understand@dlandon decision too.

Don’t like the idea to move to a VM though since I can’t even fit an extra GPU in my server. So for me it isn’t a choice :(.

I checked the “AppStore” and saw that@ich777 had a docker with Debian Buster that had Nvidia support. Maybe it is possible to install everything in that one just to test while waiting/working on a better solution.
Link to comment
2 hours ago, Sic79 said:

I checked the “AppStore” and saw that@ich777 had a docker with Debian Buster that had Nvidia support. Maybe it is possible to install everything in that one just to test while waiting/working on a better solution.

Good luck with that.  The only platform that seems to support the latest in GPU hardware is Ubuntu 20.04.  I really don't think you realize how much work is involved in the ML part of ES.  My Docker made it real easy to just configure ML and go.  All the setup and configuration was pretty much done for you.  Maybe I made it too easy as hardly anyone seems to have an appreciation for the amount of work involved to contribute towards my efforts.

Link to comment
2 hours ago, Jokerigno said:

sorry to jump in the conversation but how you create videos ? My alarm only generate multiple jpg. I need to manual export via webui to create a video file.

 

Thank you in advance!

You need to go into your camera config and change how it records, read the ZM documentation for more details on what does what:

image.png.fbc8a42b55a057c240dac0bdff2ff0c8.png

 

2 hours ago, repomanz said:

I believe you're correct here; looking through dlandon's container based on a fresh install (this morning) it appears even eventserver items have been stripped.  Therefore within this container; I am unable to configure a remote ML server as that information / configuration exists within the objectconfig.ini which is no longer there.

further, the mlapi project looks to have dependencies with zm itself, for example in the documentation states you need to run: python3 mlapi_adduser.py but it errors because the ./db path is empty.  I can only assume that is zoneminder's db path which doesn't exist on a separate vm while being used by dlandon's container.

I may have to move everything to a separate VM to get going again.  Side note; are there any ubuntu 20 containers out there I can install on unraid? I really wanted the GPU configuration of this installation but I can't hook both in containers and vm's at the same time.

Yah, there's a lot going on, I don't pretend to have found it all, mlapi isn't designed to be completely on a remote device just yet, reading between the lines it seems one reason for making the api was so that the ML models could stay in memory so they don't have to be loaded each time, which apparently they have to be loaded on each 'image check' for one reason or another?  I'm not a code monkey to catch evverything, only know enough to be dangerous.

 

2 hours ago, dlandon said:

Are you all beginning to see what a beast ES/ML can be and why it was so hard for me to work it out in a Docker?  Every time I thought I had it going properly, things in ES would change and I had to start over.  I know that the Docker was a pretty quick and easy way to get the ES/ML going, but apparently very few here thought my efforts were valuable enough to support my work.

 

14 minutes ago, dlandon said:

Good luck with that.  The only platform that seems to support the latest in GPU hardware is Ubuntu 20.04.  I really don't think you realize how much work is involved in the ML part of ES.  My Docker made it real easy to just configure ML and go.  All the setup and configuration was pretty much done for you.  Maybe I made it too easy as hardly anyone seems to have an appreciation for the amount of work involved to contribute towards my efforts.

Never doubted it was a octopus crossed with an ivy tressel of a project, probably never really capable nor intended to run this way.  Put it on a resume for those that can begin to understand what all zm ml docker has to do.

 

I'll be honest, I've always played with ML each time I refactored my 'server solutions' as I would get bored/annoyed with current solution, but would inevitably roll back to no ML and just ZM record 24/7, sometimes with this docker, sometimes a vm, sometimes freenas/truenas plugin (don't start me on that one).  Lets be honest a cctv system is 'only good' if it A) records everything, and B) never misses anything and C) that in a pinch you could go back and find if you had to in case of breakin/theft etc.  Everything else is sex appeal and gravy, as fantastic and valuable as it can be, and I can't fault anyone buying/building a race car when you only need a honda/toyota to get around ;) .  This last 'install' for me was probably the first time I had the confluence of knowledge, understanding, documentation, examples, code state, luck, and so on to get it all working and work in a way that wouldn't be an annoyance with zmninja/etc.  And then it shutters (which I do not fault).  But I probably owe you a few pints for the entertainment value, you're right.

 

Again, the only real annoyance for me being told 'you're on your own' in the end is that docker, right now, is probably the 'easiest' way to share gpu power (caveat emptor) amongst home server tasks (plex,ml,tinkering,etc) but not as easy as spooling a VM from the development/deployment standpoint.

 

2 hours ago, Sic79 said:


So sad it had to be like this since it worked so well, but sure I can understand@dlandon decision too.

Don’t like the idea to move to a VM though since I can’t even fit an extra GPU in my server. So for me it isn’t a choice :(.

I checked the “AppStore” and saw that@ich777 had a docker with Debian Buster that had Nvidia support. Maybe it is possible to install everything in that one just to test while waiting/working on a better solution.

I'll have to investigate this.  I remember early days docker (for me) not understanding how docker is designed to be immutable, messing around inside docker, updating/upgrading it, and then wondering 'where did all the stuff I did go?'  If this ends up being a solution one will have to be VERY careful not to accidentally kill this docker with an update/upgrade etc.

  • Like 1
Link to comment
10 hours ago, dlandon said:

Good luck with that.  The only platform that seems to support the latest in GPU hardware is Ubuntu 20.04.  I really don't think you realize how much work is involved in the ML part of ES.  My Docker made it real easy to just configure ML and go.  All the setup and configuration was pretty much done for you.  Maybe I made it too easy as hardly anyone seems to have an appreciation for the amount of work involved to contribute towards my efforts.

Hm seems like a bit of sarcasm here. Well, if you remember I was one of them who helped you trial and error the first steps on getting the opencv compile working in the first place. So yes, I understand complexness in getting everything working smooth even if I´m not on your level of knowledge in dockers. I work as a IT-pro/dev so I can see your point in that ppl doesn´t realise how much work something that can seem very simple on the surface is very complex underneat.

 

We just want a new solution that´s is not a VM so we can share a single GPU amongst dockers.

I can see that this is not going anywhere so discussion on a new solution should be elsewhere.

 

So @dlandon thanks for the time, it´s been great using your ZM docker with ML.

 

Link to comment
1 hour ago, Sic79 said:

Hm seems like a bit of sarcasm here.

Sorry, didn't mean to come across like that.  You need to understand that the Unraid community is only a small base of the Zoneminder users.  People saw the Docker and started using it on all manner of hosts - Macs, Windows, Debian, etc and then asked for support for that particular platform.  There is no way possible for me to do that.  ES/ML is also not that easy to get into a Docker and have it act like a Docker.

 

I think the way forward is for the Docker to be rebuilt into a ZM and ES/ML only Docker and get rid of the environment variables to configure the Docker at runtime.  The Docker that is in CA right now could stay for those like me that don't use ML.  The issue is the amount of time that will be needed to accomplish this that I just can't 'donate' at this point.  I'm thinking of a funding campaign to get funding so I can commit the time to the task.

 

This is a difficult decision for me because I volunteer my time for Unraid, and I'm not here to make money.  Unfortunately for me, this Docker has been too successful with over 5 Million downloads and has ended up taking way too much time from my other work.

Link to comment
1 minute ago, dlandon said:

Sorry, didn't mean to come across like that.  You need to understand that the Unraid community is only a small base of the Zoneminder users.  People saw the Docker and started using it on all manner of hosts - Macs, Windows, Debian, etc and then asked for support for that particular platform.  There is no way possible for me to do that.  ES/ML is also not that easy to get into a Docker and have it act like a Docker.

 

I think the way forward is for the Docker to be rebuilt into a ZM and ES/ML only Docker and get rid of the environment variables to configure the Docker at runtime.  The Docker that is in CA right now could stay for those like me that don't use ML.  The issue is the amount of time that will be needed to accomplish this that I just can't 'donate' at this point.  I'm thinking of a funding campaign to get funding so I can commit the time to the task.

 

This is a difficult decision for me because I volunteer my time for Unraid, and I'm not here to make money.  Unfortunately for me, this Docker has been too successful with over 5 Million downloads and has ended up taking way too much time from my other work.

No worries, and no hard feelings.

 

Well, maybe a campaign is not a bad idea at all. If you think about the alternatives to Zoneminder, there are not many complete solutions (if none?) that include everything for free. This is also a great way to make people really show their appreciation for your work/time and to actually support the developement in their way.

Link to comment
14 minutes ago, Sic79 said:

No worries, and no hard feelings.

 

Well, maybe a campaign is not a bad idea at all. If you think about the alternatives to Zoneminder, there are not many complete solutions (if none?) that include everything for free. This is also a great way to make people really show their appreciation for your work/time and to actually support the developement in their way.

One of the things that's a problem with Unraid users is that the funding campaign will have to be significant to cover all the users outside the Unraid community.  I do not expect Unraid users to bear the brunt of the funding and fully fund the efforts.  With so many users out there, it shouldn't be a problem.

 

I'm working on it.

  • Like 1
Link to comment
9 hours ago, Sic79 said:

We just want a new solution that´s is not a VM so we can share a single GPU amongst dockers.

I just want to echo this. The benefit of sharing Unraid resources, CPU and GPU included, is one of the primary reasons I leverage this docker rather than an alternate solution.  If the only way to get ML going forward is to move to a VM then I'm probably inclined to look at other options as well, BlueIris probably.

Link to comment

regarding # sudo -u www-data /var/lib/zmeventnotification/bin/zm_event_start.sh 1 1 Traceback (most recent call last):   File "/var/lib/zmeventnotification/bin/zm_detect.py", line 27, in <module>     import zmes_hook_helpers.utils as utils ModuleNotFoundError: No module named 'zmes_hook_helpers'

On 12/30/2020 at 1:06 PM, ThreeFN said:

 

 

This seems to be the fix for the time being:

 

https://github.com/dlandon/zoneminder/issues/145

Does anyone happen to know or have the info that was posted at the github link above? The repo is archived and link to issue gone.  I'm just looking at getting this going after installing quite some time ago with a very brief success before an update broke thingst.  Looks like I have the above issue and am hoping to get the version of the unRaid docker I have installed working for now until I decide a new path forward.  I'm concerned upgrading now will just cause more issue for me even though I'll be a bit outdated.  

 

That said if I upgrade to the latest deprecated full docker with hook support, does it actually work with people/object detection or is it broken? I don't necessarily need GPU support for now.  It's a bit tough to tell from the recent comments what state it's in for issues.  

 

and thanks Dlandon for the efforts you have put in, it sounds like a bear to manage. I wish I had worked on this sooner...  I'd like to think I would have been a supporter if I could have got this working and still will if you decide to go to some pay model. Easy to say after the fact I know...  Heck, even if I can get the deprecated one working I'd be happy to contribute.  I can't tell you how much time I have put in previously on getting basic zoneminder working on a VM let alone the object ML stuff which I never touched.  I really think you have something here with the docker/hook processing when it works and was so looking forward to that ability/option.

Link to comment
  • dlandon changed the title to [support] dlandon - Zoneminder 1.36

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.