D34DC3N73R Posted April 11, 2020 Share Posted April 11, 2020 (edited) Netdata with Nvidia GPU monitoring in a container. ⚠️ 5/9/23 - Netdata v1.39.0 - LAST RELEASE OF d34dc3n73r/netdata-glibc ⚠️ Everything this image set out to do can now be accomplished with the stock netdata image. I will be updating the template to work with netdata/netdata, and my custom image will no longer be used. The template will still be set up to work with GPUs, but when the switch to the official netdata image happens, users MUST edit python.d.conf or go.d.conf to enable GPU charts. NOTE: An Nvidia GPU and the Unraid Nvidia Plugin are recommended. If you wish to use it without a gpu, turn on Advanced View and remove '--runtime=nvidia' from Extra Parameters, and remove NVIDIA_VISIBLE_DEVICES or set it to 'void'. d34dc3n73r/netdata-glibc info This image was created due to netdata/netdata using Alpine, a musl distribution, as a base. Nvidia drivers are only compatible with glibc distributions. This image uses netdata/netdata as a base and adds a GNU C library to run binaries linked against glibc. This image does not contain nvidia-smi, but is compatible with nvidia-docker2, nvidia-container-toolkit and the Unraid Nvidia Plugin. TO INSTALL: This template is available in Community Applications. Search 'netdata glibc' and install. OVERRIDE DIRECTORY: Override support works like the official Netdata image: https://learn.netdata.cloud/docs/installation/installation-methods/docker#configure-agent-containers This image includes an override directory. The container path /etc/netdata will be mounted to /mnt/user/appdata/netdata/override. The override directory contains placeholder directories, a generated netdata.conf file, and the edit-config script. The edit-config script can be used to make edits on any stock conf file. For instance, to edit python.d.conf do the following. From the container console: /etc/netdata/edit-config python.d.conf This command with load python.d.conf file from the stock configuration directory /usr/lib/netdata/conf.d using vi as an editor. The edited file will be saved to /etc/netdata and will override the stock configuration when netdata is restarted. Subsequent edits on a file will load the file from /etc/netdata. LINKS: ----------------------------------------- Github Repository: https://github.com/D34DC3N73R/netdata-glibc Docker Hub: https://hub.docker.com/r/d34dc3n73r/netdata-glibc Unraid Template: https://raw.githubusercontent.com/D34DC3N73R/unraid-templates/master/netdata-glibc.xml ----------------------------------------- Base Project: https://www.netdata.cloud/ Base Github Repository: https://github.com/netdata/netdata Base Docker Hub: https://hub.docker.com/r/netdata/netdata Edited May 9, 2023 by D34DC3N73R update for Netdata v1.39 1 Quote Link to comment
rjlan Posted April 15, 2020 Share Posted April 15, 2020 Very cool, works great out-of-the-box. Is there a way to make a custom dashboard without having to work inside the container? https://docs.netdata.cloud/web/gui/custom/ Quote Link to comment
D34DC3N73R Posted April 15, 2020 Author Share Posted April 15, 2020 That's a good question. It seems like custom dashboards in docker containers is something the netdata team is working on. I'll try out a couple options and see if I can figure out a solution in the meantime. Quote Link to comment
D34DC3N73R Posted April 15, 2020 Author Share Posted April 15, 2020 @rjlan The current suggestion is mount a volume as a subdirectory of /usr/share/netdata/web. To accomplish that, add a new path Name: Custom Dashboards Container Path: /usr/share/netdata/web/custom Host Path: /mnt/user/appdata/netdata/web After adding your custom dashboard.html to /mnt/user/appdata/netdata/web/ visit http://YOUR.SERVER.IP:19999/custom/dashboard.html I tested this with their tv.html and their simple demo.html. One caveat is make sure to use full urls in place of relative links to dashboard.js example: <script type="text/javascript" src="dashboard.js?v20190902-0"></script> becomes <script type="text/javascript" src="http://localhost:19999/dashboard.js?v20190902-0"></script> I'll think about adding this and an override path (for netdata.conf, etc) to the template, but I'd like to see if netdata implements any changes first. Quote Link to comment
D34DC3N73R Posted April 28, 2020 Author Share Posted April 28, 2020 (edited) Update: Custom configuration support is now enabled. See the netdata configuration guide for details. /mnt/user/appdata/netdata/override is mapped to /etc/netdata. The nvidia-smi plugin is now enabled in the stock configuration so GPU support will still work with a blank override directory. Note: /mnt/user/appdata/netdata/override will be empty by design. Stock configuration files are loaded from /usr/lib/netdata/conf.d/ If you'd like to alter one of these config files, you can copy it to the /etc/netdata directory and make edits. Edit: I've done some work on the image so files from /etc/netdata will appear in the volume mount /mnt/user/appdata/netdata/override. Please reach out if you have any problems. Also, side note in regards to custom dashboards: It appears netdata is working on a react dashboard, so I'm probably not going to put much time into custom dashboards for the current implementation. Edited May 3, 2020 by D34DC3N73R add override update 1 Quote Link to comment
glompy Posted May 11, 2020 Share Posted May 11, 2020 I just installed this on a fresh system and get the following error when attempting to visit the web-ui Access to file is not permitted: /usr/share/netdata/web//index.html Quote Link to comment
glompy Posted May 12, 2020 Share Posted May 12, 2020 I fixed this by opening up the docker console and executing the following command chown netdata:netdata -R /usr/share/netdata 1 Quote Link to comment
D34DC3N73R Posted May 13, 2020 Author Share Posted May 13, 2020 (edited) On 5/11/2020 at 5:14 PM, glompy said: I fixed this by opening up the docker console and executing the following command chown netdata:netdata -R /usr/share/netdata I'm unable to reproduce this on a clean install. If you didn't edit the default template settings (specifically mounting a volume for custom dashboards), I'm guessing this was a problem with the netdata image and was fixed in a recent build. Please let me know if you run into any other problems. Edit: it does appear to be a problem with v1.22.0 and was fixed in v1.22.1 Edited May 13, 2020 by D34DC3N73R add fix info Quote Link to comment
glompy Posted May 14, 2020 Share Posted May 14, 2020 I can verify that upgrading the docker container to v1.22.1 does not have the permission issue. Thanks for taking a look and linking the fixed version! Quote Link to comment
Megaman69 Posted February 22, 2021 Share Posted February 22, 2021 Im trying to reverse proxy this with swag. I get File does not exist, or is not accessible: /usr/share/netdata/web/netdata/ Its probably something simble but im an idiot. Any suggestions? Quote Link to comment
derbo Posted March 5, 2021 Share Posted March 5, 2021 Anyone using this with 6.9? I haven't used this in a long time and noticed it's now not showing nvidia-smi. Docker variable for Nvidia visible devices is set to all. Quote Link to comment
D34DC3N73R Posted March 8, 2021 Author Share Posted March 8, 2021 On 2/21/2021 at 8:16 PM, Megaman69 said: Im trying to reverse proxy this with swag. I get File does not exist, or is not accessible: /usr/share/netdata/web/netdata/ Its probably something simble but im an idiot. Any suggestions? Are you trying to use custom dashboards? Does it work locally? Quote Link to comment
D34DC3N73R Posted March 8, 2021 Author Share Posted March 8, 2021 On 3/4/2021 at 4:26 PM, derbo said: Anyone using this with 6.9? I haven't used this in a long time and noticed it's now not showing nvidia-smi. Docker variable for Nvidia visible devices is set to all. I haven't tried this out with 6.9 yet. Do you have other containers that use the GPU passthrough and are they working? I'll have to look into 6.9 to see if they changed the way GPU settings work. Quote Link to comment
derbo Posted March 10, 2021 Share Posted March 10, 2021 On 3/8/2021 at 3:05 PM, D34DC3N73R said: I haven't tried this out with 6.9 yet. Do you have other containers that use the GPU passthrough and are they working? I'll have to look into 6.9 to see if they changed the way GPU settings work. Not sure what changed, since my post and your post but I just tried it with 6.9.1 and the latest and its working fine now. /shrug. Quote Link to comment
arostad Posted May 22, 2021 Share Posted May 22, 2021 any chance you can update the icon URL in your template since the existing icon went 404? this one looks like a good replacement.. https://raw.githubusercontent.com/netdata/netdata/master/web/gui/dashboard/images/ms-icon-310x310.png thanks in advance! 1 Quote Link to comment
D34DC3N73R Posted May 29, 2021 Author Share Posted May 29, 2021 On 5/22/2021 at 4:28 PM, arostad said: any chance you can update the icon URL in your template since the existing icon went 404? this one looks like a good replacement.. https://raw.githubusercontent.com/netdata/netdata/master/web/gui/dashboard/images/ms-icon-310x310.png thanks in advance! Updated, thanks for the heads up. 1 Quote Link to comment
Simon Posted July 17, 2021 Share Posted July 17, 2021 (edited) On 3/9/2021 at 10:07 PM, derbo said: Not sure what changed, since my post and your post but I just tried it with 6.9.1 and the latest and its working fine now. /shrug. I just tried this for the first time in a while and notice the GPU charts are missing too. The logs show: 2021-07-17 00:21:16: python.d INFO: nvidia_smi[nvidia_smi] : chart 'nvidia_smi.gpu0_processes_mem' was suppressed due to non updating 2021-07-17 00:21:16: python.d INFO: nvidia_smi[nvidia_smi] : chart 'nvidia_smi.gpu0_user_mem' was suppressed due to non updating The GPU is working fine with Plex for transcoding and nvidia-smi returns output from the command line. I'll dig deeper another time. Edited July 17, 2021 by Simon Quote Link to comment
Simon Posted July 17, 2021 Share Posted July 17, 2021 The image updated last night and nvidia smi is back! Quote Link to comment
D34DC3N73R Posted July 19, 2021 Author Share Posted July 19, 2021 I pushed and updated the template with some more recent stable release builds so latest isn't the only option. 1 Quote Link to comment
StephenCND Posted January 28, 2022 Share Posted January 28, 2022 I'm very interested in this docker, but unfortunately I do not have an Nvidia GPU. (Old AMD Radeon HD 5800). Is there any way I can install this docker? I tried installing but it failed regardles of the "branch" I selected. (I did see the note about "turn on Advanced View and remove '--runtime=nvidia' from Extra Parameters, and remove NVIDIA_VISIBLE_DEVICES or set it to 'void'", but couldn't figure out how to do so. Any assistance would be greatly appreciated. Quote Link to comment
D34DC3N73R Posted February 16, 2022 Author Share Posted February 16, 2022 On 1/28/2022 at 8:10 AM, StephenCND said: I'm very interested in this docker, but unfortunately I do not have an Nvidia GPU. (Old AMD Radeon HD 5800). Is there any way I can install this docker? I tried installing but it failed regardles of the "branch" I selected. (I did see the note about "turn on Advanced View and remove '--runtime=nvidia' from Extra Parameters, and remove NVIDIA_VISIBLE_DEVICES or set it to 'void'", but couldn't figure out how to do so. Any assistance would be greatly appreciated. You can use the standard netdata image, unless you'd like to use it for the override support. In that case, make sure you're in the advanced view (upper right corner after clicking "install") In "Extra Parameters" delete `--runtime=nvidia` Then click "Show more settings ..." and click "Remove" on the `NVIDIA_VISIBLE_DEVICES` environment variable. Quote Link to comment
tourist Posted June 16, 2022 Share Posted June 16, 2022 Much thanks for creating this, it's been very helpful. Question: what about the 'Sign Up To Cloud' button in the upper-right? I clicked it, gave an email, got an email taking me to directions for all the ways to connect my systems to netdata's cloud storage. One of them was for Docker, and the main difference seemed to be that it specified a base 64 rando identity param named NETDATA_CLAIM_TOKEN, and another NETDATA_CLAIM_URL=https://app.netdata.cloud So I edited my netdata-glibc config and added those at the end of Extra Parameters- --runtime=nvidia --cap-add SYS_PTRACE --security-opt apparmor=unconfined -e NETDATA_CLAIM_TOKEN=rando-string-goes-here -e NETDATA_CLAIM_URL=https://app.netdata.cloud Clicked apply, container restarts, however the 'Sign Up To Cloud' button doesn't disappear, and following signin at https://app.netdata.cloud/spaces it says I have no nodes. Has anybody else figured out how to link netdata-glibc with netdata.cloud? Quote Link to comment
D34DC3N73R Posted June 17, 2022 Author Share Posted June 17, 2022 It's something I've been testing as well, but I haven't been able to put much time into it recently. I'll update the template when I can get it working reliably. Quote Link to comment
Reynald Posted November 11, 2022 Share Posted November 11, 2022 Hello, I've sorted the claim thing by adding a volume and claiming via script. It survives reboots Added this volume mount in the template: /var/lib/netdata/cloud.d/ -> /mnt/user/appdata/netdata/cloud.d/ As per read here: https://learn.netdata.cloud/docs/agent/claim#connect-an-agent-running-in-docker Quote For the connection process to work, the contents of /var/lib/netdata must be preserved across container restarts using a persistent volume. See our recommended docker run and Docker Compose examples for details. (well, this doc is quite outdated because mounting /etc/netdata or /var/lib/netdata won't work as we know...) Then ran this command on host: docker exec -it netdata netdata-claim.sh -token=TOKEN -url=https://api.netdata.cloud As per documentation: https://learn.netdata.cloud/docs/agent/claim#using-docker-exec Maybe 'netdata-claim.sh -token=TOKEN -url=https://api.netdata.cloud' works in container console from unraid GUI instead of ssh'ing in the host (but as I'm an SSH man ...) Happy supervision! Reynald 1 Quote Link to comment
Reynald Posted November 11, 2022 Share Posted November 11, 2022 (edited) 2 hours ago, Reynald said: Maybe 'netdata-claim.sh -token=TOKEN -url=https://api.netdata.cloud' works in container console from unraid GUI instead of ssh'ing in the host (but as I'm an SSH man ...) I confirm this is working Edited November 11, 2022 by Reynald Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.