Arbadacarba Posted April 26 Share Posted April 26 Living in a rural area I have been struggling with internet connections for years, and now that I have Fiber being installed (in a year maybe?) I have found the tool I always wished I had: NetProbe Sadly there is no listing in Community Applications... I managed to install it from the Git repository (https://github.com/plaintextpackets/netprobe_lite) using Compose.Manager and make it work, but boy does it make a mess of my Docker Tab. I'm just posting ot see if there is enough interest, someone might decide to make a more functional Docker out of it. Further, am I wrong in thinging it would be better to have one install of Grafana rather than multiple? Same with MariaDB? Thanks Arbadacarba Quote Link to comment
bmartino1 Posted April 28 Share Posted April 28 (edited) ? so your wanting a all in one encompassing docker with graffan / mysql for this service? or to introudce this to the community app store? see: https://forums.unraid.net/topic/151967-publishing-to-community-apps/ https://forums.unraid.net/topic/101424-how-to-publish-docker-templates-to-community-applications-on-unraid/ Closest I can find of a premade Docker pull is https://hub.docker.com/r/aojea/docker-netprobe/tags But no template though... Looks interesting. Though Edited April 28 by bmartino1 Forum crazy ness with post linkes and deleting words Quote Link to comment
bmartino1 Posted April 28 Share Posted April 28 (edited) ?Care to share your Docker compose? Reviewing as this may need multiple servers. I may make a LXC. 1. Give me a ubuntu VM close to docker and allows me to have a vm environment to run install and do networking things. 2. would be easier for me to implement and make 3. I don't know how to make a docker image container... after reviewing the youtube video and docker data in github. I will make a template guide in next post to recreate this and manage it... Then post instruction hare latter. This is cool and a Friend was looking for something like this as we experimented with other dockers and stuff in past. Project a bit bigger then myself to do / chew on to make a all in one... it be a bit of a waste. but ?lxc docker run script? lxc vm ip is grafana web ui.... Edited April 28 by bmartino1 out of scope Quote Link to comment
bmartino1 Posted April 28 Share Posted April 28 (edited) @Zafoquat To use as is... atm as there are 5 dockers this creates and runs this. Install docker compose plugin and user script plugin (nerd tools plugin for screen application). Now in a terminal. cd to the appdata folder cd /mnt/user/appdata run git command to download application: git clone https://github.com/plaintextpackets/netprobe_lite.git This will make a netprobe folder and data in the app data location. next cd in the downloaded folder and edit enviroments file: cd netprobe_lite root@BMM-Unraid:/mnt/user/appdata/netprobe_lite# ls Dockerfile README.md compose.yml config/ entrypoint.sh helpers/ logs/ netprobe.py presentation.py requirements.txt nano .env and make changes here is my config: Quote root@BMM-Unraid:/mnt/user/appdata/netprobe_lite# cat .env DEVICE_ID="44c42e36-f2eb-44d0-babc-1a749498e016" SITE_ID="f0ac2f33-1a20-471d-b430-d905900e0e71" PROBE_INTERVAL="30" PROBE_COUNT="50" SITES="google.com,youtube.com,amazon.com,uillinois.edu" DNS_TEST_SITE="google.com" # This is the site used in the DNS test DNS_NAMESERVER_1="Google_DNS" # Change these values (lines 7 - 12) if you want to test other DNS servers DNS_NAMESERVER_1_IP="8.8.8.8" DNS_NAMESERVER_2="OPENDNS" DNS_NAMESERVER_2_IP="208.67.222.222" DNS_NAMESERVER_3="CloudFlare_DNS" DNS_NAMESERVER_3_IP="1.1.1.1" DNS_NAMESERVER_4="My_DNS_Server" # Do not change this line at all! DNS_NAMESERVER_4_IP="192.168.1.10" # Replace this IP with the DNS server you use at home REDIS_URL = "netprobe-redis" REDIS_PORT = "6379" REDIS_PASSWORD = "password" # Health Score Weights weight_loss = ".6" # Loss is 60% of score weight_latency = ".15" # Latency is 15% of score weight_jitter = ".2" # Jitter is 20% of score weight_dns_latency = "0.05" # DNS latency is 0.05 of score threshold_loss = "5" # 5% loss threshold as max threshold_latency = "100" # 100ms latency threshold as max threshold_jitter = "30" # 30ms jitter threshold as max threshold_dns_latency = "100" # 100ms dns latency threshold as max PRESENTATION_PORT = "5000" PRESENTATION_INTERFACE = "0.0.0.0 I chose to edit the public sites for ping add my piehole dns and remove quad9 and use opendns as theses are the primary dns servers we use for illinois LAN tcp/ip net traffic. then in the same folder after making your edits run docker compose up --This will grab and install and run all the necessary dockers and options. ***LEAVE THIS WINDOWS OPEN!!! we will be using it latter... Your docker page will look like this when done: Thats OK we are just installing them atm: use the user script plugin to start this compose up from the app data folder at startup of array. we can edit the compose.yml file to make docker edits if necessary but I didnt' find that necessary. The other dockers can be ignored(but required as they are running the codes and python to collect the data we want, we are focused on the netprobe-grafana docker. as that will show us the data. a compose edit will be needed to add web ui to docker page. As its using bridge to connect to the others and host to access Grafana. Otherwise on lan go to unraid ip address of Grafana docker port 3001 login with admin:admin so http://192.168.1.254:3001 ok now to fix some things to use as is: go back to the terminal windows and hit ctrl +c to stop them gracefully: If you closed it you can still stop them via: root@BMM-Unraid:/mnt/cache/appdata/netprobe_lite# docker compose stop FYI this also leaves all the dockers in unraid, but they still need to be edited via the docker compose file...: ^I would need to dysect the docker compose for the template docker run variables to each docker. to make it its own template... to use unraids docker web ui to edit the data in the dockers... So lets add the web ui option for: docker run -l net.unraid.docker.webui='https://[IP]:[PORT:3001]' to the compose file. we want to touch grafan only. this is a yaml file syntax spacing is a must! grafana: restart: always image: grafana/grafana-enterprise labels: - net.unraid.docker.webui='' #example: http access webUI http://192.168.1.254:3001 container_name: netprobe-grafana volumes: - ./config/grafana/datasources/automatic.yml:/etc/grafana/provisioning/datasources/automatic.yml - ./config/grafana/dashboards/main.yml:/etc/grafana/provisioning/dashboards/main.yml - ./config/grafana/dashboards/netprobe.json:/var/lib/grafana/dashboards/netprobe.json ports: - '3001:3000' networks: - custom_network # Attach to the custom network Now let's make our start script: I recommend installing nerd tool to have the screen command: https://linux.die.net/man/1/screen add a new script: Gear> Edit script: #!/bin/bash #sleep 30 #screen -S "Netprobe" -d -m /path/to /script.sh cd /mnt/cache/appdata/netprobe_lite && docker compose up -d --In theory unraid web UI can autostart all dockers. But to maintain control via terminal screen / detached would be needed... https://docs.docker.com/reference/cli/docker/compose/up/ At this point you could technically make a new stack and copy the compose file into the unraid gui as the dockers have already been created with the env file. Edited April 28 by bmartino1 compose file fix Quote Link to comment
bmartino1 Posted April 28 Share Posted April 28 (edited) I don't have time too,,,, I would love to learn to make a docker image. But this is a lot of editing to make. But possible to run similar to test and use a database to collect data. More of a issue to maintain/ keep its parts updated. all in all the git and scripts are not bad and properly setup. Thanks for sharing. Sorry, used post as a hijacked rock wall for notes. Not sure what you did to "have mess of docker tab"? Edited April 28 by bmartino1 Quote Link to comment
bmartino1 Posted May 17 Share Posted May 17 (edited) With some updates on how this projects uses its data. if i were to remove the docker or update the docker I lose all the stats colected. To fix and use with unraids to use the docker compose plugin we have to make a simple change and not use user script. add a new stack, hit advance and set the git download pat to the docker: This will create a docekr-compose.yml file in that path we want to rename the compose.yml file as a backup Due to the project layout. we want to keep the netproe data in/from Grafana. That data should persist between docker stops, image deletes, upgrades and Grafana changes. and use the web UI to add the compose file: to do this, I made some adjustments to the compose file: version: "3" networks: custom_network: driver: bridge ipam: config: - subnet: 172.21.0.0/16 services: redis: restart: always container_name: netprobe-redis image: redis:latest labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/A75G/docker-templates/master/templates/icons/redis.png' volumes: - ./config/redis/redis.conf:/etc/redis/redis.conf networks: - custom_network logging: options: max-size: 50m max-file: 1 netprobe: restart: always container_name: netprobe-probe build: . volumes: - .:/netprobe_lite environment: MODULE: NETPROBE labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/simonjenny/fastcom-mysql/master/fastlogo.jpg' networks: - custom_network logging: options: max-size: 50m max-file: 1 presentation: restart: always container_name: netprobe-presentation build: . volumes: - .:/netprobe_lite environment: MODULE: PRESENTATION labels: - net.unraid.docker.icon='https://github.com/chvb/docker-templates/raw/master/chvb/img/OnlyOfficeDocumentServer.jpg' networks: - custom_network logging: options: max-size: 50m max-file: 1 prometheus: restart: always container_name: netprobe-prometheus image: prom/prometheus labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/selfhosters/unRAID-CA-templates/master/templates/img/prometheus.png' volumes: - ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml networks: - custom_network logging: options: max-size: 50m max-file: 1 grafana: restart: always image: grafana/grafana-enterprise:latest labels: - net.unraid.docker.webui=http://[IP]:[PORT:3001] - net.unraid.docker.icon='https://github.com/atribe/unRAID-docker/raw/master/icons/grafana.png' container_name: netprobe-grafana volumes: - ./config/grafana/datasources/automatic.yml:/etc/grafana/provisioning/datasources/automatic.yml - ./config/grafana/dashboards/main.yml:/etc/grafana/provisioning/dashboards/main.yml - ./config/grafana/data/:/var/lib/grafana/ ports: - 3001:3000 networks: - custom_network logging: options: max-size: 50m max-file: 1 environment: - GF_SECURITY_ADMIN_PASSWORD=admin ^ with constant fights with how logging and data that is taken up within the docker image. I've added temp login to help reduce and lower the data. we care about the other database but don't need them as grafan is reported and keeps the log for display. Their is more that need to be done to grafan to maintain and keep accounts. ATM i've decided to keep it admin admin and added Grafana environment stuff to fix. For grafan review: https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/ I've also cleaned it up to use Unraids webui and icons across the community apps Please not the volume for grafana as this has changed. To keep its data, we need to make a change! ^this is more to grab the .db file to save on disk as this is where the dashboard is keeping that data. Chagned: Quote - ./config/grafana/dashboards/netprobe.json:/var/lib/grafana/dashboards/netprobe.json to - ./config/grafana/data/:/var/lib/grafana/ we will need to create and move/copy the data to fix this: There is a quite a few things we need to do to recreate and fix. #echo Path to your git download. cd /mnt/user/appdata/netprobe_lite cd ./config/grafana #echo make save location to keep grafana data mkdir data cd data #echo copy the missing data that we chagned from compsoe: cp /mnt/user/appdata/netprobe_lite/config/grafana/dashboards /mnt/user/appdata/netprobe_lite/config/grafana/data/ #echo Recreate the file paths: mkdir csv mkdir pdf mkdir plugins mkdir png #echo fixing grafa file paths: mkdir alerting cd alerting mkdir 1 cd 1 touch __default__.tmpl cd .. cd .. touch grafana.db #echo seting corect docker permissions: chmod 777 -R * chown nobody:users -R * ^added as zipped contents here We may need to delete previous image data in docekr click advance togle: Finally, with the folder data in we need to run update stack to rebuild and update any dockers... now we have a docker tab web ui controlled netprobe system with icons, web ui and data saved between docker updates and changes *Not sure the full need of the multiple database in these configurations. not sure if redis is being used and have not used promethus db and/or docekr redis db... docker before some research is need before we can re-script this to use MySQL... -But with this setup I can keep my records in Grafana DB and happy where its at. data.zip Edited May 17 by bmartino1 Quote Link to comment
bmartino1 Posted May 21 Share Posted May 21 (edited) I Also need to do similar tweaks and edit to promethus now. thats it. Updated docker compose to faciltate changes. make a directory in git download. lets call it data: mkdir /mnt/user/appdata/netprobe_lite/config/prometheus/data version: "3" networks: custom_network: driver: bridge ipam: config: - subnet: 172.21.0.0/16 services: redis: restart: always container_name: netprobe-redis image: redis:latest labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/A75G/docker-templates/master/templates/icons/redis.png' volumes: - ./config/redis/redis.conf:/etc/redis/redis.conf networks: - custom_network logging: options: max-size: 50m max-file: 1 netprobe: restart: always container_name: netprobe-probe build: . volumes: - .:/netprobe_lite environment: MODULE: NETPROBE labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/simonjenny/fastcom-mysql/master/fastlogo.jpg' networks: - custom_network logging: options: max-size: 50m max-file: 1 presentation: restart: always container_name: netprobe-presentation build: . volumes: - .:/netprobe_lite environment: MODULE: PRESENTATION labels: - net.unraid.docker.icon='https://github.com/chvb/docker-templates/raw/master/chvb/img/OnlyOfficeDocumentServer.jpg' networks: - custom_network logging: options: max-size: 50m max-file: 1 prometheus: restart: always container_name: netprobe-prometheus image: prom/prometheus labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/selfhosters/unRAID-CA-templates/master/templates/img/prometheus.png' volumes: - ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - ./config/prometheus/data/:/prometheus/ networks: - custom_network logging: options: max-size: 50m max-file: 1 grafana: restart: always image: grafana/grafana-enterprise:latest labels: - net.unraid.docker.webui=http://[IP]:[PORT:3001] - net.unraid.docker.icon='https://github.com/atribe/unRAID-docker/raw/master/icons/grafana.png' container_name: netprobe-grafana volumes: - ./config/grafana/datasources/automatic.yml:/etc/grafana/provisioning/datasources/automatic.yml - ./config/grafana/dashboards/main.yml:/etc/grafana/provisioning/dashboards/main.yml - ./config/grafana/data/:/var/lib/grafana/ ports: - 3001:3000 networks: - custom_network logging: options: max-size: 50m max-file: 1 environment: - GF_SECURITY_ADMIN_USER=admin - GF_SECURITY_ADMIN_PASSWORD=admin Forum acting weird for me today... docker compose is yaml syntax watch your spacing. Now when a docker updates and changes images, we keep previous collected data. You will need to delete the Prometheus docker image to recreate that database data in the data folder on unraid. set a admin user and password for grafana to not lose accounts Edited May 21 by bmartino1 forum issues Quote Link to comment
bmartino1 Posted May 21 Share Posted May 21 (edited) since this is a compilation of multiple configs, dockers and systems. A community dev could make a docker image like ubuntu run a docker inside docker with a compose script similar to the above and template with the environment fixes for multiple data paths. I don't see this being on the Community app as it is a compilation of multiple dockers(hard to get gnpu license ok from multiple sources to use...) Definitely an interesting project I don't see it as too bad of an issue with this in the docker tab: Per docker compose plugin unless you known for sure that the docker needs an update. Their is a known bug with how unriad detects and says the docker needs an update: You will not be able to update the docker via the update all or click apply update error: Per Primeval_god in compose forum the correct way to do this is update stack at bottom right of compose. I also have a user script to assist me: #!/bin/bash #docker login -u %username% --password %passwordviadocker% #https://docs.docker.com/reference/cli/docker/login/ docker pull redis:latest docker pull prom/prometheus docker pull grafana/grafana-enterprise:latest Presentation and probe are linux docker created via the compose script and data from github to install python and run the commands to collect data via the databases. Edited May 21 by bmartino1 Quote Link to comment
bmartino1 Posted May 25 Share Posted May 25 (edited) looks like netprobe added another docker and option in grafana... speed tests.. which would need antoerh docker... speedtest: restart: always container_name: netprobe-speedtest image: "plaintextpackets/netprobe:latest" pull_policy: always volumes: - .:/netprobe_lite environment: MODULE: "SPEEDTEST" networks: - netprobe-net dns: - 8.8.8.8 - 8.8.4.4 Copy the data folder for Prometheus/Grafana in a new folder for backup - carry over. ^it's bet to delete the data folders and make new, As changes may double up on garages due to existing data and changes in the env file. Can be tricky to go in and delete the bad data in Prometheus... Looks like speed test is new, and the netprobe group are trying to fix their compose file... I had to delete the stack, delete the netprobe folder. Git clone again. Make my 2 data folder in the config folder for grafana/Prometheus that i make and cahnge. Eidt the env file and edit the docker-compose file... From what I can tell they are trying to incoperate a speediest docker via python... I would normally not include this as we have this via speedtest cli and other dockers options... I don't see the need for the networks dns options... docker networking should be handling that correctly via /etc/rsolve.conf... I added logging as to not fill unraids temp Docker image and logs... I added the docker pull policy, so the user script is not needed. I recommend it though as it uses docker login to guarantee a connection to the docker hub especial for dockers that don't have a "existing" image and are not configured via the code. the label options in this compose file for unraid only and are used for unraids web UI to have one click solutions... Here is my updated compose to incoperate there docker. I have enabled the speediest true int eh env file. Have yet to see data in Grafana yet. But here is the up-to-date unraid compose file for this: This is what i run. I can ## and comment out speedtest docker. I most likely will but wanted to give unraid docker compose users the updated based on the projects evolutions. Seeing that it has been forked many times... version: "3" networks: custom_network: driver: bridge ipam: config: - subnet: 172.21.0.0/16 services: redis: pull_policy: always restart: always container_name: netprobe-redis image: redis:latest labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/A75G/docker-templates/master/templates/icons/redis.png' volumes: - ./config/redis/redis.conf:/etc/redis/redis.conf networks: - custom_network logging: options: max-size: 50m max-file: 1 netprobe: pull_policy: always restart: always container_name: netprobe-probe build: . volumes: - .:/netprobe_lite environment: MODULE: NETPROBE labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/simonjenny/fastcom-mysql/master/fastlogo.jpg' networks: - custom_network logging: options: max-size: 50m max-file: 1 presentation: pull_policy: always restart: always container_name: netprobe-presentation build: . volumes: - .:/netprobe_lite environment: MODULE: PRESENTATION labels: - net.unraid.docker.icon='https://github.com/chvb/docker-templates/raw/master/chvb/img/OnlyOfficeDocumentServer.jpg' networks: - custom_network logging: options: max-size: 50m max-file: 1 prometheus: pull_policy: always restart: always container_name: netprobe-prometheus image: prom/prometheus labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/selfhosters/unRAID-CA-templates/master/templates/img/prometheus.png' volumes: - ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - ./config/prometheus/data/:/prometheus/ networks: - custom_network logging: options: max-size: 50m max-file: 1 grafana: pull_policy: always restart: always image: grafana/grafana-enterprise:latest labels: - net.unraid.docker.webui=http://[IP]:[PORT:3001] - net.unraid.docker.icon='https://github.com/atribe/unRAID-docker/raw/master/icons/grafana.png' container_name: netprobe-grafana volumes: - ./config/grafana/datasources/automatic.yml:/etc/grafana/provisioning/datasources/automatic.yml - ./config/grafana/dashboards/main.yml:/etc/grafana/provisioning/dashboards/main.yml - ./config/grafana/data/:/var/lib/grafana/ ports: - 3001:3000 networks: - custom_network logging: options: max-size: 50m max-file: 1 environment: - GF_SECURITY_ADMIN_USER=admin - GF_SECURITY_ADMIN_PASSWORD=admin speedtest: restart: always container_name: netprobe-speedtest image: "plaintextpackets/netprobe:latest" pull_policy: always labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/selfhosters/unRAID-CA-templates/master/templates/img/speedtest-tracker.png' volumes: - .:/netprobe_lite environment: MODULE: "SPEEDTEST" networks: - custom_network logging: options: max-size: 50m max-file: 1 ^change grafana user and passowrd at docker level. Default admin/admin new env file: # CUSTOM VARIABLES # Notes: # 1. Do not change any variable names # 2. Read instructions carefully # Site target list # - If modifying this list, make sure you limit to 5 websites and use the domain name as shown here #SITES="google.com,facebook.com,twitter.com,youtube.com,amazon.com" SITES="fast.com,google.com,youtube.com,amazon.com,uillinois.edu" # DNS test site # - This is the site which is resolved in DNS to test DNS servers, if modified only use one domain name DNS_TEST_SITE="uillinois.edu" # This is the site used in the DNS test # DNS name servers # - This is the list of DNS servers which are tested # - Netprobe only supports 4 DNS servers, you can change the value of "DNS_NAMESERVER_4_IP" to test your own home DNS server # - Note: do not change the value of "DNS_NAMESERVER_4" DNS_NAMESERVER_1="Google_DNS" DNS_NAMESERVER_1_IP="8.8.8.8" DNS_NAMESERVER_2="Quad9_DNS" DNS_NAMESERVER_2_IP="9.9.9.9" DNS_NAMESERVER_3="CloudFlare_DNS" DNS_NAMESERVER_3_IP="1.1.1.1" DNS_NAMESERVER_4="My_DNS_Server" # Do not change this line at all! DNS_NAMESERVER_4_IP="192.168.2.10" # Replace this IP with the DNS server you use at home # Health Score Weights # - These are the relative weights used to calculate your 'Internet Quality Score', they can be modified but must add up to 1.0 weight_loss = ".6" # Loss is 60% of score weight_latency = ".15" # Latency is 15% of score weight_jitter = ".2" # Jitter is 20% of score weight_dns_latency = "0.05" # DNS latency is 0.05 of score # Health Score Thresholds # - These threshold values are used in the calculation of your 'Internet Quality Score', they can be modified if required threshold_loss = "5" # 5% loss threshold as max threshold_latency = "100" # 100ms latency threshold as max threshold_jitter = "30" # 30ms jitter threshold as max threshold_dns_latency = "100" # 100ms dns latency threshold as max # Speetest configuration (be very careful when running on a metered connection!) # - This configuration is for setting up a "speed test" or rather a test of your internet bandwidth. # - In order to test your upload and download bandwidth we use speedtest.net as source. So your client will connect there and upload and download some data. # - That also means that a random server is selected for the test (usually the nearest one) # - Setting the SPEEDTEST_INTERVAL too agressively will cause speedtest.net to block your requests, recommend 15 minutes (900 seconds) and above SPEEDTEST_ENABLED="True" # set this to "True" to enable speed test function SPEEDTEST_INTERVAL="3600" # interval on which the speedtest will run, in seconds - note using a prime number helps reduce the number of collisions between netprobe and speed tests # SYSTEM VARIABLES - DO NOT TOUCH PRESENTATION_PORT = "5000" PRESENTATION_INTERFACE = "0.0.0.0" REDIS_URL = "netprobe-redis" REDIS_PORT = "6379" REDIS_PASSWORD = "password" PROBE_INTERVAL="30" PROBE_COUNT="50" ^- I use pihole... so a custom dns.. I edited some sites... enabled speedtest. Set speed test to 1 hour. Will wait 24 to see if changes worked. but will most likely remove speediest for other tools. ^-I run firewall and some / most of the tools in this are baked into the phone app / firewall runs similar to get this data... https://firewalla.com/products/firewalla-purple-se Seeing as i'm the only one commenting here.... and the nature of this docker is a bit all over and starting to become more complex and advanced setup is needed to fix... this will be my last update to net probe on this forum.. Thanks again for sharing Arbadacarba I was aware of tools and dockers before but this had a all in 1 place look and glance. i would use it more for 30 days go to Grafana option i'm concerned about inspect data: and download a Excel of that data from the database. the graph is perfect, but sometimes I needs raw text data... and the compose edit to Grafana are needed to save chagnes to the dashboard... My "code changes" are more unraid fixes to not lose data collected and to be able to update without fear of lossing that data... issues that should ahve been though of during its creation to fix then not after the fact... since the netprobe gorp is using other dockers alot are just them needing to review the dockers they are using. Example due to their DB implementation grafana every once in a while gets database lock responses: logger=sqlstore.transactions t=2024-05-25T21:56:32.491917786Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" Then there's the 2 Linux docker to run python and corn scripts to go to the databases.. I'm still a noob to redis very little knowledge on it from nextcloud... And as far as I can tell redis is not actually in use or not configured... (no config...): I'm also happy that whatever code is running is working and that it's not malicious.. I'm not invested Enough to go through their python and fix simple/complex error due to how they misinterpret the data.. But all in all a great 3rd party tool There script is ok but its fairly faulty: I don't want to rewrite the codes as I already have other tools and these post were more scattered documentation for other friends to implement and use. now that its implemented, this is dead and I only care about the docker image updates to maintain CVE... Hope this helps other unraid users to use this project Edited May 26 by bmartino1 spelling Quote Link to comment
Arbadacarba Posted May 29 Author Share Posted May 29 I don't know why but I havenet been getting alerts that you have been working in here... This is excellent, I'm in the process of working through what you have done here and installing it. 1 Quote Link to comment
Solution bmartino1 Posted May 29 Solution Share Posted May 29 (edited) On 5/29/2024 at 12:49 PM, Arbadacarba said: I don't know why but I havenet been getting alerts that you have been working in here... This is excellent, I'm in the process of working through what you have done here and installing it. Welcome back. I will try to rewrite the installation procedures to reproduce. There's a lot and i used this as notes/sound board. Assumptions: I assume you already have a data spot in mind and the compose plugin installed. I chose /mnt/user/appdata Step 1 open terminal and go to the installation path of your choosing. I chose to keep this together with my docker data. Unraid terminal script setup: run one line at a time (./config is the config folder where your dat path is) #It is best to use full paths. ./config/ in my case is /mnt/user/appdata/netprobe_lite/config #Download GitHub project and files/scripts cd /mnt/user/appdata git clone https://github.com/plaintextpackets/netprobe_lite.git #enter project to make changes cd netprobe_lite/ #step 1 prepare for unraid UI rename existing compose file cp compose.yml compose.yml.org rm compose.yml #step 2 prepare for better docker Data path to keep between docker updates. #First Grafana: cd ./config/grafana #echo make save location to keep Grafana data mkdir data cd data #echo copy the dashboard to correct spot -- missing data that we changed in compose(I use MC to copy the folder): cp -R ./config/grafana/dashboards ./config/grafana/data/ #echo Recreate the empty folder paths Grafana needed for latter use inside the “./config/grafana/data/” (issue in past with docker not making folder needed Docker should make this at first run): cd data mkdir csv mkdir pdf mkdir plugins mkdir png #echo fixing grafana files and paths(grafana entprise config) docker should make these at first run this should be in “./config/grafana/data/”: mkdir alerting cd alerting mkdir 1 cd 1 touch __default__.tmpl cd .. cd .. touch grafana.db #Next Promethus - The actual data that needs saved to keep between dockers updates and deletion! “./config/prometheus" makin its "/data/” location #Drop back to config folder “./config/”: cd .. cd .. cd prometheus/ mkdir data #Fix permission – set unraid docker safe perms….: #Drop back to config folder: cd .. #setting unraid docker safe permissions to config folder only!: #echo setting correct docker permissions “./config/”: chmod 777 -R * chown nobody:users -R * Step 2 setup docker compose: our data exist: at /mnt/user/appdata/netprobe_lite/ This will make the compose file we use in unraid and import the .env file used to configure and control netprobe. Next we want to edit the stack: Step 3 env file first: Github default env file: https://github.com/plaintextpackets/netprobe_lite/blob/master/.env we want to make a few changes: DOUBLE CHECK YOUR .ENV FILE FOR IP and EDITS YOU WANT. If you start this docker and then edit this Grafana may display weird by picking up the data in the Prometheus database. You will need to delete the contents of the “./config/prometheus/data this will erase all collected data and use the env file config. Quote # CUSTOM VARIABLES # Notes: # 1. Do not change any variable names # 2. Read instructions carefully # Site target list # - If modifying this list, make sure you limit to 5 websites and use the domain name as shown here #SITES="google.com,facebook.com,twitter.com,youtube.com,amazon.com" SITES="fast.com,google.com,youtube.com,amazon.com,uillinois.edu" # DNS test site # - This is the site which is resolved in DNS to test DNS servers, if modified only use one domain name DNS_TEST_SITE="uillinois.edu" # This is the site used in the DNS test # DNS name servers # - This is the list of DNS servers which are tested # - Netprobe only supports 4 DNS servers, you can change the value of "DNS_NAMESERVER_4_IP" to test your own home DNS server # - Note: do not change the value of "DNS_NAMESERVER_4" DNS_NAMESERVER_1="Google_DNS" DNS_NAMESERVER_1_IP="8.8.8.8" DNS_NAMESERVER_2="Quad9_DNS" DNS_NAMESERVER_2_IP="9.9.9.9" DNS_NAMESERVER_3="CloudFlare_DNS" DNS_NAMESERVER_3_IP="1.1.1.1" DNS_NAMESERVER_4="My_DNS_Server" # Do not change this line at all! DNS_NAMESERVER_4_IP="192.168.2.10" # Replace this IP with the DNS server you use at home # Health Score Weights # - These are the relative weights used to calculate your 'Internet Quality Score', they can be modified but must add up to 1.0 weight_loss = ".6" # Loss is 60% of score weight_latency = ".15" # Latency is 15% of score weight_jitter = ".2" # Jitter is 20% of score weight_dns_latency = "0.05" # DNS latency is 0.05 of score # Health Score Thresholds # - These threshold values are used in the calculation of your 'Internet Quality Score', they can be modified if required threshold_loss = "5" # 5% loss threshold as max threshold_latency = "100" # 100ms latency threshold as max threshold_jitter = "30" # 30ms jitter threshold as max threshold_dns_latency = "100" # 100ms dns latency threshold as max # Speetest configuration (be very careful when running on a metered connection!) # - This configuration is for setting up a "speed test" or rather a test of your internet bandwidth. # - In order to test your upload and download bandwidth we use speedtest.net as source. So your client will connect there and upload and download some data. # - That also means that a random server is selected for the test (usually the nearest one) # - Setting the SPEEDTEST_INTERVAL too agressively will cause speedtest.net to block your requests, recommend 15 minutes (900 seconds) and above SPEEDTEST_ENABLED="True" # set this to "True" to enable speed test function SPEEDTEST_INTERVAL="3600" # interval on which the speedtest will run, in seconds - note using a prime number helps reduce the number of collisions between netprobe and speed tests # SYSTEM VARIABLES - DO NOT TOUCH PRESENTATION_PORT = "5000" PRESENTATION_INTERFACE = "0.0.0.0" REDIS_URL = "netprobe-redis" REDIS_PORT = "6379" REDIS_PASSWORD = "password" PROBE_INTERVAL="30" PROBE_COUNT="50" ^Here we enabled speedtest it runs every hour. I then set a custom dns IP(I run Pihole) if you don't run a in house dns server use opendns 208.67.222.222 it will be labled as My_DNS_Server *^ Do NOT! Change the name the grafana main and other dashboard file call it weirdly to display correctly. I run piehole docker, a custom DNS, and it currently set to that IP. Step 4: Then edit the docker compose file with the following edits: Quote version: "3" networks: custom_network: driver: bridge ipam: config: - subnet: 172.21.0.0/16 services: redis: pull_policy: always restart: always container_name: netprobe-redis image: redis:latest labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/A75G/docker-templates/master/templates/icons/redis.png' volumes: - ./config/redis/redis.conf:/etc/redis/redis.conf networks: - custom_network logging: options: max-size: 50m max-file: 1 netprobe: pull_policy: always restart: always container_name: netprobe-probe build: . volumes: - .:/netprobe_lite environment: MODULE: NETPROBE labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/simonjenny/fastcom-mysql/master/fastlogo.jpg' networks: - custom_network logging: options: max-size: 50m max-file: 1 presentation: pull_policy: always restart: always container_name: netprobe-presentation build: . volumes: - .:/netprobe_lite environment: MODULE: PRESENTATION labels: - net.unraid.docker.icon='https://github.com/chvb/docker-templates/raw/master/chvb/img/OnlyOfficeDocumentServer.jpg' networks: - custom_network logging: options: max-size: 50m max-file: 1 prometheus: pull_policy: always restart: always container_name: netprobe-prometheus image: prom/prometheus labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/selfhosters/unRAID-CA-templates/master/templates/img/prometheus.png' volumes: - ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - ./config/prometheus/data/:/prometheus/ networks: - custom_network logging: options: max-size: 50m max-file: 1 grafana: pull_policy: always restart: always image: grafana/grafana-enterprise:latest labels: - net.unraid.docker.webui=http://[IP]:[PORT:3001] - net.unraid.docker.icon='https://github.com/atribe/unRAID-docker/raw/master/icons/grafana.png' container_name: netprobe-grafana volumes: - ./config/grafana/datasources/automatic.yml:/etc/grafana/provisioning/datasources/automatic.yml - ./config/grafana/dashboards/main.yml:/etc/grafana/provisioning/dashboards/main.yml - ./config/grafana/data/:/var/lib/grafana/ ports: - 3001:3000 networks: - custom_network logging: options: max-size: 50m max-file: 1 environment: - GF_SECURITY_ADMIN_USER=admin - GF_SECURITY_ADMIN_PASSWORD=admin speedtest: restart: always container_name: netprobe-speedtest image: "plaintextpackets/netprobe:latest" pull_policy: always labels: - net.unraid.docker.icon='https://raw.githubusercontent.com/selfhosters/unRAID-CA-templates/master/templates/img/speedtest-tracker.png' volumes: - .:/netprobe_lite environment: MODULE: "SPEEDTEST" networks: - custom_network logging: options: max-size: 50m max-file: 1 ^Here is my working unraid Compose file that keeps data. Note the Compose items for lables, theses http links and data go in the UI lables when prompted. This adds the docker image to the docker tab. You can set an admin account at this time by editing the grafana section option of ... environment: - GF_SECURITY_ADMIN_USER=admin - GF_SECURITY_ADMIN_PASSWORD=admin Default admin/admin Then set the UI labels (compose no longer fills in the data.) ^In the UI lables copy from the compose file Edited June 23 by bmartino1 finalizing data. script at top is guidelines 2 Quote Link to comment
Arbadacarba Posted May 29 Author Share Posted May 29 Data location: mnt/systems/system/appdata That's where all my dockers are, its a separate pool on an SSD Array in step 2 after: #step 2 prepare for better docker Data path to keep between docker updates. #First Grafana: cd ./config/grafana #echo make save location to keep grafana data mkdir data cd data #echo copy the dashboard to correct spot -- missing data that we chagned in compsoe: cp ./config/grafana/dashboards ./config/grafana/data/ i get: cp: cannot stat './config/grafana/dashboards': No such file or directory Quote Link to comment
bmartino1 Posted May 29 Share Posted May 29 (edited) 3 minutes ago, Arbadacarba said: Data location: mnt/systems/system/appdata That's where all my dockers are, its a separate pool on an SSD Array in step 2 after: #step 2 prepare for better docker Data path to keep between docker updates. #First Grafana: cd ./config/grafana #echo make save location to keep grafana data mkdir data cd data #echo copy the dashboard to correct spot -- missing data that we chagned in compsoe: cp ./config/grafana/dashboards ./config/grafana/data/ i get: cp: cannot stat './config/grafana/dashboards': No such file or directory you want to copy the dashbaord folder into the data folder: Edited May 29 by bmartino1 Quote Link to comment
Arbadacarba Posted May 29 Author Share Posted May 29 cp -R /mnt/systems/system/appdata/netprobe_lite/config/grafana/dashboards /mnt/systems/system/appdata/netprobe_lite/config/grafana/data/ Worked 1 Quote Link to comment
Arbadacarba Posted May 29 Author Share Posted May 29 #echo Recreate the empty folder paths nedded for latter(issue in past with docker not making folder needed): cd data mkdir csv mkdir pdf mkdir plugins mkdir png Where am I starting here? I think I am already in the data folder Quote Link to comment
bmartino1 Posted May 29 Share Posted May 29 2 minutes ago, Arbadacarba said: cp -R /mnt/systems/system/appdata/netprobe_lite/config/grafana/dashboards /mnt/systems/system/appdata/netprobe_lite/config/grafana/data/ Worked if its a fresh install the docker will recreate the files and folder necessary. the dashboard folder as the file to make the Netprobe display in graffana (I have had issues when dockers new and existing have problems with folder/files path missing and made empty/ temp files for use when it loads. You need a data folder in Prometheus and graffana. Then copy the dashboard in graffan config into data to complte full docker file pathing. as i changed the config to save Grafana database and other edits to maintin when grafan get docker updates. the 2 folders that need to be made for docker compose is the data folder in Grafana and Prometheus. Quote Link to comment
bmartino1 Posted May 29 Share Posted May 29 1 minute ago, Arbadacarba said: #echo Recreate the empty folder paths nedded for latter(issue in past with docker not making folder needed): cd data mkdir csv mkdir pdf mkdir plugins mkdir png Where am I starting here? I think I am already in the data folder correct this needs to be in the data folder these are the emapy files path that grafan should created. Quote Link to comment
Arbadacarba Posted May 29 Author Share Posted May 29 OK, so I've made it to the editing the env 1 Quote Link to comment
Arbadacarba Posted May 29 Author Share Posted May 29 You mention changes that needed to be made to the env and compose files... But I don't see them. Did you just mean the ones pasted in to previous messages? Thanks so much for your work here. Quote Link to comment
bmartino1 Posted May 29 Share Posted May 29 (edited) 38 minutes ago, Arbadacarba said: You mention changes that needed to be made to the env and compose files... But I don't see them. Did you just mean the ones pasted in to previous messages? Thanks so much for your work here. I quoted them again they should be the ones in the post marked as solution. There was a bit that I changed from the original env. Set different site I would normally go to for ping. Quote #SITES="google.com,facebook.com,twitter.com,youtube.com,amazon.com" SITES="fast.com,google.com,youtube.com,amazon.com,uillinois.edu" Set a university in my area for a dns name check Quote # - This is the site which is resolved in DNS to test DNS servers, if modified only use one domain name DNS_TEST_SITE="uillinois.edu" # This is the site used in the DNS test Set a custom dns in house server to compare to in house v public. (pihole uses fastest in return) Quote DNS_NAMESERVER_4_IP="192.168.2.10" # Replace this IP with the DNS server you use at home Set speed test to enable: Quote # - Setting the SPEEDTEST_INTERVAL too agressively will cause speedtest.net to block your requests, recommend 15 minutes (900 seconds) and above SPEEDTEST_ENABLED="True" # set this to "True" to enable speed test function SPEEDTEST_INTERVAL="3600" # interval on which the speedtest will run, in seconds - note using a prime number helps reduce the number of collisions between netprobe and speed tests Then Set speed test to run 1 every hour. Docker compose was quite a bit from the original. Added labeling for unraid icons and web ui added logging: more to keep logs small form other dockers *-When I first downloaded and used their compose file to make my own, there was no dns option -Fixed docker network for unraid creation with compose up from stack. Changed the dashboard to inc operate Grafana environment variables and save the DB set docker execution policy to download and update the docker each time they are started. -added admin account creation to docker compose. (when Grafana is taken down this reset accounts) another variable can be added to move the conf file to physical data disk and used similar to the data folder... Most of Grafana needed variables from official dock: https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/ For this to be as a CA it would need to be used as 1 docker. it may be a Linux image that runs docker in docker. Then trying to come up with multi path between the 2 and that wasn't worth the fight. Edited May 29 by bmartino1 Spelling 1 Quote Link to comment
Arbadacarba Posted May 30 Author Share Posted May 30 I missed the Edit. Thanks again Quote Link to comment
hbilling Posted June 2 Share Posted June 2 Hi bmartino1, Thank you so much for your detailed posts. I have been trying to work through this but couldn't get things to work properly. But have been through again and your post marked as the solution got me a lot further. Thank you for your initial posts and coming back with more info too I have now successfully been able to get all the apps to start in docker, and been able to log into Grafana. However I am missing the dashboard with all the info that I was expecting. Clearly I have missed something. Or is there some config I need to do once I log into grafana? watching the YouTube video - the dashboard is just there... I did also got stuck on the ./config thing but managed to work it out (thought later I realised I should have kept reading lol) I just used your post with the env file details and docker compose files. Only changing the last dns setting. Short of deleting it all and starting again, did you have any ideas? Thanks heaps for your work on this! Been a project I've been wanting to get on unraid for some time now! Kept checking the CA but i now see its too complex to be there. Quote Link to comment
bmartino1 Posted June 2 Share Posted June 2 (edited) 3 hours ago, hbilling said: Hi bmartino1, Thank you so much for your detailed posts. I have been trying to work through this but couldn't get things to work properly. But have been through again and your post marked as the solution got me a lot further. Thank you for your initial posts and coming back with more info too I have now successfully been able to get all the apps to start in docker, and been able to log into Grafana. However I am missing the dashboard with all the info that I was expecting. Clearly I have missed something. Or is there some config I need to do once I log into grafana? watching the YouTube video - the dashboard is just there... I did also got stuck on the ./config thing but managed to work it out (thought later I realised I should have kept reading lol) I just used your post with the env file details and docker compose files. Only changing the last dns setting. Short of deleting it all and starting again, did you have any ideas? Thanks heaps for your work on this! Been a project I've been wanting to get on unraid for some time now! Kept checking the CA but i now see its too complex to be there. I moved the dashboard to make grafana keep its data. you need to make the data folder and copy the dashbaord fodler into the data folder. run theses camoands 1 at atime folowing the above notes: #Download github project and files/scripts cd /mnt/user/appdata git clone https://github.com/plaintextpackets/netprobe_lite.git #enter project to make changes cd netprobe_lite/ #step 1 prepare for unraid UI rename existing compse file cp compose.yml compose.yml.org rm compose.yml #step 2 prepare for better docker Data path to keep between docker updates. #First Grafana: cd ./config/grafana #echo make save location to keep grafana data mkdir data cd data #echo copy the dashboard to correct spot -- missing data that we chagned in compsoe: cp -R ./config/grafana/dashboards ./config/grafana/data/ #^copy the netprgbe grafan dashboard into the data folder ^ #echo Recreate the empty folder paths nedded for latter(issue in past with docker not making folder needed): cd data mkdir csv mkdir pdf mkdir plugins mkdir png #echo fixing grafana file paths(grafana entprise config): mkdir alerting cd alerting mkdir 1 cd 1 touch __default__.tmpl cd .. cd .. touch grafana.db #Next Promethus - The actualy data that needs saved! #Drop back to config folder: cd .. cd .. cd prometheus/ mkdir data #Fix permission: #Drop back to config folder: cd .. #setting unraid docker safe permissions to config folder only!: #echo seting corect docker permissions: chmod 777 -R * chown nobody:users -R * seen more here: copy the netprbe grafan dash board folder (2 files) into graffan data dashboard folder. Edited June 2 by bmartino1 Quote Link to comment
hbilling Posted June 2 Share Posted June 2 Thanks for the quick reply, tried again and all up this time. I clearly didn't get a step right copying the dashboard directory. Cheers! 😁 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.