[Plugin] Prometheus unRAID Plugins


Recommended Posts

2 hours ago, unw1red said:

I am not running keepalived. They are individual instances of PiHole with both designated in my DHCP scope.

You can basically put the start command in the go file as long as the instance is already up when you start unRAID.

 

What you can also do is that you write a custom cron script or a script that runs on boot like this:

#!/bin/bash
echo "prometheus_pihole_exporter \
  -pihole_api_token YOURTOKEN \
  -pihole_hostname IPFROMTHEPIHOLEINSTANCE \
  -pihole_protocol http \
  -pihole_port WEBUIPORTFROMPIHOLEINSTANCE \
  -port PORTWHERETHEEXPORTERRUNSON \
  -interval 10s" | at now

YOURTOKEN= The token from the other PiHole instance

IPFROMTHEPIHOLEINSTANCE= The IP from the second PiHole instance

WEGUIPORTFROMPIHOLOEINSTANCE= The port for the WebGUI - usually 80

PORTWHERETHEEXPORTERRUNSON= Something like 9618 - important that it is a different port than the plugin runs on (9617) and you have to insert this as a new target to your prometheus.yml

 

Hope this makes sense, please keep in mind that I don't designed the plugin supports more than one instance at a time but with this in the go script or a cron job at start from unRAID this will totally work.

(You can also put 'sleep 30s' after the line '#!/bin/bash' so that the execution will 30 seconds delayed after the script is called)

 

I never tried this but it should work fine too. :)

 

Note: If you put this in the go file you don't need the line '#!/bin/bash'

Link to comment
On 7/4/2021 at 11:52 PM, ich777 said:

Do you run your containers in a custom network on br0?

If yes you need to enable Allow Host Access in the Docker settings.

 

I do, and it is enabled.  I even tried toggling it on and off and rebooting my server and still getting that same 404 server down error

Link to comment
1 hour ago, boosting1bar said:

 

I do, and it is enabled.  I even tried toggling it on and off and rebooting my server and still getting that same 404 server down error

Then there has to be something else wrong.

 

Try to set the network to bridge four troubleshooting reasons and look if you get the same error please.

 

A 404 error tells you that Prometheus can't connect to the service.

 

Can you reach YOURUNRAIDIP:9100/metrics from your browser?

 

Please make sure that your IP in the yml file matches your unRAID IP.

Link to comment
Posted (edited)
3 hours ago, ich777 said:

Then there has to be something else wrong.

 

Try to set the network to bridge four troubleshooting reasons and look if you get the same error please.

 

A 404 error tells you that Prometheus can't connect to the service.

 

Can you reach YOURUNRAIDIP:9100/metrics from your browser?

 

Please make sure that your IP in the yml file matches your unRAID IP.

I've tried it on bridge and host and it kicks me the same error so far and shows localhost up and ip:9100 404 down.

 

If I try to reach unraidIP:9100/metrics I just get a blank white page with text that says Not Found at the top, if I take off /metrics it just gives me a blank white page so it's getting some response and not just a can't load this page error.

 

And I've checked the IP in the yml and it's correct.  I even tried adding http:// but that breaks Prometheus it looks like.  I don't have any errors in the Prometheus log for the container.

Edited by boosting1bar
Link to comment
1 hour ago, boosting1bar said:

If I try to reach unraidIP:9100/metrics I just get a blank white page with text that says Not Found at the top, if I take off /metrics it just gives me a blank white page so it's getting some response and not just a can't load this page error.

Are you sure that the Prometheus Node Exporter is installed and not in the Plugin Error tab on the plugin page?

 

Your problem is that the exporter doesn't seem to run.

 

Try to uninstall and reinstall the plugin again.

Also when you install the plugin don't close the window with the red X on top, click Done!

 

Eventually also try to reboot your server if that all doesn't help.

Link to comment
11 hours ago, ich777 said:

Are you sure that the Prometheus Node Exporter is installed and not in the Plugin Error tab on the plugin page?

 

Your problem is that the exporter doesn't seem to run.

 

Try to uninstall and reinstall the plugin again.

Also when you install the plugin don't close the window with the red X on top, click Done!

 

Eventually also try to reboot your server if that all doesn't help.

Yep just went through and uninstalled and reinstalled exporter again and getting the same error and "Not found" /metrics page.  I've reboot again as well and still no change.  I really appreciate your help but I'm not going to keep chewing up your time trying to track this down, I'm just going to delete and not junk up the thread!

Link to comment
48 minutes ago, boosting1bar said:

Yep just went through and uninstalled and reinstalled exporter again and getting the same error and "Not found" /metrics page.  I've reboot again as well and still no change.  I really appreciate your help but I'm not going to keep chewing up your time trying to track this down, I'm just going to delete and not junk up the thread!

Nothing to worry about, I think this issue can be solved pretty easily. Can you issue the command: 'pidof prometheus_node_exporter' (without quotes) from an unRAID terminal and post the output here (with the plugin installed)?

Link to comment

Awesome info you've shared, thank you so much for taking the time to do so!!

 

Could you possibly cover how to setup alerts?

 

I'm setup with this and is working great, I just can't quite get my head around setting alerts, especially with things that are true/false...

 

For example - I have a dashboard for my router, and want to set alerts to notify of VPN or WAN connections going up or down.

Link to comment

Hi ich777, thanks for your hard work!

 

I tried your guide, and setup Prometheus and Prometheus node exporter without any issues. 

 

However, when i setup the pihole plugin, and enter my pihole ip and port 80 and api token, and click change and start, it runs for a while and stops automatically.

 

I get the following error:

 

2021/07/11 01:30:52 Starting HTTP server
2021/07/11 01:31:02 An error has occured during retrieving PI-Hole statistics Get "http://192.168.1.46:281/admin/api.php?summaryRaw&overTimeData&topItems&recentItems&getQueryTypes&getForwardDestinations&getQuerySources&jsonForceObject&auth=*****************************************": dial tcp 192.168.1.46:281: connect: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x811daa]

goroutine 37 [running]:
github.com/eko/pihole-exporter/internal/pihole.(*Client).getStatistics(0xc0001663f0, 0xc000042790)
/root/prometheus_pihole_exporter/pihole-exporter/internal/pihole/client.go:163 +0x2ea
github.com/eko/pihole-exporter/internal/pihole.(*Client).Scrape(0xc0001663f0)
/root/prometheus_pihole_exporter/pihole-exporter/internal/pihole/client.go:61 +0x88
created by main.initPiHoleClient
/root/prometheus_pihole_exporter/pihole-exporter/main.go:37 +0xc5

 

192.168.1.46 is my pihole ip and using port 80

 

please help, im a newbie to linux and have no idea what these is all about.

 

Thanks again for all your hard work.

 

Edit: NVM, it just worked after i tried again. Maybe i entered the wrong port by mistake.

Again, thanks for your hard work!

 

Edit 2: some of my dockers stoped working after enabling host access to custom networks, disabled this and they works again. any idea how to fix this?

Edited by kjames2001
Link to comment
6 hours ago, kjames2001 said:

Edit: NVM, it just worked after i tried again. Maybe i entered the wrong port by mistake.

Again, thanks for your hard work!

Have you CA Backup installed? Keep in mind, everytime you restart PiHole or strictly speaking stop it, the exporter will also stop.

 

If you want to workaround this you have to change the setting for PiHole in CA Backup so that it don't stops (how to do it is in the tutorial).

 

14 hours ago, jack0w said:

Could you possibly cover how to setup alerts?

I will look into this and add this to the second post, but give me some time, have a lot of projects going on currently.

  • Thanks 1
Link to comment
Just now, ich777 said:

Have you CA Backup installed? Keep in mind, everytime you restart PiHole or strictly speaking stop it, the exporter will also stop.

 

If you want to workaround this you have to change the setting for PiHole in CA Backup so that it don't stops (how to do it is in the tutorial).

i have and have set it up like in the tutorial, and it works. but now some of my dockers stoped working after enabling host access to custom networks, disabled this and they works again. any idea how to fix this?

Link to comment
4 minutes ago, kjames2001 said:

i have and have set it up like in the tutorial, and it works. but now some of my dockers stoped working after enabling host access to custom networks, disabled this and they works again. any idea how to fix this?

This should not be the case, if you enable host access all the containers in br0 can "speak" to unRAID and they should still work fine and I can think of nothing why they shouldn't work anymore with host access enabled.

The other way around, if you disable host access the containers an br0 can't talk to unRAID...

 

Are you trying to get PiHole running?

If so you can also get PiHole running on Host mod with the IP from unRAID, but please be aware that you need to male some tweaks to the Docker template and a few other modifications.

Link to comment
16 minutes ago, ich777 said:

This should not be the case, if you enable host access all the containers in br0 can "speak" to unRAID and they should still work fine and I can think of nothing why they shouldn't work anymore with host access enabled.

The other way around, if you disable host access the containers an br0 can't talk to unRAID...

 

Are you trying to get PiHole running?

If so you can also get PiHole running on Host mod with the IP from unRAID, but please be aware that you need to male some tweaks to the Docker template and a few other modifications.

its cloudflared and zerotier that's not working. will try to change that setting again later, cant stop docker now.

 

Edit: just retried, now everything works just fine. Thanks!

Edited by kjames2001
  • Like 2
Link to comment

When I set up piHole I am getting an error of connection refused in Prometheus and then it is obviously not working in Grafana. Below are my configs, if someone sees something wrong any help would be appreciated. 

Error in Prometheus (4th one is piHole):

image.thumb.png.8af1c3ae2efd0216995334685e395386.png

 

piHole Exporter: 

image.png.2f2e693f182ebb9ebfc97daf9b95684b.png

 

PiHole:

image.thumb.png.7a595aaa4ffa7135518472ddc577821f.png

 

Thanks in advance 

Edited by stelks
Link to comment
1 hour ago, stelks said:

Error in Prometheus (4th one is piHole):

You have set the IP wrong in the prometheus.yml you have to specify the unRAID IP not the PiHole IP.

The PiHole Exporter is a bridge between your PiHole and Prometheus that runs on unRAID.

  • Like 1
Link to comment
2 hours ago, ich777 said:

You have set the IP wrong in the prometheus.yml you have to specify the unRAID IP not the PiHole IP.

The PiHole Exporter is a bridge between your PiHole and Prometheus that runs on unRAID.

Thank you for your help that got it working in Prometheus!(Could of sworn I tried both lol) 

 

Do you have any idea why it would not be pulling the data into Grafana now that that is configured correctly. All of my other dashboards in Grafana that use Prometheus work fine but the piHole one just says no data. I have restarted all of the containers and deleted and re-imported the template.

image.thumb.png.27f3c7ed80f9bb9b670fc89afbfa4dbd.pngimage.thumb.png.462532b96891df7d38fe47bd7f57ad12.png

Link to comment
Just now, stelks said:

Do you have any idea why it would not be pulling the data into Grafana now that that is configured correctly. All of my other dashboards in Grafana that use Prometheus work fine but the piHole one just says no data. I have restarted all of the containers and deleted and re-imported the template.

Please give it a little time to pick up the data and make sure the exporter is running in the plugin page.

PiHole can take a little to update if you configured it for the first time.

  • Thanks 1
Link to comment
1 hour ago, kjames2001 said:

It turns out i still have the same issue: zerotier is not working when i have host access to custom networks enabled.

This is really strange, is a reconfiguration of your PiHole possible so that you actually let it run in Host mode so you don't have to run it on it's own IP?

Link to comment
45 minutes ago, ich777 said:

6.9.0+

If you are on 6.8.3 you can't find it in the CA App.

Hmm i'm on 6.9.0-rc2, so that must be it. Can't run any higher version until Unraid fixes the "non existing crashing issue" 😕

Oh well, thanks anyway!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.