[Plugin] Prometheus unRAID Plugins

[unw1red]

I am not running keepalived. They are individual instances of PiHole with both designated in my DHCP scope.

 

Edited July 6, 2021 by unw1red

[ich777]

2 hours ago, unw1red said:

I am not running keepalived. They are individual instances of PiHole with both designated in my DHCP scope.

You can basically put the start command in the go file as long as the instance is already up when you start unRAID.

 

What you can also do is that you write a custom cron script or a script that runs on boot like this:

#!/bin/bash
echo "prometheus_pihole_exporter \
  -pihole_api_token YOURTOKEN \
  -pihole_hostname IPFROMTHEPIHOLEINSTANCE \
  -pihole_protocol http \
  -pihole_port WEBUIPORTFROMPIHOLEINSTANCE \
  -port PORTWHERETHEEXPORTERRUNSON \
  -interval 10s" | at now

YOURTOKEN= The token from the other PiHole instance

IPFROMTHEPIHOLEINSTANCE= The IP from the second PiHole instance

WEGUIPORTFROMPIHOLOEINSTANCE= The port for the WebGUI - usually 80

PORTWHERETHEEXPORTERRUNSON= Something like 9618 - important that it is a different port than the plugin runs on (9617) and you have to insert this as a new target to your prometheus.yml

 

Hope this makes sense, please keep in mind that I don't designed the plugin supports more than one instance at a time but with this in the go script or a cron job at start from unRAID this will totally work.

(You can also put 'sleep 30s' after the line '#!/bin/bash' so that the execution will 30 seconds delayed after the script is called)

 

I never tried this but it should work fine too.

 

Note: If you put this in the go file you don't need the line '#!/bin/bash'

[unw1red]

I figured out what the issue was. I did not have default selected for the Prometheus data source in Grafana. The data was populating the database, but not being displayed in Grafana. 

[boosting1bar]

On 7/4/2021 at 11:52 PM, ich777 said:

Do you run your containers in a custom network on br0?

If yes you need to enable Allow Host Access in the Docker settings.

 

I do, and it is enabled.  I even tried toggling it on and off and rebooting my server and still getting that same 404 server down error

[ich777]

1 hour ago, boosting1bar said:

 

I do, and it is enabled.  I even tried toggling it on and off and rebooting my server and still getting that same 404 server down error

Then there has to be something else wrong.

 

Try to set the network to bridge four troubleshooting reasons and look if you get the same error please.

 

A 404 error tells you that Prometheus can't connect to the service.

 

Can you reach YOURUNRAIDIP:9100/metrics from your browser?

 

Please make sure that your IP in the yml file matches your unRAID IP.

[boosting1bar]

3 hours ago, ich777 said:

Then there has to be something else wrong.

 

Try to set the network to bridge four troubleshooting reasons and look if you get the same error please.

 

A 404 error tells you that Prometheus can't connect to the service.

 

Can you reach YOURUNRAIDIP:9100/metrics from your browser?

 

Please make sure that your IP in the yml file matches your unRAID IP.

I've tried it on bridge and host and it kicks me the same error so far and shows localhost up and ip:9100 404 down.

 

If I try to reach unraidIP:9100/metrics I just get a blank white page with text that says Not Found at the top, if I take off /metrics it just gives me a blank white page so it's getting some response and not just a can't load this page error.

 

And I've checked the IP in the yml and it's correct.  I even tried adding http:// but that breaks Prometheus it looks like.  I don't have any errors in the Prometheus log for the container.

Edited July 7, 2021 by boosting1bar

[ich777]

1 hour ago, boosting1bar said:

If I try to reach unraidIP:9100/metrics I just get a blank white page with text that says Not Found at the top, if I take off /metrics it just gives me a blank white page so it's getting some response and not just a can't load this page error.

Are you sure that the Prometheus Node Exporter is installed and not in the Plugin Error tab on the plugin page?

 

Your problem is that the exporter doesn't seem to run.

 

Try to uninstall and reinstall the plugin again.

Also when you install the plugin don't close the window with the red X on top, click Done!

 

Eventually also try to reboot your server if that all doesn't help.

[SpencerJ]

If you would like to donate generously to @ich777 for this and all of his other great work;

 

 please see here!

[boosting1bar]

11 hours ago, ich777 said:

Are you sure that the Prometheus Node Exporter is installed and not in the Plugin Error tab on the plugin page?

 

Your problem is that the exporter doesn't seem to run.

 

Try to uninstall and reinstall the plugin again.

Also when you install the plugin don't close the window with the red X on top, click Done!

 

Eventually also try to reboot your server if that all doesn't help.

Yep just went through and uninstalled and reinstalled exporter again and getting the same error and "Not found" /metrics page.  I've reboot again as well and still no change.  I really appreciate your help but I'm not going to keep chewing up your time trying to track this down, I'm just going to delete and not junk up the thread!

[ich777]

48 minutes ago, boosting1bar said:

Yep just went through and uninstalled and reinstalled exporter again and getting the same error and "Not found" /metrics page.  I've reboot again as well and still no change.  I really appreciate your help but I'm not going to keep chewing up your time trying to track this down, I'm just going to delete and not junk up the thread!

Nothing to worry about, I think this issue can be solved pretty easily. Can you issue the command: 'pidof prometheus_node_exporter' (without quotes) from an unRAID terminal and post the output here (with the plugin installed)?

[TheDragon]

Awesome info you've shared, thank you so much for taking the time to do so!!

 

Could you possibly cover how to setup alerts?

 

I'm setup with this and is working great, I just can't quite get my head around setting alerts, especially with things that are true/false...

 

For example - I have a dashboard for my router, and want to set alerts to notify of VPN or WAN connections going up or down.

[kjames2001]

Hi ich777, thanks for your hard work!

 

I tried your guide, and setup Prometheus and Prometheus node exporter without any issues. 

 

However, when i setup the pihole plugin, and enter my pihole ip and port 80 and api token, and click change and start, it runs for a while and stops automatically.

 

I get the following error:

 

2021/07/11 01:30:52 Starting HTTP server
2021/07/11 01:31:02 An error has occured during retrieving PI-Hole statistics Get "http://192.168.1.46:281/admin/api.php?summaryRaw&overTimeData&topItems&recentItems&getQueryTypes&getForwardDestinations&getQuerySources&jsonForceObject&auth=*****************************************": dial tcp 192.168.1.46:281: connect: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x811daa]

goroutine 37 [running]:
github.com/eko/pihole-exporter/internal/pihole.(*Client).getStatistics(0xc0001663f0, 0xc000042790)
/root/prometheus_pihole_exporter/pihole-exporter/internal/pihole/client.go:163 +0x2ea
github.com/eko/pihole-exporter/internal/pihole.(*Client).Scrape(0xc0001663f0)
/root/prometheus_pihole_exporter/pihole-exporter/internal/pihole/client.go:61 +0x88
created by main.initPiHoleClient
/root/prometheus_pihole_exporter/pihole-exporter/main.go:37 +0xc5

 

192.168.1.46 is my pihole ip and using port 80

 

please help, im a newbie to linux and have no idea what these is all about.

 

Thanks again for all your hard work.

 

Edit: NVM, it just worked after i tried again. Maybe i entered the wrong port by mistake.

Again, thanks for your hard work!

 

Edit 2: some of my dockers stoped working after enabling host access to custom networks, disabled this and they works again. any idea how to fix this?

Edited July 11, 2021 by kjames2001

[ich777]

6 hours ago, kjames2001 said:

Edit: NVM, it just worked after i tried again. Maybe i entered the wrong port by mistake.

Again, thanks for your hard work!

Have you CA Backup installed? Keep in mind, everytime you restart PiHole or strictly speaking stop it, the exporter will also stop.

 

If you want to workaround this you have to change the setting for PiHole in CA Backup so that it don't stops (how to do it is in the tutorial).

 

14 hours ago, jack0w said:

Could you possibly cover how to setup alerts?

I will look into this and add this to the second post, but give me some time, have a lot of projects going on currently.

[kjames2001]

Just now, ich777 said:

Have you CA Backup installed? Keep in mind, everytime you restart PiHole or strictly speaking stop it, the exporter will also stop.

 

If you want to workaround this you have to change the setting for PiHole in CA Backup so that it don't stops (how to do it is in the tutorial).

i have and have set it up like in the tutorial, and it works. but now some of my dockers stoped working after enabling host access to custom networks, disabled this and they works again. any idea how to fix this?

[ich777]

4 minutes ago, kjames2001 said:

i have and have set it up like in the tutorial, and it works. but now some of my dockers stoped working after enabling host access to custom networks, disabled this and they works again. any idea how to fix this?

This should not be the case, if you enable host access all the containers in br0 can "speak" to unRAID and they should still work fine and I can think of nothing why they shouldn't work anymore with host access enabled.

The other way around, if you disable host access the containers an br0 can't talk to unRAID...

 

Are you trying to get PiHole running?

If so you can also get PiHole running on Host mod with the IP from unRAID, but please be aware that you need to male some tweaks to the Docker template and a few other modifications.

[kjames2001]

16 minutes ago, ich777 said:

This should not be the case, if you enable host access all the containers in br0 can "speak" to unRAID and they should still work fine and I can think of nothing why they shouldn't work anymore with host access enabled.

The other way around, if you disable host access the containers an br0 can't talk to unRAID...

 

Are you trying to get PiHole running?

If so you can also get PiHole running on Host mod with the IP from unRAID, but please be aware that you need to male some tweaks to the Docker template and a few other modifications.

its cloudflared and zerotier that's not working. will try to change that setting again later, cant stop docker now.

 

Edit: just retried, now everything works just fine. Thanks!

Edited July 11, 2021 by kjames2001

[stelks]

When I set up piHole I am getting an error of connection refused in Prometheus and then it is obviously not working in Grafana. Below are my configs, if someone sees something wrong any help would be appreciated. 

Error in Prometheus (4th one is piHole):

 

piHole Exporter: 

 

PiHole:

 

Thanks in advance 

Edited July 12, 2021 by stelks

[ich777]

1 hour ago, stelks said:

Error in Prometheus (4th one is piHole):

You have set the IP wrong in the prometheus.yml you have to specify the unRAID IP not the PiHole IP.

The PiHole Exporter is a bridge between your PiHole and Prometheus that runs on unRAID.

[kjames2001]

It turns out i still have the same issue: zerotier is not working when i have host access to custom networks enabled.

[stelks]

2 hours ago, ich777 said:

You have set the IP wrong in the prometheus.yml you have to specify the unRAID IP not the PiHole IP.

The PiHole Exporter is a bridge between your PiHole and Prometheus that runs on unRAID.

Thank you for your help that got it working in Prometheus!(Could of sworn I tried both lol) 

 

Do you have any idea why it would not be pulling the data into Grafana now that that is configured correctly. All of my other dashboards in Grafana that use Prometheus work fine but the piHole one just says no data. I have restarted all of the containers and deleted and re-imported the template.

[ich777]

Just now, stelks said:

Do you have any idea why it would not be pulling the data into Grafana now that that is configured correctly. All of my other dashboards in Grafana that use Prometheus work fine but the piHole one just says no data. I have restarted all of the containers and deleted and re-imported the template.

Please give it a little time to pick up the data and make sure the exporter is running in the plugin page.

PiHole can take a little to update if you configured it for the first time.

[ich777]

1 hour ago, kjames2001 said:

It turns out i still have the same issue: zerotier is not working when i have host access to custom networks enabled.

This is really strange, is a reconfiguration of your PiHole possible so that you actually let it run in Host mode so you don't have to run it on it's own IP?

[Luckyjoe]

Is the Node Exporter plugin only for 6.9.2? I can't find it under Apps?

I can see the other plugins, but nothing comes up when searching for Node Export or browsing ich777's Repository

[ich777]

20 minutes ago, Luckyjoe said:

Is the Node Exporter plugin only for 6.9.2? I can't find it under Apps?

6.9.0+

If you are on 6.8.3 you can't find it in the CA App.

[Luckyjoe]

45 minutes ago, ich777 said:

6.9.0+

If you are on 6.8.3 you can't find it in the CA App.

Hmm i'm on 6.9.0-rc2, so that must be it. Can't run any higher version until Unraid fixes the "non existing crashing issue" 😕

Oh well, thanks anyway!