[Plugin] Prometheus unRAID Plugins


Recommended Posts

Hi there,

I am trying to install the node exporter plugin but it's failing to do so.

 

I have docker disabled and don't intend to enable it since I have other servers running prometheus, etc. I just want the node exporter running so that I can access it externally.

 

The error it gives me:

Something really wrong went on during onStartupScreen
Post the ENTIRE contents of this message in the Community Applications Support Thread

OS: 6.9.2
Browser: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36
Language:

<br /> <b>Warning</b>: parse_ini_file(/boot/config/docker.cfg): failed to open stream: No such file or directory in <b>/usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php</b> on line <b>50</b><br /> <br /> <b>Warning</b>: array_replace_recursive(): Expected parameter 2 to be an array, bool given in <b>/usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php</b> on line <b>50</b><br /> {"status":false}

 

This also forcefully unmounts my flashdrive, requiring a reboot to fix that. Is it possible to install the plugin without docker enabled or is that not an intended use case?

Link to comment
4 hours ago, FirstDose said:

Is it possible to install the plugin without docker enabled or is that not an intended use case?

I've now tested it and it works as expected:

grafik.thumb.png.b664b2f8ecb7fbebf6c6b33626573408.png

 

As you can see from the screenshot I have Docker also disabled and have the Prometheus Node Exporter running, here is also a screenshot from the output from the exporter:

grafik.thumb.png.24545ba62e94bc96dfeafba7fc23e694.png

 

I have also rebooted the machine to check if it is working as it should and it rebooted without an issue (test was done on 6.9.2 but I can also test it on another version from unRAID but this will make no difference).

 

Do you have enough space on the USB Boot device left?

 

@Squid do you have a idea why he is getting this message from above?

Link to comment

I have over a dozen of GB left on the flashdrive and I also successfully installed the nvidia smi exporter. But for some reason the node exporter is not working.

 

Here is what it says when I try to install it:
 

plugin: installing: https://github.com/ich777/unraid-prometheus_node_exporter/raw/master/prometheus_node_exporter.plg
plugin: downloading https://github.com/ich777/unraid-prometheus_node_exporter/raw/master/prometheus_node_exporter.plg
plugin: downloading: https://github.com/ich777/unraid-prometheus_node_exporter/raw/master/prometheus_node_exporter.plg ... done
plugin: downloading: https://github.com/ich777/unraid-prometheus_node_exporter/raw/master/packages/prometheus_node_exporter-2021.11.18.tgz ... done

+==============================================================================
| Installing new package /boot/config/plugins/prometheus_node_exporter/prometheus_node_exporter-2021.11.18.tgz
+==============================================================================

Verifying package prometheus_node_exporter-2021.11.18.tgz.
Installing package prometheus_node_exporter-2021.11.18.tgz:
PACKAGE DESCRIPTION:
Package prometheus_node_exporter-2021.11.18.tgz installed.
-------------Starting Prometheus Node Exporter!--------------

Warning: copy(/boot/config/plugins/prometheus_node_exporter.plg): failed to open stream: Input/output error in /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin on line 535
plugin: prometheus_node_exporter.plg installed

Updating Support Links



Finished Installing. If the DONE button did not appear, then you will need to click the red X in the top right corner
DONE

 

Once I click DONE it gives me the error I posted earlier. When I go to the CA tab it gives me the following error as my flashdrive has been forcefully unmounted:
 


Fatal Flash Drive Error
This usually means your flash drive is corrupted or has dropped offline. The rest of the Unraid GUI will also be affected. Post a new thread in the Unraid forums if this error continues


OK

 

Rebooting fixes the flashdrive.

Link to comment
  • 2 weeks later...

I would like to add authentication for the node-exporter and the Prometheus metrics endpoint. What I did so far:

 

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 15s
    static_configs:
      - targets: ["localhost:9090"]
      - targets: ["<UNRAID_IP>:9100"]
    basic_auth:
      username: myuser
      password: mypassword

 

I then updated the data source with the basic auth option in Grafana given the credentials listed above. Though, it seems like Grafana does not receive any metrics after that.

 

Do I have to add the basic auth credentials to the node-exporter too? If so, how can I pass the web.yml with the field basic_auth_users and the bcrypted password to the node-exporter plugin then?

 

Follow-up question: How can I pass a web.yml to the Prometheus container configuring basic authenticiation for connections to the Prometheus expression browser and HTTP API (reference) ?

 

Would love to see an "Authentication" section in one of the entry posts.

Edited by T0a
Link to comment
3 hours ago, T0a said:

If so, how can I pass the web.yml with the field basic_auth_users and the bcrypted password to the node-exporter plugin then?

You are talking about securing Prometheus am I right? I think you should head over to the Prometheus support thread to get help with that.

...no you shouldn't need to modify anything to the node-exporter plugin because Prometheus is polling the plugin.

Link to comment
  • 3 weeks later...

Hello ich777 et al,

I am having a little difficulty with generating the dashboard following the guide on the unraid blog

I've followed the instructions on the blog but my Grafana dashboard doesn't display any stats:
image.png

 

The Prometheus target appears to be up and functioning according the target stats page and I am receiving an output from the /metrics endpoint:
image.png

 

image.png

 

I've restarted my server and let waited for Grafana to pick up the output from Prometheus for over an hour, any thoughts on what might be wrong with my setup?

Thanks in advance! 

Edited by deadsix
Link to comment
1 hour ago, deadsix said:

I've restarted my server and let waited for Grafana to pick up the output from Prometheus for over an hour, any thoughts on what might be wrong with my setup?

Are you sure that you've picked the right data source? Have you also other sources in Grafana?

Link to comment
13 minutes ago, ich777 said:

Are you sure that you've picked the right data source? Have you also other sources in Grafana?


I believe so I've included screenshots of my configuration below.
 

I have no other sources and I received a 'Data source is working' response when I added the Prometheus data source seen below: 

image.thumb.png.6b174497abe8e746b1ea3d3708025dd8.png

 

I am importing the dashboard to my local Grafana with the following options:
image.png.6fab52a3817ad079c9b9b84dabd87210.png
 

Link to comment
8 hours ago, deadsix said:

I am importing the dashboard to my local Grafana with the following options:

I think that's an issue with the Dashboard itself, can you go into a graph itself and manually changing the data source to Prometheus, I think a few pages back a user experienced a similar issue with another Dasboard who has had also another Data Source in Grafana.

Link to comment
11 hours ago, ich777 said:

I think that's an issue with the Dashboard itself, can you go into a graph itself and manually changing the data source to Prometheus, I think a few pages back a user experienced a similar issue with another Dasboard who has had also another Data Source in Grafana.

 

I went into the dashboard and manually changed the data source to Prometheus, but the dashboard still isn't displaying any data. While I was poking around I did notice that the query inspector is stating that the response is empty:
image.thumb.png.399fcc27d78f96d6a2d3d008755eeef9.png

 

Edit: I figured it out, it was a simple misconfiguration issue Prometheus didn't have the right target port in the YAML file. For others who may encounter issues the Prometheus Node Exporter is only on port 9100. Thanks ich777 for your help and great plugin! 

 

 

Edited by deadsix
  • Like 1
Link to comment
  • 2 weeks later...

Dear Ich777,

 

read the whole treat, having following problem:

 

I am using the Node exporter, its works fine with Grafana and prometheus.yml, feeding data etc. Thank you for your efforts.

 

Secondly I wanted to set up the Adguard Home exporter too - installed it, added the additional line to promethus.yml with IP of the Adguard docker server (same as Unraid but with port 9617. Adguard Home Exporter is set with user pass and IP:3000 and its running. Restarting the Prometheus, when I open WebUI the Adguard home and node exporter are UP, loading 13300 Adguard to Grafana and targer Prometheus but no data is feeded.

 

Going back to Adguard Exporter refreshing the page shows it as stopped - starting it and getting back to WebUI of Prometheus -but it shows the Adguard Home instance down with following error: Get "http://192.168.0.11:9617/metrics": dial tcp 192.168.0.11:9617: connect: connection refused

If I restart the Adguard exporter and refresh Prometheus, some times it shows same error and sometimes its up,but Grafana does not show any data.

 

What could be the reason for the Adguard home exporter to keep stopping when I refresh the page - in Backup pluging have set it to Don't stop, as suggested. Adguard Home is running in docker as host. Have Enabled host access in docker settings, although I am not running in a custom network.

 

Would be great if you could suggest a solution. Your efforts and time taken is highly appreciated in advanced.

 

Regards,
Fotis

Link to comment
6 hours ago, Fotis Yanchev said:

"http://192.168.0.11:9617/metrics": dial tcp 192.168.0.11:9617: connect: connection refused

Can you connect from a local browser to this URL?

http://192.168.0.11:9617/metrics

 

6 hours ago, Fotis Yanchev said:

What could be the reason for the Adguard home exporter to keep stopping when I refresh the page

The reason might be that it tries to connect and it shows that it's actually connected but fails after a few seconds and if you refresh the page it shows the "real" status of the exporter.

 

I think something might be wrong with the connection itself, please try if you can connect to the URL above.

 

Adguard is on the host network and Prometheus and Grafana are on the "bridge" network, correct?

  • Like 1
Link to comment

When I start the exporter and before it stops - I open the http://192.168.0.11:9617/metrics and it shows that data is feeding - if I refresh the page several times - lets say after 10-20 sec. it shows site as unreachable, at that time i check the exporter and it shows as stopped.

When the exporter is running, at the link there is data visible, but Grafana does load any data - maybe the time the exporter is running is not enough for the data to be collected and shown in Grafana, but as said when the exporter is running the link shows the data, before the exporter stops. 

firefox_pU8wI7N3b9.png

firefox_fepoGkA6iM.png

firefox_PiyNuU9Vp3.png

Link to comment
1 hour ago, Fotis Yanchev said:

When the exporter is running, at the link there is data visible, but Grafana does load any data

This may take a while to load the data in Grafana itself.

 

Have you yet tried to restart unRAID itself?

 

Do you have any other data sources in Grafana than Prometheus?

  • Like 1
Link to comment
1 minute ago, Fotis Yanchev said:

No only Prometheus, installed both with Grafana just yesterday, will restart unRaid now and start the exporter. I know it takes time to load data in Grafana, but if the exporter stops, it will not load any data.

This is really strange, I will try it on my test server with a fresh installed Prometheus Exporter, please give me some time since I'm currently on vacation.

  • Like 1
Link to comment

I will test and keep you updated. Its nothing urgent, enjoy of course your vacation. Will restart the server and try again, is there any log that I can run on the AdGuard home Exporter to investigate the reason for stopping after 10-20 sec. Maybe it could be of help for you.

Sorry for dummy questions- its my first server with Unraid and just adding some apps/plugins and trying to get to know the whole thing.

Link to comment
3 minutes ago, Fotis Yanchev said:

I will test and keep you updated. Its nothing urgent, enjoy of course your vacation. Will restart the server and try again, is there any log that I can run on the AdGuard home Exporter to investigate the reason for stopping after 10-20 sec. Maybe it could be of help for you.

In this case no. But we can try something else next week if that's not too late for you.

 

I will test it and get back to you.

  • Thanks 1
Link to comment
  • 2 weeks later...

Hi everyone,

 

I've been reading your tutorial but booting up Prometheus's docker is already a mess. 

 

I'm getting the following message in the logs before it goes back to "stopped" status :

 

ts=2022-01-13T17:23:18.190Z caller=main.go:520 level=info build_context="(go=go1.17.5, user=root@54b6dbd48b97, date=20211217-22:08:06)"
ts=2022-01-13T17:23:18.191Z caller=main.go:521 level=info host_details="(Linux 5.14.15-Unraid #1 SMP Thu Oct 28 09:56:33 PDT 2021 x86_64 947eed537a1c (none))"
ts=2022-01-13T17:23:18.191Z caller=main.go:522 level=info fd_limits="(soft=40960, hard=40960)"
ts=2022-01-13T17:23:18.191Z caller=main.go:523 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-01-13T17:23:18.205Z caller=query_logger.go:86 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"

panic: Unable to create mmap-ed active query log

 

Sounds like an permission problem.

 

http://hostip/metrics is accessible whatever but not relevant

 

Just in case, the docker folder has been set on a zfs dataset, but i doubt that it could lead to the source of the trouble.

 

I'm not fully instructed to docker so if you may have an idea on how to resolve this little problem?

 

Thanks.

Edited by gyto6
Link to comment
1 hour ago, gyto6 said:

Sounds like an permission problem.

Exactly, I would recommend to start over and delete the container and also the directory for this container.

Also this seems more like an issue with the Prometheus container so I would recommend that you ask in the support forums from the container iteslf.

 

1 hour ago, gyto6 said:

Just in case, the docker folder has been set on a zfs dataset, but i doubt that it could lead to the source of the trouble.

Are you on 6.9.2 then it could be possible that if you have certain application files from Docker on ZFS that it simply doesn't work.

  • Like 1
Link to comment
31 minutes ago, ich777 said:

Also this seems more like an issue with the Prometheus container so I would recommend that you ask in the support forums from the container iteslf.

 

Are you on 6.9.2 then it could be possible that if you have certain application files from Docker on ZFS that it simply doesn't work.

My mistake! I forgot that you didn't design the docker by itself.. 

 

I'm doing my way on the docker support forum.

 

Thanks for your help, I'll try your way around, even if for a first install, it sounds quite weird.

Link to comment
43 minutes ago, ich777 said:

Exactly, I would recommend to start over and delete the container and also the directory for this container.

Man/Woman, your solution's worked.. I should have done this before 😅

 

Thank you!

Edited by gyto6
  • Like 1
Link to comment
On 9/23/2021 at 11:03 AM, ich777 said:

Prometheus Fritzbox Exporter

 

  1. Download and install the Prometheus Fritzbox Exporter plugin from the CA App:
    grafik.png.81d0b5d47e26a5f6f25aaf9287c6d6d0.png
     
  2. Log in to your Fritzbox and go to "System -> FRITZ!Box-Benutzer":
    1.png.85fd439e28dc83e8a944bf78d71db52b.png
     
  3. Click on "Benutzer hinzufügen":
    2.thumb.png.0c19f57b0c9cc2cb8966eb584a3479db.png
     
  4. Create a new user with a password for the exporter (in this example "grafana") and tick the following check boxes ("FRITZ!Box Einstellungen", "Sprachnachrichten, Faxnachrichten, FRITZ!App Fon und Anrufliste" & "Smart Home"):
    3.thumb.png.9558ce3ab28c1fed0b1338a220e8fa84.png
     
  5. Go to "Heimnetz -> Netzwerk -> Netzwerkeinstellungen" and select "Statusinformationen über UPnP übertragen":
    4.thumb.png.bff9cef511ed651cf87662f10cca5374.png
     
  6. Go to the plugin settings by clicking on "Settings -> Fritzbox Exporter" (at the bottom of the Settings page) :
    6.png.252254d95c43d25e5b4c4e47f931857c.png
     
  7. Enter the username and password that you've created in Step 3 for the exporter in your Fritzbox and click on "Confirm & Start":
    grafik.thumb.png.c73190510e94ac9b64ee9801a697393b.png
     
  8. After that you should see in the right top corner that the Exporter is running and details about it:
    grafik.thumb.png.056ce392dc0b9f1ea66514266236e15f.png
     
  9. Open up the prometheus.yml (Step 4 + 5 from the first post), add a line with '- targets: ["YOURSERVERIP:9042"]' (please change "YOURSERVERIP" to your Server IP), save and close the file:
    grafik.png.a995d2f4c920bda6e1bb170be31259fe.png
     
  10. Go to the Docker page and restart Prometheus:
  11. grafik.png.5ce625ba0089855ec275263b094e4a1a.png
     
  12. Open up the Grafana WebUI:
    grafik.png.44941fa8fff8d2a61d93b0b7c7e18d23.png
     
  13. In Grafana click on "+ -> Import":
    grafik.png.74e5ee056c9b2b36be0799d9433d6088.png
     
  14. Now we are going to import a preconfigured Dashboard for the Fritzbox Exporter from Grafana.com (Source), to do this simply enter the ID (12579) from the Dasboard and click "Load":
    grafik.png.cb2ed3299ec9ce6687c1d9ac6c45323c.png
     
  15. In the next screen rename the Dashboard to your liking, select "Prometheus" as datasource and click on "Import":
    grafik.png.9f28b30cfa6dff242fd29d23d2703f00.png
     
  16. Now you should be greeted with something like this (please keep in mind that the Dashboard can display N/A at some values, especially at the gauges, since there is not enough data available, wait a few minutes and you will see that the values are filled in):
    5.thumb.png.33b64fb87bc29df372ba6812383c8d16.png

 

 

This seems to be a nice plugin. But anyway I cant get it to work. Sadly. But anyway thank you for this!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.