[Plugin] Prometheus unRAID Plugins

[ich777]

15 hours ago, glennbrown said:

@ich777 I submitted a pull request against the smartctl exporter textfile collector. There was a spelling mistake in the one nested if conditional when checking node_exporters settings.cfg causing it to always return null. I also did some minor bash syntax cleanup in that nested IF.

Thank you for the PR, merged and update the plugin.

 

15 hours ago, glennbrown said:

Also wondering did you have any plans to update node-exporter to 1.5.0 and would be open to a pull request to enable the cpu info collector by default?

Ah, sorry yes, completely forgot about that, something wasn't working with the automated build from 1.5.0 and I forgot to look into it.

Will look into it ASAP, keep an eye up for a updated from the plugin (should be done until tomorrow).

Done, update! Thank you for the heads up.

[Sqibble]

On 7/6/2022 at 4:09 PM, ich777 said:

Have you yet seen this: Click

 

I have not released this to the public because I have no testers out there and this it is in "beta"...

 

You can install it by simply putting this line into the plugin installation:

https://raw.githubusercontent.com/ich777/unraid-prometheus_smartctl_exporter/master/prometheus_smartctl_exporter.plg

 

 

Sorry for the noobish question but, what exactly is needed to make this work?

 

I "put the line into the plugin installation" and it installed successfully. Do I also follow the instructions on https://github.com/micha37-martins/S.M.A.R.T-disk-monitoring-for-Prometheus ? Do I need to reboot, or restart the exporter? What are the metrics named as?

[ich777]

13 minutes ago, Sqibble said:

Sorry for the noobish question but, what exactly is needed to make this work?

Try this plugin instead:

https://github.com/ich777/unraid-prometheus_smartctl_exporter

 

Setting up such add-ons for the node exporter as the one that you've linked can be really tedious on Unraid.

[Sqibble]

12 minutes ago, ich777 said:

Try this plugin instead

That's the one I quoted from you before 😋. I have that installed but see no additional metrics come in.

[ich777]

41 minutes ago, Sqibble said:

That's the one I quoted from you before 😋. I have that installed but see no additional metrics come in.

Do you have the node_exporter installed?

 

I really don't know because I haven't installed the exporter anymore but I assume it is working because someone opened a PR and added proper support for NVME drives.

[Sqibble]

6 minutes ago, ich777 said:

Do you have the node_exporter installed?

I do yes. Beyond that I simply installed https://raw.githubusercontent.com/ich777/unraid-prometheus_smartctl_exporter/master/prometheus_smartctl_exporter.plg. If that's all it should take then I'll just try a reboot and see what happens.

 

Thanks for the quick responses and your numerous contributions to Unraid, I use many and I love it 😁!

[francispb]

I tried to follow the tutorial, but when I get to the step of opening Grafana's WebUI, the page that opens is an error: This page is not working, ERR_TOO_MANY_REDIRECTS

 

 

 

I realize that the browser address is repeating the IP of my Unraid server, when I force the address to 192.168.1.100:3000/login I get this message:

 

 

But I don't use proxy or reverse proxy. How do I resolve this?

[ich777]

22 minutes ago, francispb said:

I tried to follow the tutorial, but when I get to the step of opening Grafana's WebUI, the page that opens is an error: This page is not working, ERR_TOO_MANY_REDIRECTS

You should report that in the Grafan support thread, I can only give support for the Plugins that I've created for Unraid.

[codefaux]

Bug report. Prometheus Node Exporter throws an error every single time it's polled by Prometheus.

EDIT: As of version 2023.02.26 -- first time using this plugin, unsure when bug first appeared

 

Text is;

ts=2023-03-24T17:51:45.859Z caller=collector.go:169 level=error msg="collector failed" name=mdadm duration_seconds=0.000470481 err="error parsing mdstatus: error parsing mdstat \"/proc/mdstat\": not enough fields in mdline (expected at least 3): sbName=/boot/config/super.dat"

 

Edited March 24, 2023 by codefaux

[ich777]

2 hours ago, codefaux said:

Prometheus Node Exporter throws an error every single time it's polled by Prometheus.

A little bit more information would be helpful.

Where do you get that error?

I can't reproduce that.

[codefaux]

22 hours ago, ich777 said:

A little bit more information would be helpful.

Where do you get that error?

I can't reproduce that.

Literally just run the binary from the ssh terminal. I hadn't written its config file yet so I noted how it was executed, killed it, wrote the config file, and executed it by hand in a terminal.

 

[ich777]

1 hour ago, codefaux said:

Literally just run the binary from the ssh terminal. I hadn't written its config file yet so I noted how it was executed, killed it, wrote the config file, and executed it by hand in a terminal.

Why are you doing that by hand? What settings did you change? I installed the plugin and have no issue whatsoever and can connect to it just fine.

The plugin is only meant to be installed and that's it. You can however run it with custom parameters but I think in your case something is wrong if you changed some settings, that's my best guess.

[codefaux]

35 minutes ago, ich777 said:

Why are you doing that by hand? What settings did you change?

I already explained that, but I'll be more verbose.

 

I downloaded the plugin before I wrote its config file. You wrote the instructions. Steps four and five. I did not perform steps four and five until after I had installed the plugin. I also did not do anything beyond those steps, regrading modifying settings, configuration or parameters.

 

There's no "restart" control that I could find, and I didn't feel like restarting my entire server (or the array and all of my services) simply to restart a single plugin. Thus, I used the console, killed the running process, and restarted it by hand. No custom parameters, I didn't change some settings.

 

42 minutes ago, ich777 said:

I installed the plugin and have no issue whatsoever and can connect to it just fine.

You'll note that I never said there was a user-facing issue, or that I couldn't connect or report metrics from it. It functions just fine, but on my system it's burping a line at the console about an internal (non-critical) error, every time Prometheus connects to it and retrieves metrics.

 

The only difference between now and if I uninstall/reinstall/reboot is that the errors will be sent to a logfile someplace or discarded entirely -- I have no idea which -- instead of being sent to the console, since I ran it by hand.

 

What I'm realizing though, is this is above your head, so to speak. If you run it by hand yourself, does it throw an error every time Prometheus polls it? I'll file a bug upstream with the actual node collector's maintainer, as it's now clear to me that the actual collector is mishandling the output of /proc/mdstat on my system and it has nothing to do with the small wrapper plugin you wrote.

 

49 minutes ago, ich777 said:

The plugin is only meant to be installed and that's it.

Mmmmmmm no, though. It's not "only meant to be installed."  It's meant to be learned about, your system is meant to be manually prepared for it, and then it's meant to be installed. Steps four and five could/should be handled by either the previous step's container, or this plugin itself if it notices the configuration file is not present. Furthermore, installing the plugin starts the collector, which already expects the config file to be present, so steps four and five should actually be before step three.

 

If the installation process weren't so complicated, I would've noticed that this wasn't your problem earlier. I installed the plugin by finding it in the CA app and going "hey that does what I need" and then discovering that it wasn't working. And in NO situation do you merely "install the plugin and that's it" so that's just a flat inaccurate thing to claim.

[ich777]

6 hours ago, codefaux said:

I downloaded the plugin before I wrote its config file. You wrote the instructions. Steps four and five. I did not perform steps four and five until after I had installed the plugin. I also did not do anything beyond those steps, regrading modifying settings, configuration or parameters.

Oh, sorry now I get it...

 

Steps four and five for the Docker container not the plugin, sorry for that...

The Prometheus Docker container which is needed is simply the bridge between the Prometheus Node Exporter plugin and Grafana and has strictly speaking nothing to do with the plugin itself.

 

6 hours ago, codefaux said:

Steps four and five could/should be handled by either the previous step's container, or this plugin itself if it notices the configuration file is not present

Yes sure, I'll agree on that, but that's a thing that I can't change since this is not my Docker container, I often thought about removing these step including installing Grafana and only advice to install Grafan and Prometheus. These steps gave me some issues in the past because on some systems the Promehteus Docker continaer wouldn't work on some systems (which is out of my control).

 

The plugin can't install a configuration file for a Docker container because first I don't like changing things which are not directly controlled by the plugin and second the plugin can't know where the configuration for your Prometheus Docker container is, there are simply too many variables (name, paths,...).

 

6 hours ago, codefaux said:

There's no "restart" control that I could find, and I didn't feel like restarting my entire server (or the array and all of my services) simply to restart a single plugin. Thus, I used the console, killed the running process, and restarted it by hand. No custom parameters, I didn't change some settings.

Usually you don't have to restart the plugin itself because the config that you are changing is related to the Docker container not the plugin.

 

6 hours ago, codefaux said:

If you run it by hand yourself, does it throw an error every time Prometheus polls it?

Yes, it's actually exactly the same on my system but TBH, I've never noticed it because I've always ran it as a service:

 

6 hours ago, codefaux said:

I'll file a bug upstream with the actual node collector's maintainer, as it's now clear to me that the actual collector is mishandling the output of /proc/mdstat on my system and it has nothing to do with the small wrapper plugin you wrote.

I've found already another report here on the forums but that's related to the Docker container:

 

But please keep in mind that Unraid doesn't use the default md driver and instead it uses it's custom md driver for the Array.

I think the issue is caused by that which is also mentioned in the bug report from the Docker container.

 

Anyways, thank you for the report, I will look into that when I got a bit more spare time.

 

For now you can, of course if you want to, disable the mdadm metrics entirely.

Simply edit your "/boot/config/plugins/prometheus_node_exporter/settings.cfg" file so that it looks like this:

start_parameters=--no-collector.mdadm

 

Of course you have to restart your server because, as you've mentioned earlier, there is not restart function implemented in the plugin because this is more a niche use case to pass arguments to the plugin itself, however if you want to start it in the background as usual simply do that:

echo "/usr/bin/prometheus_node_exporter --no-collector.mdadm" | at now

of course you first have to kill the background exporter process first with something like that:

kill $(pidof prometheus_node_exporter)

 

or if you want to test the command first without starting it in the background:

/usr/bin/prometheus_node_exporter --no-collector.mdadm

 

 

Hope that helps for now!

[gambler32k]

Anyone elses Prometheus AdGuard Exporter plugin stopped working?

i had everything setup and working for months but now nothing happends when pressing start button.

Tried reinstalling the plugin with no luck.

[codefaux]

On 3/25/2023 at 9:32 PM, ich777 said:

Oh, sorry now I get it...

 

Steps four and five for the Docker container not the plugin, sorry for that...

lol it's ok, I was just very confused. I'm not even using the Docker containers, I JUST needed the exporter; I'm running the rest on another system. That's the main reason I went into this so sideways; I only wanted the exporter, but it had all these extra steps and I had to assume which went to what. I did so incorrectly, but that's ok.

 

 

On 3/25/2023 at 9:32 PM, ich777 said:

The plugin can't install a configuration file for a Docker container because first I don't like changing things which are not directly controlled by the plugin and second the plugin can't know where the configuration for your Prometheus Docker container is, there are simply too many variables (name, paths,...).

For sure! I thought the file went with the plugin, that's all.

 

 

On 3/25/2023 at 9:32 PM, ich777 said:

Usually you don't have to restart the plugin itself because the config that you are changing is related to the Docker container not the plugin.

Yup, but I didn't know that and it was all in one giant block so I assumed (incorrectly) and here we are, lol

 

 

On 3/25/2023 at 9:32 PM, ich777 said:

Yes, it's actually exactly the same on my system but TBH, I've never noticed it because I've always ran it as a service:

That's very much what I had expected, literally everyone using this plugin is silently ignoring repeated errors because the stdout isn't directed anywhere people know about...

 

 

On 3/25/2023 at 9:32 PM, ich777 said:

keep in mind

Synonymous with "remember that" -- it's something you say to people who know the thing you're about to say.

 

 

On 3/25/2023 at 9:32 PM, ich777 said:

Unraid doesn't use the default md driver

COMPLETELY did not know that, but given the reply upstream that does not surprise me anymore lol.

 

 

 

On 3/25/2023 at 9:32 PM, ich777 said:

For now you can, of course if you want to, disable the mdadm metrics entirely.

I'll remember that. Since the logs aren't kept, it won't be filling up a logfile. I haven't restarted my array in a few months and I don't intend to soon, so I'll likely just leave them running in a detached tmux until the issue is properly resolved.

 

 

I'd already opened a bug report upstream, I'll link them here and add this information but it seems for now the best bet would be to patch the plugin to disable the md collector on your end.

 

Edit: Github issue https://github.com/prometheus/node_exporter/issues/2642

Edited March 28, 2023 by codefaux
add github issue link

[codefaux]

4 hours ago, gambler32k said:

Anyone elses Prometheus AdGuard Exporter plugin stopped working?

i had everything setup and working for months but now nothing happends when pressing start button.

Tried reinstalling the plugin with no luck.

Might be worth running it from the console to see if there's any ignored error spam like in my comment. Could be an error being dropped instead of forwarded to system logs, like on my (and everyone else's) system using the Node Exporter.

[ich777]

4 hours ago, gambler32k said:

Anyone elses Prometheus AdGuard Exporter plugin stopped working?

Did you reboot your server or was rhere anpower outage?

Does you AdGuard container run on br0?

 

If yes restart the Docker service one time.

[codefaux]

Hey here's another suggestion for an easy addition full of feature delivery.

 

There exists a handful of community scripts to ammend data provided by node_exporter. Most of them seem to be intended to pipe to a text file and read via node_exporter's textfile collector.

 

The metrics go into a file wherever, then to the exporter as a launch parameter, you add `--collector.textfile.directory=/var/lib/node_exporter` for example, and all of the readable files within /var/lib will be exported as metrics if they're in the correct format.

 

For example, smartmon.sh writes smartmon statistics such as temperature and unallocated blocks. nvme_metrics.sh might be of interest, btrfs_stats.py, maybe even directory-size.sh for some folks.

 

The most simple way I can think of is for your plugin to create a directory in the ram filesystem at /var/log and add `--collector.textfile.directory=/var/log/node_exporter` and suggest users execute the desired scripts, writing into /var/log/node_exporter in per-script files. I can see two ways of doing this.

- One, users copy script files to someplace like /boot/config/scripts (one-time write for the scripts, no flash wear) and execute them via the User Scripts plugin as such;

 

 

Scheduled similarly;

 

 

The /var/log filesystem will exist on any system, won't cause flash wear, is wiped on reboot.  The path should have plenty of space for a few kb (or even dozen MB) of metrics being rewritten every few minutes. If it doesn't, failure case is that logging fails on the system -- not ideal but it's mounted with 128MB by Unraid and should never be near full unless a user has a serious problem elsewhere. If it's filling, the absense of this plugin's proposed metrics won't prevent it or even delay it by much.

 

These metrics are designed to be overwritten, not appeneded, so they should never grow more than a few dozen MB in the most obscene scenario. Plugins seem to run as root, so permissions shouldn't be a problem.

 

I'm also going to ping the user scripts developer to allow stdout to be piped to a file per-script, so users can simply paste the scripts into User Scripts and forward the stdout, instead of needing to save them to /boot/config/scripts manually and write a User Script to run it manually.

 

Edited March 29, 2023 by codefaux
extra image oops

[ich777]

8 hours ago, codefaux said:

For example, smartmon.sh writes smartmon statistics such as temperature and unallocated blocks. nvme_metrics.sh might be of interest, btrfs_stats.py, maybe even directory-size.sh for some folks.

You can also use my plugin from here, it never made it to the CA App because I couldn't fully test it (I have a lot of things to maintain nawadays).

 

Just go to the Plugins section, click on the Install Plugin tab, paste this URL and click Install:

https://raw.githubusercontent.com/ich777/unraid-prometheus_smartctl_exporter/master/prometheus_smartctl_exporter.plg

 

8 hours ago, codefaux said:

One, users copy script files to someplace like /boot/config/scripts

I would not recommend the boot drive for this, not because of wear reasons, but maybe consider maybe switching over to /mnt/user/appdata/scripts and run it from there.

[codefaux]

On 3/29/2023 at 10:57 PM, ich777 said:

You can also use my plugin

That doesn't support any of the other metrics from any of the other dozen or so scripts at the link I mentioned, a few of which I'm interested in/using now manually. That's why I suggested adding the text collector as a supported feature on the plugin by default. It won't impact anyone except people who want to use it if it merely watches a folder for scripts to execute.

 

 

On 3/29/2023 at 10:57 PM, ich777 said:

I would not recommend the boot drive for this, not because of wear reasons

Could you be bothered to explain why? I don't mean to sound dismissive or aggressive or anything, but would you follow a recommendation from a stranger with no justification or explanation?

 

On 3/29/2023 at 10:57 PM, ich777 said:

but maybe consider maybe switching over to /mnt/user/appdata/scripts and run it from there

I'm not looking for suggestions; I'm not the plugin author here lol

 

I'll allow them to be run from anywhere you require them to be placed. I'm suggesting that you add in support for the text collector to your prometheus_node_exporter plugin, since prometheus_node_exporter already supports it. I'm suggesting where you should place them. You disagree; place them anywhere you like.

[MVLP]

I have a issue with the FritzBox Exporter.

The amount of connected WiFi Devices are wrong, which get written into Prometheus.

 

I have several Mesh ACs and the devices which are connected to the ACs will not get Exported.

The only exported WiFi Connections are the Devices which are direclty Connected to my Fritzbox.

Is there any chance to get the devices which are connected via the Mesh ACs?

[ich777]

1 hour ago, MVLP said:

The amount of connected WiFi Devices are wrong, which get written into Prometheus.

Please report that as an issue over here on GitHub since I've only made the plugin for Unraid.

[MVLP]

1 hour ago, ich777 said:

Please report that as an issue over here on GitHub since I've only made the plugin for Unraid.

Thank you for the Information.

I will create an Bug Report.

[Masterwishx]

@ich777 can you maybe add NUT (ups) prometheus expoter if you will have time for it ?

something like https://github.com/HON95/prometheus-nut-exporter or https://github.com/DRuggeri/nut_exporter