[Plugin] Prometheus unRAID Plugins

[ich777]

4 minutes ago, Bitbass said:

I've rebooted as part of the 6.10.3 upgrades on both sides, but I just restarted the Prometheus container on the primary and same result.

What are the two subnets?

Are you really sure that those are not the Docker subnets that you are trying to connect to?

 

I'm really not sure what is going on but barely nothing is hard coded to my plugins, the exporters should listen to all IPs since I'm mostly using 0.0.0.0

[Bitbass]

On the primary I have several docker containers that are internally using 172.17.0.0. My primary's subnet is 192.168.169.0/24. The target subnet is 172.16.172.0/24, which you can see in the logs. The target was running docker for another container but I've turned that off. It's really only the Prom node exporter plugin on the target Unraid system. Nothing else running on there.

[ich777]

11 minutes ago, Bitbass said:

On the primary I have several docker containers that are internally using 172.17.0.0. My primary's subnet is 192.168.169.0/24. The target subnet is 172.16.172.0/24, which you can see in the logs. The target was running docker for another container but I've turned that off. It's really only the Prom node exporter plugin on the target Unraid system. Nothing else running on there.

That is really weird but I really can't help there, I can only thing of a routing issue in your network somewhere since everything is working on many other networks just fine.

Did you also set up proper routing tables on Unraid too?

[Bitbass]

I don't think it's routing related. I get an immediate response from the Ubuntu server target, which is also in the same target network. And my Mac is in the same source network (192.168.169.0/24) and it can connect to the Unraid target without issue. The default route is set correctly in both Unraid systems for their respective subnets and there are no firewall rules that would allow the Mac (as source) or the Ubuntu (as target) to work while preventing other things.

 

I'm pretty sure this has something to do with Unraid (which one?) getting confused about talking to another Unraid system that's on a different subnet.

 

I get it that this is fringe stuff and that it works for everyone else. I'm not expecting you to have the magic answer, just hoping if we keep talking about it, it might generate an idea for you, me or someone else reading this.

 

Thanks!

[ich777]

53 minutes ago, Bitbass said:

I get it that this is fringe stuff and that it works for everyone else. I'm not expecting you to have the magic answer, just hoping if we keep talking about it, it might generate an idea for you, me or someone else reading this.

My issue with this is that I don't know your exact network setup and I'm really not a network expert when it comes to something like your setup, sure basic VLANs and such are no issue for me but when it comes to a more complicated setup I really can't help as much as I want too...

 

I will think about this a bit more but I really can't imagine what's preventing the two system to talk to each other...

[Bitbass]

You're in luck, because I am a network guy by trade!

 

What I lack is the understanding of how the plug-ins talk to the networking or how Unraid itself does networking. It doesn't necessarily behave like a router or firewall and that gets me turned around some times.

 

I might just have to figure out how to pull a pcap from my network and look at the actual TCP session. The verbose curl output isn't quite enough detail and it's only curl.

[Bitbass]

Well, I just figured it out because of Wireshark. I had set the MTU to 9000 on the target system to support jumbo frames. This isn't a bad thing, unless its crossing to another network through a firewall and the firewall isn't happy about having to fragment it. I changed the target system MTU to 1500 and it started working immediately. I'm not certain it was the firewall, but it's pretty likely. My Mac, which has always been set to 1500, was always able to connect. No fragmentation.

 

Thanks for helping me work through this! Sometimes you just need to voice ideas to have the answer present itself.

[jakami99]

Hey

 

Just a little hint for people having issues, since nothing was working for me the last 2 hours.

Make sure that UNRAID can reach your fritzbox over fritz.box over the web terminal

ping fritz.box

 

I didn't realize it is accessing this way, maybe a possibility to insert an IP in the setup would help a lot.

Thanks for the plugin love it, now where it's working

Edited June 21, 2022 by jakami99
grammitic

[ich777]

54 minutes ago, jakami99 said:

I didn't realize it is accessing this way, maybe a possibility to insert an IP in the setup would help a lot.

Please see here:

 

[Masterwishx]

@ich777 can i add --collector.systemd --collector.processes to node exporter .cfg for full view data in grafana  like author of 1860 

wrote :

(Recommended for prometheus-node-exporter the arguments '--collector.systemd --collector.processes' because the graph uses some of their metrics.)

 

[ich777]

24 minutes ago, Masterwishx said:

can i add --collector.systemd --collector.processes to node exporter .cfg for full view data in grafana  like author of 1860 

wrote

Yes, why not?

 

But I don‘t understand why you would add —collector.systemd since Unraid doesn‘t use systemd

[Masterwishx]

12 minutes ago, ich777 said:

But I don‘t understand why you would add —collector.systemd since Unraid doesn‘t use systemd

 

Thanks for tip ...

i only asked becose dont have a knowledge about it , trying to learn more about promethus right now ...

so only processes i can add ...

[ich777]

12 hours ago, Masterwishx said:

i only asked becose dont have a knowledge about it , trying to learn more about promethus right now ...

This has strictly speaking nothing to do with the exporter itself this is more of a Unraid thing because Unraid doesn't use systemd.

 

12 hours ago, Masterwishx said:

so only processes i can add ...

You can try of course to add both but I don't think that you have any benefit when adding the systemd parameter because as said above Unraid doesn't use systemd.

[Bitbass]

Is there a way to get the Node Exporter to report on HDD temps? It looks like it has all of the other smartd metrics but I don't see temp.

[ich777]

1 hour ago, Bitbass said:

Is there a way to get the Node Exporter to report on HDD temps? It looks like it has all of the other smartd metrics but I don't see temp.

Have you yet seen this: Click

 

I have not released this to the public because I have no testers out there and this it is in "beta"...

 

You can install it by simply putting this line into the plugin installation:

https://raw.githubusercontent.com/ich777/unraid-prometheus_smartctl_exporter/master/prometheus_smartctl_exporter.plg

 

[Bitbass]

1 hour ago, ich777 said:

Have you yet seen this: Click

 

I have not released this to the public because I have no testers out there and this it is in "beta"...

 

You can install it by simply putting this line into the plugin installation:

https://raw.githubusercontent.com/ich777/unraid-prometheus_smartctl_exporter/master/prometheus_smartctl_exporter.plg

 

Check that out! Works like a charm, although Grafana is reporting one of the drives at 119c. It shows up as 33c in the Unraid dashboard and in the actually Need Exporter Metrics page. Not sure what's going on there.

[Bitbass]

24 minutes ago, Bitbass said:

Check that out! Works like a charm, although Grafana is reporting one of the drives at 119c. It shows up as 33c in the Unraid dashboard and in the actually Need Exporter Metrics page. Not sure what's going on there.

 

24 minutes ago, Bitbass said:

Check that out! Works like a charm, although Grafana is reporting one of the drives at 119c. It shows up as 33c in the Unraid dashboard and in the actually Need Exporter Metrics page. Not sure what's going on there.

For some reason, "value" is reporting incorrectly in the exporter but "raw_value" is correct. I'll let this run for a while and let you know if I have any problems. Thanks for making this!

[Bitbass]

Is there an array space metric that I'm missing? I currently capture array used and total space through telegraph (I think) using "disk" and "/mnt/user" and pulling the space from there. Trying to move to Prometheus entirely. 

[ich777]

2 minutes ago, Bitbass said:

Is there an array space metric that I'm missing?

I'm not entirely sure what you want to know here, you have to add all "diskX" to get the Array size.

[Bitbass]

Currently have this in my dashboard: The query is pulling out of Influx.

[ich777]

Just now, Bitbass said:

Currently have this in my dashboard: The query is pulling out of Influx.

I'm really not that familiar with Influx that I can help here...

Maybe someone else has a clue...

[Bitbass]

Not so concerned about Influx. I want to replicate this with the Prometheus plugin, if I can. I figured this out but it's only available bytes. I guess I can do a calculation with the total bytes.

[Bitbass]

Took a little noodling but I got to where I wanted to be:

[ich777]

4 minutes ago, Bitbass said:

Took a little noodling but I got to where I wanted to be:

Wouldn't it be better to use /mnt/user0?

That should be actually the Array, or at least all disks without any cache drive.

[Bitbass]

You're probably right, but I'm going off of what I had before. I'll look at changing it.