[Plugin] Prometheus unRAID Plugins

[alturismo]

2 hours ago, MoonshineMagician said:

I attached a diagnostics from a "clean" system state, I've uninstalled the plugin already.

 

it looks like you have macvlan call traces

 

Jul  3 03:58:24 Exos kernel: ------------[ cut here ]------------
Jul  3 03:58:24 Exos kernel: WARNING: CPU: 3 PID: 17973 at net/netfilter/nf_conntrack_core.c:1210 __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Jul  3 03:58:24 Exos kernel: Modules linked in: af_packet bluetooth ecdh_generic ecc udp_diag wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha macvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter dm_crypt dm_mod md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel
Jul  3 03:58:24 Exos kernel: ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel drm crypto_simd cryptd mei_hdcp mei_pxp rapl intel_cstate i2c_i801 intel_gtt nvme mei_me i2c_smbus intel_wmi_thunderbolt wmi_bmof agpgart e1000e ahci intel_uncore syscopyarea nvme_core mei i2c_core joydev libahci sysfillrect sysimgblt fb_sys_fops video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_pad acpi_tad button unix
Jul  3 03:58:24 Exos kernel: CPU: 3 PID: 17973 Comm: kworker/u24:1 Tainted: P     U     O       6.1.79-Unraid #1
Jul  3 03:58:24 Exos kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B460M Pro4, BIOS P1.60 07/29/2021
Jul  3 03:58:24 Exos kernel: Workqueue: events_unbound macvlan_process_broadcast [macvlan]
Jul  3 03:58:24 Exos kernel: RIP: 0010:__nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack]
Jul  3 03:58:24 Exos kernel: Code: 44 24 10 e8 e2 e1 ff ff 8b 7c 24 04 89 ea 89 c6 89 04 24 e8 7e e6 ff ff 84 c0 75 a2 48 89 df e8 9b e2 ff ff 85 c0 89 c5 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 18 dd ff ff e8 93 e3 ff ff e9 72 01
Jul  3 03:58:24 Exos kernel: RSP: 0018:ffffc900001fcd98 EFLAGS: 00010202
Jul  3 03:58:24 Exos kernel: RAX: 0000000000000001 RBX: ffff88824101a300 RCX: 0bb6741e754bc2f0
Jul  3 03:58:24 Exos kernel: RDX: 0000000000000000 RSI: 00000000000002c4 RDI: ffff88824101a300
...
  ..
  .

 

you are

 

1/ using a bond with 1 NIC only (useless ...)

2/ bridging enabled

3/ may using a Fritz ? 192.168.178.... known issue 6.12.0 - solved in 6.12.4

 

in terms you are german, take a look at the german part under Anleitungen / macvlan call traces or the unraid docs or the update changelog from 6.12.4 (when i remember correctly)

[ich777]

4 hours ago, MoonshineMagician said:

Should it happen again I'll try to get a syslog out, any other logs I should attempt to pull manually?

As @alturismo already pointed out your network is configured incorrectly and that what's causing the issues and not the plugin.

 

You either can configure:

1. Network Settings: Bridge enabled
   Docker Settings: IPVLAN
    
2. Network Settings: Bridge disabled
   Docker Settings: MACVLAN

In your case you have configured Bridge enabled with Docker Settings set to MACVLAN which will always crash the server at some point.

[T_Matz]

I have the ad guard exporter up and running, it starts just fine but I can't get it to work in Prometheus. The link is always down. I added the target to the yaml. I have other exporters up and they work with zero issues so not sure what the problem is.

 

Ad guard runs on a stand alone device with a static IP. I'm able to ping ad guard from unraid. The exported is running. It's added to the Prometheus yaml. The one thing I can't confirm is that I was told it might not work if there are special characters in the pass word, is this still a fact or is this a question better suited on the Prometheus support page? Any insight anyone can provide would be greatly appreciated 

[ich777]

11 minutes ago, T_Matz said:

Any insight anyone can provide would be greatly appreciated 

Special characters are still an issue.

 

Does the exporter page from the exporter itself display anything (IP:PORT/metrics in the description from the plugin)?

[MoonshineMagician]

On 7/6/2024 at 6:28 AM, alturismo said:

in terms you are german, take a look at the german part under Anleitungen / macvlan call traces or the unraid docs or the update changelog from 6.12.4 (when i remember correctly)

I took a look at it. I'm connected to a fritzbox, yeah. To get WireGuard running a long time ago I messed with my network settings a bit and after it finally worked as expected I never touched them again. With the new implementation I changed it to bonding & bridging disabled now. I'm on 6.12.10, so I guess I should be all good.

What I don't understand is yes I had the macvlan call trace, but the system worked fine after that nevertheless. Maybe the plugin puts some stress on the networking and at some point it collapses with the prometheus plugin installed.

Anyways I've reinstalled the plugin now and I hope it works as expected, should I see issues again I'll try to get logs manually and report back. Thanks to you two!

[ich777]

2 minutes ago, MoonshineMagician said:

What I don't understand is yes I had the macvlan call trace, but the system worked fine after that nevertheless. Maybe the plugin puts some stress on the networking and at some point it collapses with the prometheus plugin installed.

No, not the plugin is what causes these issues it's actually the newer Kernel.

BTW, it's never good to have a Kernel trace in the log.

[alturismo]

15 minutes ago, MoonshineMagician said:

What I don't understand is yes I had the macvlan call trace, but the system worked fine after that nevertheless. Maybe the plugin puts some stress on the networking and at some point it collapses with the prometheus plugin installed.

it somehow depends on the traffic, i could easily enforce the failure with tvheadend live tv streams ... withing seconds i had the traces, with regular idle usage its been a lottery, some hours, some days, some weeks until it crashed, but finally solved through limetech and we are good now.

[T_Matz]

19 hours ago, ich777 said:

Special characters are still an issue.

 

Does the exporter page from the exporter itself display anything (IP:PORT/metrics in the description from the plugin)?

I'm not quite sure what you mean. Here is a snap shot of the exporter. 

I've read that changing the password is a pain but I'll work through that and see if it makes a difference.

[ich777]

7 hours ago, T_Matz said:

I'm not quite sure what you mean. Here is a snap shot of the exporter. 

 

Is there any output if you visit the site?

[T_Matz]

3 hours ago, ich777 said:

 

Is there any output if you visit the site?

Nope, states site can't be reached.

[ich777]

1 minute ago, T_Matz said:

Nope, states site can't be reached.

On the exporter plugin page it says Running correct?

If that is the case something fundamentally is not working.

 

Have you yet tried if the Docker container is working:

[T_Matz]

5 hours ago, ich777 said:

 

The exporter does state that it is running. Ad guard home is up and working. I'm going to uninstall everything and delete the folders reinstall and see if that fixes it.

Quick question, the exporter connects to ad guard home via port 80 or the 9617? 

[ich777]

5 minutes ago, T_Matz said:

Quick question, the exporter connects to ad guard home via port 80 or the 9617? 

You see that right here in your screenshot:

[nebb00]

is it possible that this plugin does/can support unraid's apcupsd?

[ich777]

7 minutes ago, nebb00 said:

is it possible that this plugin does/can support unraid's apcupsd?

Which plugin? The node exporter doesn't support it AFAIK.

 

I don't have any plans to release a plugin that supports apcupsd.

[Jarsky]

On 5/3/2024 at 5:32 PM, ich777 said:

I look into that and report back, please give me a few days.

 

Is the interface also hosted at pi.hole or is it hosted at pi.hole/admin ?

 

Apologies I never replied. 

The new v6 Pi-Hole has the API @ pi.hole/api while the admin interface is still pi.hole/admin

[ich777]

14 hours ago, Jarsky said:

Apologies I never replied. 

No worries, sadly enough I have to deprecate the AdGuard Exporter since it seems the maintainer no longer supports it...

[7eventy7]

How can I add paths to the node_filesystem_files metric? I would like to monitor my media libraries but I'm not sure how to get the metric exported, thanks in advance for any help

[ich777]

2 hours ago, 7eventy7 said:

How can I add paths to the node_filesystem_files metric? I would like to monitor my media libraries but I'm not sure how to get the metric exported, thanks in advance for any help

I think that's a bit complicated to do since you have to write a custom script that actually watches the folder(s) that you want to watch, write it to a file so that the exporter is actually able to pick the metrics up.

 

EDIT: I think that if you want to do something like you’ve requested on another system it is pretty much the same and involves also a custom script.

[7eventy7]

4 hours ago, ich777 said:

I think that's a bit complicated to do since you have to write a custom script that actually watches the folder(s) that you want to watch, write it to a file so that the exporter is actually able to pick the metrics up.

 

EDIT: I think that if you want to do something like you’ve requested on another system it is pretty much the same and involves also a custom script.

Thanks for the lead! Any guidance on how I can write such a script? Examples maybe?

[ich777]

57 minutes ago, 7eventy7 said:

Any guidance on how I can write such a script? Examples maybe?

TBH I never played around with that, I think your best resource will be the Node Exporter GitHub repo here.

[7eventy7]

19 minutes ago, ich777 said:

TBH I never played around with that, I think your best resource will be the Node Exporter GitHub repo here.

I managed to get a script that I believe is working, but I'm stuck trying to figure out how to edit the prometheus.yml file for node exporter to add the metrics I created. Any idea how I can access, and edit this file to include my newly created metrics?

 

#!/usr/bin/env sh
#
# Expose directory usage metrics for specified directories.
#
# Usage: add this to crontab:
#
# */5 * * * * /path/to/directory-size.sh | sponge /tmp/media_directory_size.prom
#
# Author: Original: Antoine Beaupré <[email protected]>
# Modified for specific directories for Sienna

# List of directories to monitor
directories=(
  "/mnt/user/media/Movies"
  "/mnt/user/media/Shows"
  "/mnt/user/media/Music"
  "/mnt/user/media/Clips"
)

# Print Prometheus metrics header
echo "# HELP node_directory_size_bytes Disk space used by specified directories"
echo "# TYPE node_directory_size_bytes gauge"

# Loop through each directory and get its size
for dir in "${directories[@]}"; do
  du --block-size=1 --summarize "$dir" \
    | awk '{ print "node_directory_size_bytes{directory=\"" $2 "\"} "  $1 }'
done

 

[ich777]

28 minutes ago, 7eventy7 said:

Any idea how I can access

That's not how it works...

 

First you have to edit your script a bit since there are some things in there which are not set up properly:

 

1) The command sponge is not available on Unraid and will ultimately fail:

30 minutes ago, 7eventy7 said:

# */5 * * * * /path/to/directory-size.sh | sponge /tmp/media_directory_size.prom

 

2) I would recommend that you put the file in a subdirectory in /tmp like:

/tmp/node_exporter/media_directory_size.prom

You have to of course make sure that the directory exists before you can write anything to it, if you add something like this at the beginning of your script will do the trick:

if [ ! -d /tmp/node_exporter ]; then
  mkdir -p /tmp/node_exporter
fi

 

 

3) Edit the file: /boot/config/plugins/prometheus_node_exporter/settings.cfg so that it looks like:

start_parameters=--collector.textfile.directory=/tmp/node_exporter

This will start the node_exporter with the parameter appended on the next reboot and tell node_exporter to look for .prom files in that specific directory and include it in it's metrics which will be polled by Prometheus.

 

 

Hope that makes sense, I really can't help any further since this is almost a step by step guide what's missing.

[7eventy7]

20 minutes ago, ich777 said:

That's not how it works...

 

First you have to edit your script a bit since there are some things in there which are not set up properly:

 

1) The command sponge is not available on Unraid and will ultimately fail:

 

2) I would recommend that you put the file in a subdirectory in /tmp like:

/tmp/node_exporter/media_directory_size.prom

You have to of course make sure that the directory exists before you can write anything to it, if you add something like this at the beginning of your script will do the trick:

if [ ! -d /tmp/node_exporter ]; then
  mkdir -p /tmp/node_exporter
fi

 

 

3) Edit the file: /boot/config/plugins/prometheus_node_exporter/settings.cfg so that it looks like:

start_parameters=--collector.textfile.directory=/tmp/node_exporter

This will start the node_exporter with the parameter appended on the next reboot and tell node_exporter to look for .prom files in that specific directory and include it in it's metrics which will be polled by Prometheus.

 

 

Hope that makes sense, I really can't help any further since this is almost a step by step guide what's missing.

You have been so immensely helpful I can't thank you enough. This step by step was the most clear and easy to follow guide on how to set this up, thank you so much for writing it. I got it working and now I can make my pretty media storage panel in grafana!