alturismo Posted July 6 Share Posted July 6 2 hours ago, MoonshineMagician said: I attached a diagnostics from a "clean" system state, I've uninstalled the plugin already. it looks like you have macvlan call traces Jul 3 03:58:24 Exos kernel: ------------[ cut here ]------------ Jul 3 03:58:24 Exos kernel: WARNING: CPU: 3 PID: 17973 at net/netfilter/nf_conntrack_core.c:1210 __nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack] Jul 3 03:58:24 Exos kernel: Modules linked in: af_packet bluetooth ecdh_generic ecc udp_diag wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha macvlan xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter dm_crypt dm_mod md_mod zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag nct6775 nct6775_core hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc bonding tls i915 intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm iosf_mbi drm_buddy i2c_algo_bit ttm drm_display_helper drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel Jul 3 03:58:24 Exos kernel: ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 aesni_intel drm crypto_simd cryptd mei_hdcp mei_pxp rapl intel_cstate i2c_i801 intel_gtt nvme mei_me i2c_smbus intel_wmi_thunderbolt wmi_bmof agpgart e1000e ahci intel_uncore syscopyarea nvme_core mei i2c_core joydev libahci sysfillrect sysimgblt fb_sys_fops video tpm_crb tpm_tis tpm_tis_core wmi tpm backlight intel_pmc_core acpi_pad acpi_tad button unix Jul 3 03:58:24 Exos kernel: CPU: 3 PID: 17973 Comm: kworker/u24:1 Tainted: P U O 6.1.79-Unraid #1 Jul 3 03:58:24 Exos kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B460M Pro4, BIOS P1.60 07/29/2021 Jul 3 03:58:24 Exos kernel: Workqueue: events_unbound macvlan_process_broadcast [macvlan] Jul 3 03:58:24 Exos kernel: RIP: 0010:__nf_conntrack_confirm+0xa4/0x2b0 [nf_conntrack] Jul 3 03:58:24 Exos kernel: Code: 44 24 10 e8 e2 e1 ff ff 8b 7c 24 04 89 ea 89 c6 89 04 24 e8 7e e6 ff ff 84 c0 75 a2 48 89 df e8 9b e2 ff ff 85 c0 89 c5 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 18 dd ff ff e8 93 e3 ff ff e9 72 01 Jul 3 03:58:24 Exos kernel: RSP: 0018:ffffc900001fcd98 EFLAGS: 00010202 Jul 3 03:58:24 Exos kernel: RAX: 0000000000000001 RBX: ffff88824101a300 RCX: 0bb6741e754bc2f0 Jul 3 03:58:24 Exos kernel: RDX: 0000000000000000 RSI: 00000000000002c4 RDI: ffff88824101a300 ... .. . you are 1/ using a bond with 1 NIC only (useless ...) 2/ bridging enabled 3/ may using a Fritz ? 192.168.178.... known issue 6.12.0 - solved in 6.12.4 in terms you are german, take a look at the german part under Anleitungen / macvlan call traces or the unraid docs or the update changelog from 6.12.4 (when i remember correctly) 2 Quote Link to comment
ich777 Posted July 6 Author Share Posted July 6 4 hours ago, MoonshineMagician said: Should it happen again I'll try to get a syslog out, any other logs I should attempt to pull manually? As @alturismo already pointed out your network is configured incorrectly and that what's causing the issues and not the plugin. You either can configure: Network Settings: Bridge enabled Docker Settings: IPVLAN Network Settings: Bridge disabled Docker Settings: MACVLAN In your case you have configured Bridge enabled with Docker Settings set to MACVLAN which will always crash the server at some point. 1 Quote Link to comment
T_Matz Posted July 7 Share Posted July 7 I have the ad guard exporter up and running, it starts just fine but I can't get it to work in Prometheus. The link is always down. I added the target to the yaml. I have other exporters up and they work with zero issues so not sure what the problem is. Ad guard runs on a stand alone device with a static IP. I'm able to ping ad guard from unraid. The exported is running. It's added to the Prometheus yaml. The one thing I can't confirm is that I was told it might not work if there are special characters in the pass word, is this still a fact or is this a question better suited on the Prometheus support page? Any insight anyone can provide would be greatly appreciated Quote Link to comment
ich777 Posted July 7 Author Share Posted July 7 11 minutes ago, T_Matz said: Any insight anyone can provide would be greatly appreciated Special characters are still an issue. Does the exporter page from the exporter itself display anything (IP:PORT/metrics in the description from the plugin)? Quote Link to comment
MoonshineMagician Posted July 7 Share Posted July 7 On 7/6/2024 at 6:28 AM, alturismo said: in terms you are german, take a look at the german part under Anleitungen / macvlan call traces or the unraid docs or the update changelog from 6.12.4 (when i remember correctly) I took a look at it. I'm connected to a fritzbox, yeah. To get WireGuard running a long time ago I messed with my network settings a bit and after it finally worked as expected I never touched them again. With the new implementation I changed it to bonding & bridging disabled now. I'm on 6.12.10, so I guess I should be all good. What I don't understand is yes I had the macvlan call trace, but the system worked fine after that nevertheless. Maybe the plugin puts some stress on the networking and at some point it collapses with the prometheus plugin installed. Anyways I've reinstalled the plugin now and I hope it works as expected, should I see issues again I'll try to get logs manually and report back. Thanks to you two! 1 Quote Link to comment
ich777 Posted July 7 Author Share Posted July 7 2 minutes ago, MoonshineMagician said: What I don't understand is yes I had the macvlan call trace, but the system worked fine after that nevertheless. Maybe the plugin puts some stress on the networking and at some point it collapses with the prometheus plugin installed. No, not the plugin is what causes these issues it's actually the newer Kernel. BTW, it's never good to have a Kernel trace in the log. Quote Link to comment
alturismo Posted July 7 Share Posted July 7 15 minutes ago, MoonshineMagician said: What I don't understand is yes I had the macvlan call trace, but the system worked fine after that nevertheless. Maybe the plugin puts some stress on the networking and at some point it collapses with the prometheus plugin installed. it somehow depends on the traffic, i could easily enforce the failure with tvheadend live tv streams ... withing seconds i had the traces, with regular idle usage its been a lottery, some hours, some days, some weeks until it crashed, but finally solved through limetech and we are good now. 1 Quote Link to comment
T_Matz Posted July 8 Share Posted July 8 19 hours ago, ich777 said: Special characters are still an issue. Does the exporter page from the exporter itself display anything (IP:PORT/metrics in the description from the plugin)? I'm not quite sure what you mean. Here is a snap shot of the exporter. I've read that changing the password is a pain but I'll work through that and see if it makes a difference. Quote Link to comment
ich777 Posted July 8 Author Share Posted July 8 7 hours ago, T_Matz said: I'm not quite sure what you mean. Here is a snap shot of the exporter. Is there any output if you visit the site? Quote Link to comment
T_Matz Posted July 8 Share Posted July 8 3 hours ago, ich777 said: Is there any output if you visit the site? Nope, states site can't be reached. Quote Link to comment
ich777 Posted July 8 Author Share Posted July 8 1 minute ago, T_Matz said: Nope, states site can't be reached. On the exporter plugin page it says Running correct? If that is the case something fundamentally is not working. Have you yet tried if the Docker container is working: Quote Link to comment
T_Matz Posted July 8 Share Posted July 8 5 hours ago, ich777 said: The exporter does state that it is running. Ad guard home is up and working. I'm going to uninstall everything and delete the folders reinstall and see if that fixes it. Quick question, the exporter connects to ad guard home via port 80 or the 9617? Quote Link to comment
ich777 Posted July 8 Author Share Posted July 8 5 minutes ago, T_Matz said: Quick question, the exporter connects to ad guard home via port 80 or the 9617? You see that right here in your screenshot: 1 Quote Link to comment
nebb00 Posted July 9 Share Posted July 9 is it possible that this plugin does/can support unraid's apcupsd? Quote Link to comment
ich777 Posted July 9 Author Share Posted July 9 7 minutes ago, nebb00 said: is it possible that this plugin does/can support unraid's apcupsd? Which plugin? The node exporter doesn't support it AFAIK. I don't have any plans to release a plugin that supports apcupsd. Quote Link to comment
Jarsky Posted July 14 Share Posted July 14 On 5/3/2024 at 5:32 PM, ich777 said: I look into that and report back, please give me a few days. Is the interface also hosted at pi.hole or is it hosted at pi.hole/admin ? Apologies I never replied. The new v6 Pi-Hole has the API @ pi.hole/api while the admin interface is still pi.hole/admin Quote Link to comment
ich777 Posted July 14 Author Share Posted July 14 14 hours ago, Jarsky said: Apologies I never replied. No worries, sadly enough I have to deprecate the AdGuard Exporter since it seems the maintainer no longer supports it... Quote Link to comment
7eventy7 Posted August 13 Share Posted August 13 How can I add paths to the node_filesystem_files metric? I would like to monitor my media libraries but I'm not sure how to get the metric exported, thanks in advance for any help Quote Link to comment
ich777 Posted August 13 Author Share Posted August 13 2 hours ago, 7eventy7 said: How can I add paths to the node_filesystem_files metric? I would like to monitor my media libraries but I'm not sure how to get the metric exported, thanks in advance for any help I think that's a bit complicated to do since you have to write a custom script that actually watches the folder(s) that you want to watch, write it to a file so that the exporter is actually able to pick the metrics up. EDIT: I think that if you want to do something like you’ve requested on another system it is pretty much the same and involves also a custom script. Quote Link to comment
7eventy7 Posted August 13 Share Posted August 13 4 hours ago, ich777 said: I think that's a bit complicated to do since you have to write a custom script that actually watches the folder(s) that you want to watch, write it to a file so that the exporter is actually able to pick the metrics up. EDIT: I think that if you want to do something like you’ve requested on another system it is pretty much the same and involves also a custom script. Thanks for the lead! Any guidance on how I can write such a script? Examples maybe? Quote Link to comment
ich777 Posted August 13 Author Share Posted August 13 57 minutes ago, 7eventy7 said: Any guidance on how I can write such a script? Examples maybe? TBH I never played around with that, I think your best resource will be the Node Exporter GitHub repo here. Quote Link to comment
7eventy7 Posted August 13 Share Posted August 13 19 minutes ago, ich777 said: TBH I never played around with that, I think your best resource will be the Node Exporter GitHub repo here. I managed to get a script that I believe is working, but I'm stuck trying to figure out how to edit the prometheus.yml file for node exporter to add the metrics I created. Any idea how I can access, and edit this file to include my newly created metrics? #!/usr/bin/env sh # # Expose directory usage metrics for specified directories. # # Usage: add this to crontab: # # */5 * * * * /path/to/directory-size.sh | sponge /tmp/media_directory_size.prom # # Author: Original: Antoine Beaupré <[email protected]> # Modified for specific directories for Sienna # List of directories to monitor directories=( "/mnt/user/media/Movies" "/mnt/user/media/Shows" "/mnt/user/media/Music" "/mnt/user/media/Clips" ) # Print Prometheus metrics header echo "# HELP node_directory_size_bytes Disk space used by specified directories" echo "# TYPE node_directory_size_bytes gauge" # Loop through each directory and get its size for dir in "${directories[@]}"; do du --block-size=1 --summarize "$dir" \ | awk '{ print "node_directory_size_bytes{directory=\"" $2 "\"} " $1 }' done Quote Link to comment
ich777 Posted August 13 Author Share Posted August 13 28 minutes ago, 7eventy7 said: Any idea how I can access That's not how it works... First you have to edit your script a bit since there are some things in there which are not set up properly: 1) The command sponge is not available on Unraid and will ultimately fail: 30 minutes ago, 7eventy7 said: # */5 * * * * /path/to/directory-size.sh | sponge /tmp/media_directory_size.prom 2) I would recommend that you put the file in a subdirectory in /tmp like: /tmp/node_exporter/media_directory_size.prom You have to of course make sure that the directory exists before you can write anything to it, if you add something like this at the beginning of your script will do the trick: if [ ! -d /tmp/node_exporter ]; then mkdir -p /tmp/node_exporter fi 3) Edit the file: /boot/config/plugins/prometheus_node_exporter/settings.cfg so that it looks like: start_parameters=--collector.textfile.directory=/tmp/node_exporter This will start the node_exporter with the parameter appended on the next reboot and tell node_exporter to look for .prom files in that specific directory and include it in it's metrics which will be polled by Prometheus. Hope that makes sense, I really can't help any further since this is almost a step by step guide what's missing. Quote Link to comment
7eventy7 Posted August 13 Share Posted August 13 20 minutes ago, ich777 said: That's not how it works... First you have to edit your script a bit since there are some things in there which are not set up properly: 1) The command sponge is not available on Unraid and will ultimately fail: 2) I would recommend that you put the file in a subdirectory in /tmp like: /tmp/node_exporter/media_directory_size.prom You have to of course make sure that the directory exists before you can write anything to it, if you add something like this at the beginning of your script will do the trick: if [ ! -d /tmp/node_exporter ]; then mkdir -p /tmp/node_exporter fi 3) Edit the file: /boot/config/plugins/prometheus_node_exporter/settings.cfg so that it looks like: start_parameters=--collector.textfile.directory=/tmp/node_exporter This will start the node_exporter with the parameter appended on the next reboot and tell node_exporter to look for .prom files in that specific directory and include it in it's metrics which will be polled by Prometheus. Hope that makes sense, I really can't help any further since this is almost a step by step guide what's missing. You have been so immensely helpful I can't thank you enough. This step by step was the most clear and easy to follow guide on how to set this up, thank you so much for writing it. I got it working and now I can make my pretty media storage panel in grafana! 1 Quote Link to comment
Recommended Posts
Posted by SpencerJ,
1 reaction
Go to this post
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.