MatuScien Posted April 13 Share Posted April 13 6 hours ago, ich777 said: Did you yet restart your Docker service? Sorry, I didn't answer you. Yes, I have restarted it without success. Quote Link to comment
ich777 Posted April 13 Author Share Posted April 13 24 minutes ago, MatuScien said: Sorry, I didn't answer you. Yes, I have restarted it without success. And what does the prometheus page tell you? Is the exporter online or offline? The pihole exporter is definitely working because you get a output from /metrics I think there is something wrong with the access but I really can‘t tell you where the issue in your case is… Quote Link to comment
Jarsky Posted May 3 Share Posted May 3 I'm using the new v6 of Pi-Hole and the Prometheus Pi-Hole Exporter plugin. It's unable to connect to the API; and theres no configuration option in the plugin to change the target address 2024-05-03 15:56:11.607 WARNING API: Bad request (The API is hosted at pi.hole/api, not pi.hole/admin/api) 2024-05-03 15:56:26.604 WARNING API: Bad request (The API is hosted at pi.hole/api, not pi.hole/admin/api) 2024-05-03 15:56:29.006 WARNING API: Bad request (The API is hosted at pi.hole/api, not pi.hole/admin/api) 2024-05-03 15:56:39.005 WARNING API: Bad request (The API is hosted at pi.hole/api, not pi.hole/admin/api) Quote Link to comment
ich777 Posted May 3 Author Share Posted May 3 1 hour ago, Jarsky said: I'm using the new v6 of Pi-Hole and the Prometheus Pi-Hole Exporter plugin. I look into that and report back, please give me a few days. Is the interface also hosted at pi.hole or is it hosted at pi.hole/admin ? Quote Link to comment
deicist Posted May 30 Share Posted May 30 Hi @ich777 first off, thanks for these great plugins. For the unraid prometheus exporter, is there any way to include the disk temperatures? I presume they're part of the SMART data rather than the hardware temperatures (the CPU & motherboard temps are included in the prometheus export). Quote Link to comment
aldumil Posted June 14 Share Posted June 14 Hello @ich777! What a wonderfully helpful walkthrough/tutorial! I appreciate it! Do you still have a link to a 'buy me a coffee' thing? I'd love to show my appreciation, I thought I saw one at some point, but can't find it again. I have the adguard exporter working perfectly, but I can't get the pi-hole exporter to work in addition. Should I not expect them both to work? Quote Link to comment
ich777 Posted June 14 Author Share Posted June 14 1 hour ago, aldumil said: I have the adguard exporter working perfectly, but I can't get the pi-hole exporter to work in addition. Should I not expect them both to work? Everything should work depending on your configuration, can you please share a bit more details? Logs, configuration, screenshots,... Quote Link to comment
Hadda Posted June 15 Share Posted June 15 (edited) Hi @ich777 I have tried to get the AdGuard Exporter to run, but it does not start. AdGuard runs on br0 and has its own IP. I have activated Host access to custom networks in the Docker settings. In the configuration of the exporter I have entered the fixed IP and all other data but the status is always stopped. Unfortunately there is also nothing in the Unraid logs. Do you have any ideas as to why this could be? I am still very new to the game 😅 Edit: In the prometheus.yml I added a target with the fixed IP from the AdGuard Container with :9617 Edited June 15 by Hadda Added missing informations Quote Link to comment
ich777 Posted June 15 Author Share Posted June 15 7 hours ago, Hadda said: Edit: In the prometheus.yml I added a target with the fixed IP from the AdGuard Container with :9617 What giving you this URL back? 7 hours ago, Hadda said: AdGuard runs on br0 and has its own IP. I have activated Host access to custom networks in the Docker settings. In the configuration of the exporter I have entered the fixed IP and all other data but the status is always stopped. Unfortunately there is also nothing in the Unraid logs. Do you have any ideas as to why this could be? Not really since two posts above yours a users is saying that AdGuard works perfectly fine. How did you configure the exporter? Quote Link to comment
aldumil Posted June 16 Share Posted June 16 On 6/14/2024 at 4:05 PM, ich777 said: Everything should work depending on your configuration, can you please share a bit more details? Logs, configuration, screenshots,... Just a screenshot of the pi-hole exporter not running, I can't find any logs that have any information about this. I checked the syslog, the prometheus container log, as far as I can tell their aren't plugin specific logs (?). I suspect the issue is on my end, if I got to the pi-hole address (192.168.54.101:80) I get a 'placeholder' page for lighttpd. My pi-hole is running in an LXC on proxmox, the API address is http://192.168.54.101/admin/api.php?. Is that all typical and I've just messed something else up somewhere? Thanks for any direction you can offer! Quote Link to comment
ich777 Posted June 16 Author Share Posted June 16 2 hours ago, aldumil said: I suspect the issue is on my end, if I got to the pi-hole address (192.168.54.101:80) I get a 'placeholder' page for lighttpd. My pi-hole is running in an LXC on proxmox, the API address is http://192.168.54.101/admin/api.php?. Is that all typical and I've just messed something else up somewhere? Seems like it since you should get the PiHole page. Have you yet considered to set up PiHole in a LXC container on Unraid because that would way easier. If you want to try it, install the LXC plugin from the CA App, configure it, visit the LXC thread and read this post: (this is a tutorial on how to set it up the easiest way) Quote Link to comment
Hadda Posted June 17 Share Posted June 17 On 6/15/2024 at 10:04 PM, ich777 said: What giving you this URL back? Firefox says it's unable to connect and prometheus says Get "http://192.168.178.42:9617/metrics": dial tcp 192.168.178.42:9617: connect: no route to host On 6/15/2024 at 10:04 PM, ich777 said: How did you configure the exporter? This is the Exporter configuration. The container config And the prometheus.yml Quote Link to comment
ich777 Posted June 17 Author Share Posted June 17 6 minutes ago, Hadda said: Firefox says it's unable to connect and prometheus says This is caused because the exporter is not running, this will only work if the exporter is actually running. 6 minutes ago, Hadda said: The container config How is the network configured on Unraid? Host Access is on correct? Try to restart (disable/enable) the Docker service and see if it is working. My recommendation is still to run AdGuard or PiHole in a LXC container because it simplifies the setup by a lot. EDIT: I saw also an issue in your prometheus.yml, all exporters are running on the Unraid IP, therefore you always have to specify the Unraid IP not your AdGuard IP. So to speak the exporter IP in your case is 192.168.178.132 with port 9617 Quote Link to comment
Hadda Posted June 18 Share Posted June 18 On 6/17/2024 at 8:18 AM, ich777 said: How is the network configured on Unraid? Did you mean these settings? Host access is activated and i have already tried to restarting Docker, but no effect. On 6/17/2024 at 8:18 AM, ich777 said: EDIT: I saw also an issue in your prometheus.yml, all exporters are running on the Unraid IP, therefore you always have to specify the Unraid IP not your AdGuard IP. So to speak the exporter IP in your case is 192.168.178.132 with port 9617 I've changed the settings but also no effect. I will try the lxc container Quote Link to comment
ich777 Posted June 18 Author Share Posted June 18 6 minutes ago, Hadda said: I will try the lxc container A short description how to install a pre-made container is here: You just have to change the URL to the following if you want to install the AdGuard version: https://raw.githubusercontent.com/ich777/unraid_lxc_adguard/main/lxc_container_template.xml Quote Link to comment
MoonshineMagician Posted June 30 Share Posted June 30 Anyone else having problems with node exporter cpu load? Every now and then my system hangs and becomes completely unresponsive, even via SSH. Today I was luckily there right when it happened, and while the web ui was already down, I was still able to log in via SSH, although the login alone took the system 2 minutes. I'm not sure which of duplicati and node exporter is the real culprit, as both have a larger than normal cpu load. But I wanted to attach it here in case anyone else sees this behaviour. Unfortunately I only checked the dmesg (nothing there after boot, so nothing related to the hang), then htop and ran the diagnostic. I left the diagnostics tool running and the ssh connection stayed open for about 1 hour, but the diagnostic never finished before the system died completely. I'm running unraid 6.2.10, node exporter unraid 2024.04.24, duplicati is the linuxserver/duplicati image tagged with version 2.0.8. The system has these rare (every 2 weeks maybe) hang issues for quite a while now (>3 months), but usually I can't get a response when it happens and my last resort is to force a reboot. I did the syslog debug thing to the flash too on one hang in the past, but even the syslog doesn't show something when this happens. Bummer. Quote Link to comment
ich777 Posted July 1 Author Share Posted July 1 16 hours ago, MoonshineMagician said: I'm not sure which of duplicati and node exporter is the real culprit, as both have a larger than normal cpu load. But I wanted to attach it here in case anyone else sees this behaviour. It definitely looks like the node exporter is what causes trouble but TBH, I've never seen that behavior. In general the node exporter shouldn't cause any issues anyways because it just gathers system data, my suspicion is that it runs into some kind of edge case or some other, maybe hardware related, issue and this causes this behavior on your system. I can only tell you for now to remove the node exporter plugin and see if the issue pops up without it. When the issue is not occurring without the node exporter after a week or so I would recommend installing the node exporter again and see if it occurs again. Quote Link to comment
Samjam927 Posted July 3 Share Posted July 3 Anyone else having issues with the MSI Afterburner Exporter? I added the IP with port 9891 to the YML and have remote server running with the same IP on the VM with that app allowed through windows firewall. What am I missing here? I keep getting this error with nothing coming through: Quote Link to comment
MoonshineMagician Posted July 5 Share Posted July 5 On 7/1/2024 at 12:59 PM, ich777 said: It definitely looks like the node exporter is what causes trouble but TBH, I've never seen that behavior. In general the node exporter shouldn't cause any issues anyways because it just gathers system data, my suspicion is that it runs into some kind of edge case or some other, maybe hardware related, issue and this causes this behavior on your system. I can only tell you for now to remove the node exporter plugin and see if the issue pops up without it. When the issue is not occurring without the node exporter after a week or so I would recommend installing the node exporter again and see if it occurs again. Thank you for your reply! Surprisingly it happened again way quicker than usual, this time I even had some messages in the dmesg: [179512.581457] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 9-.... } 41962923 jiffies s: 226673 root: 0x200/. [179512.581468] rcu: blocking rcu_node structures (internal RCU debug): [179512.581471] Sending NMI from CPU 8 to CPUs 9: [179512.581476] NMI backtrace for cpu 9 [179512.581478] CPU: 9 PID: 4366 Comm: prometheus Tainted: P UD W O 6.1.79-Unraid #1 [179512.581479] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B460M Pro4, BIOS P1.60 07/29/2021 [179512.581480] RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581485] Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 c5 40 ff ff 49 c7 c4 80 ed [179512.581486] RSP: 0000:ffffc90026cffd80 EFLAGS: 00000202 [179512.581487] RAX: 0000000000000101 RBX: ffffea0008175ca8 RCX: 000fffffc0000000 [179512.581488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffea0008175ca8 [179512.581489] RBP: 0000000000000254 R08: 0000000000000000 R09: ffffc90026cffe38 [179512.581489] R10: ffff8882d357f408 R11: ffff8882d357f40c R12: ffff8884d2074100 [179512.581490] R13: 0000000000000000 R14: ffff8882ad595940 R15: ffff8882961da800 [179512.581491] FS: 000000c000800098(0000) GS:ffff888845a40000(0000) knlGS:0000000000000000 [179512.581492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [179512.581493] CR2: 000000c0018d5000 CR3: 00000001f5078006 CR4: 00000000003706e0 [179512.581494] Call Trace: [179512.581496] <NMI> [179512.581498] ? nmi_cpu_backtrace+0xd3/0x104 [179512.581501] ? nmi_cpu_backtrace_handler+0xd/0x15 [179512.581504] ? nmi_handle+0x54/0x131 [179512.581506] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581508] ? default_do_nmi+0x66/0x15b [179512.581510] ? exc_nmi+0xbf/0x130 [179512.581511] ? end_repeat_nmi+0x16/0x67 [179512.581514] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581515] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581516] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581517] </NMI> [179512.581517] <TASK> [179512.581518] do_raw_spin_lock+0x14/0x1a [179512.581520] pmd_lock.constprop.0+0x12/0x1b [179512.581523] huge_pmd_set_accessed+0x16/0x4e [179512.581525] __handle_mm_fault+0x417/0xcf9 [179512.581529] handle_mm_fault+0x13d/0x20f [179512.581531] do_user_addr_fault+0x2c3/0x48d [179512.581534] exc_page_fault+0xfb/0x11d [179512.581536] asm_exc_page_fault+0x22/0x30 [179512.581538] RIP: 0033:0x2a3fe42 [179512.581539] Code: fd 49 89 0b 48 8b 86 d0 00 00 00 49 89 43 08 48 89 8e d0 00 00 00 48 81 c4 d8 00 00 00 5d c3 48 8b b0 f0 00 00 00 4c 8b 5e 08 <41> 84 03 45 89 cc 41 81 e1 ff 01 00 00 47 0f b7 0c 4b 45 89 cb 41 [179512.581541] RSP: 002b:000000c00114f640 EFLAGS: 00010297 [179512.581541] RAX: 000000c000ba4280 RBX: 0000000000000000 RCX: 0000000000000017 [179512.581542] RDX: 0000000000000002 RSI: 000000c000ba4298 RDI: 000000c0015083c0 [179512.581543] RBP: 000000c00114f718 R08: 000000000000001f R09: 0000000073e16eb7 [179512.581544] R10: 000000c000ba4308 R11: 000000c0018d5000 R12: 0000000000616eb7 [179512.581544] R13: ffffffffffffffff R14: 000000c003224000 R15: 0000000000000007 [179512.581546] </TASK> I deactivated the plugin now and I'll see if it helps. A hardware update to 12th Gen Intel is also planned, should I get new information on this issue I'll report back. Quote Link to comment
alturismo Posted July 5 Share Posted July 5 19 minutes ago, MoonshineMagician said: Thank you for your reply! Surprisingly it happened again way quicker than usual, this time I even had some messages in the dmesg: may a question, plugin CoreFreq installed ? Quote Link to comment
ich777 Posted July 5 Author Share Posted July 5 1 hour ago, MoonshineMagician said: Thank you for your reply! Surprisingly it happened again way quicker than usual, this time I even had some messages in the dmesg: [179512.581457] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 9-.... } 41962923 jiffies s: 226673 root: 0x200/. [179512.581468] rcu: blocking rcu_node structures (internal RCU debug): [179512.581471] Sending NMI from CPU 8 to CPUs 9: [179512.581476] NMI backtrace for cpu 9 [179512.581478] CPU: 9 PID: 4366 Comm: prometheus Tainted: P UD W O 6.1.79-Unraid #1 [179512.581479] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B460M Pro4, BIOS P1.60 07/29/2021 [179512.581480] RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581485] Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 c5 40 ff ff 49 c7 c4 80 ed [179512.581486] RSP: 0000:ffffc90026cffd80 EFLAGS: 00000202 [179512.581487] RAX: 0000000000000101 RBX: ffffea0008175ca8 RCX: 000fffffc0000000 [179512.581488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffea0008175ca8 [179512.581489] RBP: 0000000000000254 R08: 0000000000000000 R09: ffffc90026cffe38 [179512.581489] R10: ffff8882d357f408 R11: ffff8882d357f40c R12: ffff8884d2074100 [179512.581490] R13: 0000000000000000 R14: ffff8882ad595940 R15: ffff8882961da800 [179512.581491] FS: 000000c000800098(0000) GS:ffff888845a40000(0000) knlGS:0000000000000000 [179512.581492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [179512.581493] CR2: 000000c0018d5000 CR3: 00000001f5078006 CR4: 00000000003706e0 [179512.581494] Call Trace: [179512.581496] <NMI> [179512.581498] ? nmi_cpu_backtrace+0xd3/0x104 [179512.581501] ? nmi_cpu_backtrace_handler+0xd/0x15 [179512.581504] ? nmi_handle+0x54/0x131 [179512.581506] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581508] ? default_do_nmi+0x66/0x15b [179512.581510] ? exc_nmi+0xbf/0x130 [179512.581511] ? end_repeat_nmi+0x16/0x67 [179512.581514] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581515] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581516] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581517] </NMI> [179512.581517] <TASK> [179512.581518] do_raw_spin_lock+0x14/0x1a [179512.581520] pmd_lock.constprop.0+0x12/0x1b [179512.581523] huge_pmd_set_accessed+0x16/0x4e [179512.581525] __handle_mm_fault+0x417/0xcf9 [179512.581529] handle_mm_fault+0x13d/0x20f [179512.581531] do_user_addr_fault+0x2c3/0x48d [179512.581534] exc_page_fault+0xfb/0x11d [179512.581536] asm_exc_page_fault+0x22/0x30 [179512.581538] RIP: 0033:0x2a3fe42 [179512.581539] Code: fd 49 89 0b 48 8b 86 d0 00 00 00 49 89 43 08 48 89 8e d0 00 00 00 48 81 c4 d8 00 00 00 5d c3 48 8b b0 f0 00 00 00 4c 8b 5e 08 <41> 84 03 45 89 cc 41 81 e1 ff 01 00 00 47 0f b7 0c 4b 45 89 cb 41 [179512.581541] RSP: 002b:000000c00114f640 EFLAGS: 00010297 [179512.581541] RAX: 000000c000ba4280 RBX: 0000000000000000 RCX: 0000000000000017 [179512.581542] RDX: 0000000000000002 RSI: 000000c000ba4298 RDI: 000000c0015083c0 [179512.581543] RBP: 000000c00114f718 R08: 000000000000001f R09: 0000000073e16eb7 [179512.581544] R10: 000000c000ba4308 R11: 000000c0018d5000 R12: 0000000000616eb7 [179512.581544] R13: ffffffffffffffff R14: 000000c003224000 R15: 0000000000000007 [179512.581546] </TASK> I deactivated the plugin now and I'll see if it helps. A hardware update to 12th Gen Intel is also planned, should I get new information on this issue I'll report back. Do you maybe also happen to have the syslog? It seems to be a Kernel panic. Quote Link to comment
MoonshineMagician Posted July 5 Share Posted July 5 9 hours ago, ich777 said: Do you maybe also happen to have the syslog? It seems to be a Kernel panic. Unfortunately I only got the dmesg out of it, the system was super unresponsive yet again. It didn't even accept a sudo reboot anymore. I restarted using the reset button on the case.. Quote Link to comment
MoonshineMagician Posted July 5 Share Posted July 5 11 hours ago, alturismo said: may a question, plugin CoreFreq installed ? No, it's not installed. Could it help detecting the issue? Quote Link to comment
ich777 Posted July 5 Author Share Posted July 5 45 minutes ago, MoonshineMagician said: Unfortunately I only got the dmesg out of it, the system was super unresponsive yet again. It didn't even accept a sudo reboot anymore. I restarted using the reset button on the case.. Next time please pull diagnostics or at least save the syslog since it would be good indication what the cause is. Can you please post your Diagnostics? 36 minutes ago, MoonshineMagician said: Could it help detecting the issue? Nope, this was just a guess that it could cause the Kernel Panic. 1 Quote Link to comment
MoonshineMagician Posted July 6 Share Posted July 6 11 hours ago, ich777 said: Next time please pull diagnostics or at least save the syslog since it would be good indication what the cause is. I tried on both hangs to get diagnostics through but it never finishes (I gave it at least half an hour on both occasions). All I get is the output saying collection has started and then nothing. After a forced reboot there are no partial files or a .zip on the usb. Should it happen again I'll try to get a syslog out, any other logs I should attempt to pull manually? I attached a diagnostics from a "clean" system state, I've uninstalled the plugin already. exos-diagnostics-20240705-1712.zip Quote Link to comment
Recommended Posts
Posted by SpencerJ,
1 reaction
Go to this post
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.