MoonshineMagician

I took a look at it. I'm connected to a fritzbox, yeah. To get WireGuard running a long time ago I messed with my network settings a bit and after it finally worked as expected I never touched them again. With the new implementation I changed it to bonding & bridging disabled now. I'm on 6.12.10, so I guess I should be all good. What I don't understand is yes I had the macvlan call trace, but the system worked fine after that nevertheless. Maybe the plugin puts some stress on the networking and at some point it collapses with the prometheus plugin installed. Anyways I've reinstalled the plugin now and I hope it works as expected, should I see issues again I'll try to get logs manually and report back. Thanks to you two!

I tried on both hangs to get diagnostics through but it never finishes (I gave it at least half an hour on both occasions). All I get is the output saying collection has started and then nothing. After a forced reboot there are no partial files or a .zip on the usb. Should it happen again I'll try to get a syslog out, any other logs I should attempt to pull manually? I attached a diagnostics from a "clean" system state, I've uninstalled the plugin already. exos-diagnostics-20240705-1712.zip

No, it's not installed. Could it help detecting the issue?

Unfortunately I only got the dmesg out of it, the system was super unresponsive yet again. It didn't even accept a sudo reboot anymore. I restarted using the reset button on the case..

Thank you for your reply! Surprisingly it happened again way quicker than usual, this time I even had some messages in the dmesg: [179512.581457] rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 9-.... } 41962923 jiffies s: 226673 root: 0x200/. [179512.581468] rcu: blocking rcu_node structures (internal RCU debug): [179512.581471] Sending NMI from CPU 8 to CPUs 9: [179512.581476] NMI backtrace for cpu 9 [179512.581478] CPU: 9 PID: 4366 Comm: prometheus Tainted: P UD W O 6.1.79-Unraid #1 [179512.581479] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B460M Pro4, BIOS P1.60 07/29/2021 [179512.581480] RIP: 0010:native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581485] Code: c2 0f b6 d2 c1 e2 08 30 e4 09 d0 3d ff 00 00 00 76 0c 0f ba e0 08 72 1e c6 43 01 00 eb 18 85 c0 74 0a 8b 03 84 c0 74 04 f3 90 <eb> f6 66 c7 03 01 00 e9 32 01 00 00 e8 c5 40 ff ff 49 c7 c4 80 ed [179512.581486] RSP: 0000:ffffc90026cffd80 EFLAGS: 00000202 [179512.581487] RAX: 0000000000000101 RBX: ffffea0008175ca8 RCX: 000fffffc0000000 [179512.581488] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffea0008175ca8 [179512.581489] RBP: 0000000000000254 R08: 0000000000000000 R09: ffffc90026cffe38 [179512.581489] R10: ffff8882d357f408 R11: ffff8882d357f40c R12: ffff8884d2074100 [179512.581490] R13: 0000000000000000 R14: ffff8882ad595940 R15: ffff8882961da800 [179512.581491] FS: 000000c000800098(0000) GS:ffff888845a40000(0000) knlGS:0000000000000000 [179512.581492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [179512.581493] CR2: 000000c0018d5000 CR3: 00000001f5078006 CR4: 00000000003706e0 [179512.581494] Call Trace: [179512.581496] <NMI> [179512.581498] ? nmi_cpu_backtrace+0xd3/0x104 [179512.581501] ? nmi_cpu_backtrace_handler+0xd/0x15 [179512.581504] ? nmi_handle+0x54/0x131 [179512.581506] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581508] ? default_do_nmi+0x66/0x15b [179512.581510] ? exc_nmi+0xbf/0x130 [179512.581511] ? end_repeat_nmi+0x16/0x67 [179512.581514] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581515] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581516] ? native_queued_spin_lock_slowpath+0x86/0x1cf [179512.581517] </NMI> [179512.581517] <TASK> [179512.581518] do_raw_spin_lock+0x14/0x1a [179512.581520] pmd_lock.constprop.0+0x12/0x1b [179512.581523] huge_pmd_set_accessed+0x16/0x4e [179512.581525] __handle_mm_fault+0x417/0xcf9 [179512.581529] handle_mm_fault+0x13d/0x20f [179512.581531] do_user_addr_fault+0x2c3/0x48d [179512.581534] exc_page_fault+0xfb/0x11d [179512.581536] asm_exc_page_fault+0x22/0x30 [179512.581538] RIP: 0033:0x2a3fe42 [179512.581539] Code: fd 49 89 0b 48 8b 86 d0 00 00 00 49 89 43 08 48 89 8e d0 00 00 00 48 81 c4 d8 00 00 00 5d c3 48 8b b0 f0 00 00 00 4c 8b 5e 08 <41> 84 03 45 89 cc 41 81 e1 ff 01 00 00 47 0f b7 0c 4b 45 89 cb 41 [179512.581541] RSP: 002b:000000c00114f640 EFLAGS: 00010297 [179512.581541] RAX: 000000c000ba4280 RBX: 0000000000000000 RCX: 0000000000000017 [179512.581542] RDX: 0000000000000002 RSI: 000000c000ba4298 RDI: 000000c0015083c0 [179512.581543] RBP: 000000c00114f718 R08: 000000000000001f R09: 0000000073e16eb7 [179512.581544] R10: 000000c000ba4308 R11: 000000c0018d5000 R12: 0000000000616eb7 [179512.581544] R13: ffffffffffffffff R14: 000000c003224000 R15: 0000000000000007 [179512.581546] </TASK> I deactivated the plugin now and I'll see if it helps. A hardware update to 12th Gen Intel is also planned, should I get new information on this issue I'll report back.

Anyone else having problems with node exporter cpu load? Every now and then my system hangs and becomes completely unresponsive, even via SSH. Today I was luckily there right when it happened, and while the web ui was already down, I was still able to log in via SSH, although the login alone took the system 2 minutes. I'm not sure which of duplicati and node exporter is the real culprit, as both have a larger than normal cpu load. But I wanted to attach it here in case anyone else sees this behaviour. Unfortunately I only checked the dmesg (nothing there after boot, so nothing related to the hang), then htop and ran the diagnostic. I left the diagnostics tool running and the ssh connection stayed open for about 1 hour, but the diagnostic never finished before the system died completely. I'm running unraid 6.2.10, node exporter unraid 2024.04.24, duplicati is the linuxserver/duplicati image tagged with version 2.0.8. The system has these rare (every 2 weeks maybe) hang issues for quite a while now (>3 months), but usually I can't get a response when it happens and my last resort is to force a reboot. I did the syslog debug thing to the flash too on one hang in the past, but even the syslog doesn't show something when this happens. Bummer.

"mkdir wal: permission denied" tells you about the problem. Natcoso9955's config on GitHub does not specify a new folder for the wal (wal is the database for Loki to know where to continue in case the whole thing crashes). The default directory is not accessible by the container. You may fix this by specifying another folder in the loki config, e.g.: ingester: wal: dir: /loki/wal lifecycler: ...