Server crashes / Web UI not accesible

[loheiman]

Recently been having an issue where after a few days of uptime, the server seems to hang and the web UI becomes inaccessible. I've heard that using macvlan could be the culprit but haven't been able to verify that. I do use custom networks for docker containers.

 

I just noticed my server not responding and rebooted. I exported diagnostics zip from the webUI but have also been mirroring the syslos to the flash drive and have attached that directly. The attached syslog shows this right before it hung:
 

Jun  5 02:30:47 Tower kernel: BUG: kernel NULL pointer dereference, address: 0000000000000008
Jun  5 02:30:47 Tower kernel: #PF: supervisor read access in kernel mode
Jun  5 02:30:47 Tower kernel: #PF: error_code(0x0000) - not-present page
Jun  5 02:30:47 Tower kernel: PGD 0 P4D 0 
Jun  5 02:30:47 Tower kernel: Oops: 0000 [#2] PREEMPT SMP PTI
Jun  5 02:30:47 Tower kernel: CPU: 2 PID: 6743 Comm: btrfs-transacti Tainted: G    B D           5.19.17-Unraid #2
Jun  5 02:30:47 Tower kernel: Hardware name: HP HP Slim Desktop 290-p0xxx/843F, BIOS F.40 11/21/2019
Jun  5 02:30:47 Tower kernel: RIP: 0010:rb_erase+0x17c/0x294
Jun  5 02:30:47 Tower kernel: Code: 48 89 7e 10 49 89 01 e8 2d f8 ff ff 4c 89 ce 48 8b 4e 08 48 85 c9 74 05 f6 01 01 74 3f 48 8b 46 10 48 85 c0 0f 84 8b 00 00 00 <f6> 00 01 0f 85 82 00 00 00 48 8b 50 08 48 85 d2 48 89 56 10 48 89
Jun  5 02:30:47 Tower kernel: RSP: 0018:ffffc90000697d18 EFLAGS: 00010202
Jun  5 02:30:47 Tower kernel: RAX: 0000000000000008 RBX: ffff8881ef355960 RCX: 0000000000000000
Jun  5 02:30:47 Tower kernel: RDX: ffff8881ef355960 RSI: ffff888029afd690 RDI: ffff888029afd840
Jun  5 02:30:47 Tower kernel: RBP: ffff888103a9e240 R08: ffff8881ef355960 R09: ffff888029afd690
Jun  5 02:30:47 Tower kernel: R10: ffff8882e563e3f0 R11: ffff88801820ed60 R12: ffff888103a9e2a0
Jun  5 02:30:47 Tower kernel: R13: ffff888153964000 R14: 0000000000000000 R15: ffff888155678f70
Jun  5 02:30:47 Tower kernel: FS:  0000000000000000(0000) GS:ffff88845f480000(0000) knlGS:0000000000000000
Jun  5 02:30:47 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun  5 02:30:47 Tower kernel: CR2: 0000000000000008 CR3: 000000000220a006 CR4: 00000000003706e0
Jun  5 02:30:47 Tower kernel: Call Trace:
Jun  5 02:30:47 Tower kernel: <TASK>
Jun  5 02:30:47 Tower kernel: btrfs_delete_ref_head+0x30/0x52
Jun  5 02:30:47 Tower kernel: __btrfs_run_delayed_refs+0x9e6/0xb5e
Jun  5 02:30:47 Tower kernel: btrfs_run_delayed_refs+0x5b/0x12c
Jun  5 02:30:47 Tower kernel: ? preempt_latency_start+0x2b/0x46
Jun  5 02:30:47 Tower kernel: btrfs_commit_transaction+0x60/0xa66
Jun  5 02:30:47 Tower kernel: ? start_transaction+0x3d2/0x442
Jun  5 02:30:47 Tower kernel: transaction_kthread+0x105/0x17b
Jun  5 02:30:47 Tower kernel: ? btrfs_cleanup_transaction.isra.0+0x3b1/0x3b1
Jun  5 02:30:47 Tower kernel: kthread+0xe4/0xef
Jun  5 02:30:47 Tower kernel: ? kthread_complete_and_exit+0x1b/0x1b
Jun  5 02:30:47 Tower kernel: ret_from_fork+0x1f/0x30
Jun  5 02:30:47 Tower kernel: </TASK>
Jun  5 02:30:47 Tower kernel: Modules linked in: veth macvlan xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs md_mod tcp_diag inet_diag ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge 8021q garp mrp stp llc bonding tls i915 x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel iosf_mbi drm_buddy i2c_algo_bit ttm kvm drm_display_helper drm_kms_helper drm crct10dif_pclmul btusb crc32_pclmul btrtl r8169 crc32c_intel i2c_i801 btbcm input_leds ghash_clmulni_intel wmi_bmof aesni_intel crypto_simd cryptd rapl intel_cstate intel_uncore led_class i2c_smbus btintel realtek bluetooth ahci intel_gtt nvme libahci agpgart nvme_core i2c_core ecdh_generic ecc syscopyarea sysfillrect sysimgblt intel_pch_thermal fb_sys_fops fan thermal wmi tpm_crb tpm_tis video tpm_tis_core backlight tpm acpi_pad button acpi_tad unix
Jun  5 02:30:47 Tower kernel: CR2: 0000000000000008
Jun  5 02:30:47 Tower kernel: ---[ end trace 0000000000000000 ]---
Jun  5 02:30:47 Tower kernel: RIP: 0010:vfs_getattr_nosec+0x68/0x97
Jun  5 02:30:47 Tower kernel: Code: 20 05 df 07 00 00 89 06 41 f6 42 0d 08 74 08 48 c7 46 10 00 10 00 00 48 c7 46 18 00 10 20 00 49 8b 01 48 8b 78 18 49 8b 42 20 <48> 8b 40 70 48 85 c0 74 14 89 d1 41 81 e0 00 60 00 00 48 89 f2 4c
Jun  5 02:30:47 Tower kernel: RSP: 0018:ffffc900031bfe08 EFLAGS: 00010246
Jun  5 02:30:47 Tower kernel: RAX: 0000000000000000 RBX: ffffc900031bfe98 RCX: 0000000000000000
Jun  5 02:30:47 Tower kernel: RDX: 00000000000007ff RSI: ffffc900031bfe98 RDI: ffffffff8223dbc0
Jun  5 02:30:47 Tower kernel: RBP: 0000000000000000 R08: 0000000000000900 R09: ffffc900031bfe18
Jun  5 02:30:47 Tower kernel: R10: ffff888029c59010 R11: ffffc900031bfe9c R12: 0000000000000900
Jun  5 02:30:47 Tower kernel: R13: 00000000ffffff9c R14: ffff888100e5c000 R15: 00000000000007ff
Jun  5 02:30:47 Tower kernel: FS:  0000000000000000(0000) GS:ffff88845f480000(0000) knlGS:0000000000000000
Jun  5 02:30:47 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun  5 02:30:47 Tower kernel: CR2: 0000000000000008 CR3: 000000000220a006 CR4: 00000000003706e0
Jun  5 02:30:47 Tower kernel: note: btrfs-transacti[6743] exited with preempt_count 2
Jun  5 10:50:15 Tower kernel: microcode: microcode updated early to revision 0xf0, date = 2021-11-15
Jun  5 10:50:15 Tower kernel: Linux version 5.19.17-Unraid (root@Develop) (gcc (GCC) 12.2.0, GNU ld version 2.39-slack151) #2 SMP PREEMPT_DYNAMIC Wed Nov 2 11:54:15 PDT 2022

tower-diagnostics-20230605-1051.zip

syslog

Edited June 5, 2023 by loheiman

[JorgeB]

No macvlan call traces, and the ones I see look more hardware related, but you can try changing to ipvlan, running memtest would also be a good idea.