Server crashes randomly - syslog attached

[hundsboog]

Hello friendos,

 

I deal a certain time with really strange shutdowns. They appeared like half a year ago and I really do hard to find the reason for it.

 

here is the syslog, which my raspberry pi syslog server had saved shortly before the last crash:

 

 

Oct 28 22:40:01 laborpi CRON[14203]: (root) CMD (   PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker local)
Oct 28 22:45:36 Tower kernel: general protection fault, probably for non-canonical address 0xffff11207e99fe60: 0000 [#1] SMP NOPTI
Oct 28 22:45:36 Tower kernel: CPU: 35 PID: 0 Comm: swapper/35 Tainted: G S      W         5.10.28-Unraid #1
Oct 28 22:45:36 Tower kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X399 Taichi, BIOS P3.90 12/04/2019
Oct 28 22:45:36 Tower kernel: RIP: 0010:set_work_data+0x5/0x10
Oct 28 22:45:36 Tower kernel: Code: d0 21 c8 81 e6 c8 01 00 00 89 47 60 74 16 81 e1 c8 01 00 00 74 0e a9 c8 01 00 00 75 07 f0 ff 82 00 03 00 00 c3 48 8b 07 a8 01 <65> 02 0f 0b 48 09 d6 48 89 37 c3 53 89 fb 48 c7 c7 c0 58 03 82 e8
Oct 28 22:45:36 Tower kernel: RSP: 0018:ffffc90006f08eb8 EFLAGS: 00010002
Oct 28 22:45:36 Tower kernel: RAX: 00000000000008c1 RBX: ffff88903f4e8000 RCX: 0000000000000000
Oct 28 22:45:36 Tower kernel: RDX: 0000000000000005 RSI: ffff88903f4e8000 RDI: ffff88903f4dfe60
Oct 28 22:45:36 Tower kernel: RBP: ffff888100064400 R08: ffff88903f4e8000 R09: ffff88903f4e1d20
Oct 28 22:45:36 Tower kernel: R10: ffff88903f4e1d00 R11: ffff8881004004e8 R12: 0000000000000023
Oct 28 22:45:36 Tower kernel: R13: ffff88903f4dfe60 R14: 0000000000000023 R15: ffff88903f4e1d00
Oct 28 22:45:36 Tower kernel: FS:  0000000000000000(0000) GS:ffff88903f4c0000(0000) knlGS:0000000000000000
Oct 28 22:45:36 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 28 22:45:36 Tower kernel: CR2: 0000006fcf6c725a CR3: 000000010936c000 CR4: 00000000003506e0
Oct 28 22:45:36 Tower kernel: Call Trace:
Oct 28 22:45:36 Tower kernel: <IRQ>
Oct 28 22:45:36 Tower kernel: insert_work+0x19/0x53
Oct 28 22:45:36 Tower kernel: __queue_work+0x235/0x24c
Oct 28 22:45:36 Tower kernel: call_timer_fn.isra.0+0x12/0x6f
Oct 28 22:45:36 Tower kernel: ? queue_work_node+0xbe/0xbe
Oct 28 22:45:36 Tower kernel: __run_timers.part.0+0x120/0x185
Oct 28 22:45:36 Tower kernel: ? update_process_times+0x68/0x6e
Oct 28 22:45:36 Tower kernel: ? hrtimer_forward+0x73/0x7b
Oct 28 22:45:36 Tower kernel: ? tick_sched_timer+0x5a/0x64
Oct 28 22:45:36 Tower kernel: ? timerqueue_add+0x62/0x68
Oct 28 22:45:36 Tower kernel: ? timekeeping_get_ns+0x19/0x2f
Oct 28 22:45:36 Tower kernel: __do_softirq+0xc4/0x1c2
Oct 28 22:45:36 Tower kernel: asm_call_irq_on_stack+0x12/0x20
Oct 28 22:45:36 Tower kernel: </IRQ>
Oct 28 22:45:36 Tower kernel: do_softirq_own_stack+0x2c/0x39
Oct 28 22:45:36 Tower kernel: __irq_exit_rcu+0x45/0x80
Oct 28 22:45:36 Tower kernel: sysvec_apic_timer_interrupt+0x87/0x95
Oct 28 22:45:36 Tower kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Oct 28 22:45:36 Tower kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Oct 28 22:45:36 Tower kernel: Code: 00 48 83 c4 28 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 55 8b af 28 04 00 00 b8 01 00 00 00 45 31 c9 53 45 31 d2 39 c5
Oct 28 22:45:36 Tower kernel: RSP: 0018:ffffc90006613ea0 EFLAGS: 00000246
Oct 28 22:45:36 Tower kernel: RAX: ffff88903f4e2380 RBX: 0000000000000001 RCX: 000000000000001f
Oct 28 22:45:36 Tower kernel: RDX: 0000000000000000 RSI: 00000000238e364d RDI: 0000000000000000
Oct 28 22:45:36 Tower kernel: RBP: ffff8890a33ea400 R08: 000342ed73233146 R09: 000000000000025c
Oct 28 22:45:36 Tower kernel: R10: 000000000000026b R11: 071c71c71c71c71c R12: 000342ed73233146
Oct 28 22:45:36 Tower kernel: R13: ffffffff820c8c40 R14: 0000000000000001 R15: 0000000000000000
Oct 28 22:45:36 Tower kernel: cpuidle_enter_state+0x101/0x1c4
Oct 28 22:45:36 Tower kernel: cpuidle_enter+0x25/0x31
Oct 28 22:45:36 Tower kernel: do_idle+0x1a6/0x214
Oct 28 22:45:36 Tower kernel: cpu_startup_entry+0x18/0x1a
Oct 28 22:45:36 Tower kernel: secondary_startup_64_no_verify+0xb0/0xbb
Oct 28 22:45:36 Tower kernel: Modules linked in: macvlan xt_mark xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat iptable_mangle nf_tables vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype $

 

 

Could somebody help me dig into the right direction? Im not an expert in analyze error logs.

It also happens always in the night without any certain load.... 

 

Things I did so far:

Disabled the SSD trimming feature (I have a 2TB two Disk SSD Cache on btrfs)

Checked the memory with memtest

Disabled some dockers known to be not reliable.

 

 

Any help is highly appreciated!

 

Thanks!!

 

hundsboog

[Frank1940]

I would think that you will need to attach the entire syslog to your next  post.  (Don't edit your first post to attach it or no one will ever realize the your did it!)  Solving these problems is difficult and the Gurus often find that the problem starts way before the system finally goes completely off the reservation. 

[hundsboog]

On 10/30/2021 at 4:16 AM, Frank1940 said:

I would think that you will need to attach the entire syslog to your next  post.  (Don't edit your first post to attach it or no one will ever realize the your did it!)  Solving these problems is difficult and the Gurus often find that the problem starts way before the system finally goes completely off the reservation. 

Hi Frank,

 

Thank you so much. 

I searched the forum a bit and found a thread about call trace errors which lead to a kernel panic. Basically I upgraded to the new release candidate and so far, everything is ok. I will test it out and will write back. I looked at the logs and there is now another error, but some people upgraded like me, experienced the same. I will keep recording the syslogs and save the next time the server crashes the complete log, like you advised.

 

But so far, everythings up and running! Have a great day

 

Nov 1 19:08:00 Tower kernel: docker0: port 10(veth0351d9e) entered disabled state
Nov 1 19:08:00 Tower kernel: veth29e74e4: renamed from eth0
Nov 1 19:08:00 Tower avahi-daemon[7541]: Interface veth0351d9e.IPv6 no longer relevant for mDNS.
Nov 1 19:08:00 Tower avahi-daemon[7541]: Leaving mDNS multicast group on interface veth0351d9e.IPv6 with address fe80::e063:a1ff:fe3c:d18b.
Nov 1 19:08:00 Tower kernel: docker0: port 10(veth0351d9e) entered disabled state
Nov 1 19:08:00 Tower kernel: device veth0351d9e left promiscuous mode
Nov 1 19:08:00 Tower kernel: docker0: port 10(veth0351d9e) entered disabled state
Nov 1 19:08:00 Tower avahi-daemon[7541]: Withdrawing address record for fe80::e063:a1ff:fe3c:d18b on veth0351d9e.
Nov 1 19:08:00 Tower wsdd2[7516]: error: wsdd-mcast-v6: wsd_send_soap_msg: send: Cannot assign requested address
Nov 1 19:08:02 Tower kernel: docker0: port 10(veth51bbbe0) entered blocking state
Nov 1 19:08:02 Tower kernel: docker0: port 10(veth51bbbe0) entered disabled state
Nov 1 19:08:02 Tower kernel: device veth51bbbe0 entered promiscuous mode
Nov 1 19:08:02 Tower kernel: docker0: port 10(veth51bbbe0) entered blocking state
Nov 1 19:08:02 Tower kernel: docker0: port 10(veth51bbbe0) entered forwarding state
Nov 1 19:08:02 Tower kernel: docker0: port 10(veth51bbbe0) entered disabled state
Nov 1 19:08:05 Tower kernel: eth0: renamed from vethf3c118f
Nov 1 19:08:05 Tower kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth51bbbe0: link becomes ready
Nov 1 19:08:05 Tower kernel: docker0: port 10(veth51bbbe0) entered blocking state
Nov 1 19:08:05 Tower kernel: docker0: port 10(veth51bbbe0) entered forwarding state
Nov 1 19:08:06 Tower avahi-daemon[7541]: Joining mDNS multicast group on interface veth51bbbe0.IPv6 with address fe80::3485:ffff:feaf:3a45.
Nov 1 19:08:06 Tower avahi-daemon[7541]: New relevant interface veth51bbbe0.IPv6 for mDNS.
Nov 1 19:08:06 Tower avahi-daemon[7541]: Registering new address record for fe80::3485:ffff:feaf:3a45 on veth51bbbe0.*.
Nov 1 19:08:06 Tower wsdd2[7516]: error: wsdd-mcast-v6: wsd_send_soap_msg: send: Cannot assign requested address