unRaid freezes and is not reachable via webgui or ssh


makin

Recommended Posts

Hi everyone, 

 

today in the morning the unraid webgui was not reachable for some reason so I restarted the server manually and it was reachable again. Now I wanted to access it again and it just works occasionally. I was able to open the logs but was not able to download diagnostics in that timeframe and had to reboot again in order to reach unraid. So I guess the attached diagnostics won't be too much help...

 

For the last days it ran completely uneventful. The Parity Check (first time) ran through at about 2AM in the night (took about 15hrs for 8TB) and in the morning, as mentioned, it was not reachable anymore.

 

While writing this post, unraid again became unreachable...

 

PING 192.168.178.56 (192.168.178.56): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
64 bytes from 192.168.178.56: icmp_seq=3 ttl=64 time=2.367 ms
64 bytes from 192.168.178.56: icmp_seq=4 ttl=64 time=2.409 ms
64 bytes from 192.168.178.56: icmp_seq=5 ttl=64 time=2.963 ms
64 bytes from 192.168.178.56: icmp_seq=6 ttl=64 time=4.207 ms
64 bytes from 192.168.178.56: icmp_seq=7 ttl=64 time=1.888 ms

 

Extract of the errors/warnings after rebooting:

Jan 18 02:29:59 gravity kernel: ACPI: Early table checksum verification disabled
Jan 18 02:29:59 gravity kernel: mce: [Hardware Error]: Machine check events logged
Jan 18 02:29:59 gravity kernel: mce: [Hardware Error]: CPU 8: Machine Check: 0 Bank 5: bea0000000000108
Jan 18 02:29:59 gravity kernel: mce: [Hardware Error]: TSC 0 ADDR 1ffff810a4454 MISC d012000100000000 SYND 4d000000 IPID 500b000000000 
Jan 18 02:29:59 gravity kernel: mce: [Hardware Error]: PROCESSOR 2:800f11 TIME 1610965774 SOCKET 0 APIC 5 microcode 8001138
Jan 18 02:29:59 gravity kernel: floppy0: no floppy controllers found
Jan 18 02:29:59 gravity kernel: ccp 0000:07:00.2: psp initialization failed
Jan 18 02:29:59 gravity kernel: ACPI Warning: SystemIO range 0x0000000000000B00-0x0000000000000B08 conflicts with OpRegion 0x0000000000000B00-0x0000000000000B0F (\GSA1.SMBI) (20180810/utaddress-204)
Jan 18 02:29:59 gravity kernel: random: 7 urandom warning(s) missed due to ratelimiting
Jan 18 02:30:09 gravity rpc.statd[1998]: Failed to read /var/lib/nfs/state: Success
Jan 18 02:36:00 gravity root: modprobe: ERROR: could not insert 'kvm_amd': Operation not supported

 

ErrorWarningSystemArrayLogin

Jan 18 01:55:21 gravity kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Jan 18 01:55:21 gravity kernel: hrtimer_interrupt+0xf4/0x20e
Jan 18 01:55:21 gravity kernel: smp_apic_timer_interrupt+0x7b/0x93
Jan 18 01:55:21 gravity kernel: apic_timer_interrupt+0xf/0x20
Jan 18 01:55:21 gravity kernel: </IRQ>
Jan 18 01:55:21 gravity kernel: RIP: 0010:smp_call_function_many+0x1e7/0x200
Jan 18 01:55:21 gravity kernel: Code: db 74 31 83 cf ff 4c 89 ee e8 b0 c4 5b 00 3b 05 32 93 df 00 89 c7 73 1c 48 63 c7 49 8b 14 24 48 03 14 c5 20 37 db 81 8b 42 18 <a8> 01 74 04 f3 90 eb f5 eb d2 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e
Jan 18 01:55:21 gravity kernel: RSP: 0018:ffffc900026b3d08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
Jan 18 01:55:21 gravity kernel: RAX: 0000000000000003 RBX: 0000000000000001 RCX: 0000000000000002
Jan 18 01:55:21 gravity kernel: RDX: ffff88842e6a4b60 RSI: 0000000000000000 RDI: 0000000000000002
Jan 18 01:55:21 gravity kernel: RBP: ffff88842e7e0ce8 R08: ffff88842e7e0cc8 R09: 0000000000000004
Jan 18 01:55:21 gravity kernel: R10: 0000000000000000 R11: ffff888000000000 R12: ffff88842e7e0cc0
Jan 18 01:55:21 gravity kernel: R13: ffff88842e7e0cc8 R14: ffffc900026b3d50 R15: 0000000000000100
Jan 18 01:55:21 gravity kernel: ? smp_call_function_many+0x1cb/0x200
Jan 18 01:55:21 gravity kernel: ? flush_tlb_func_common.constprop.0+0xc2/0xc2
Jan 18 01:55:21 gravity kernel: flush_tlb_mm_range+0xd1/0xf0
Jan 18 01:55:21 gravity kernel: tlb_flush_mmu+0x2f/0xa8
Jan 18 01:55:21 gravity kernel: arch_tlb_finish_mmu+0x35/0x58
Jan 18 01:55:21 gravity kernel: tlb_finish_mmu+0x19/0x25
Jan 18 01:55:21 gravity kernel: madvise_free_single_vma+0x116/0x13e
Jan 18 01:55:21 gravity kernel: ? find_vma+0xe/0x54
Jan 18 01:55:21 gravity kernel: __se_sys_madvise+0x3da/0x6d2
Jan 18 01:55:21 gravity kernel: ? do_syscall_64+0x57/0xf2
Jan 18 01:55:21 gravity kernel: do_syscall_64+0x57/0xf2
Jan 18 01:55:21 gravity kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jan 18 01:55:21 gravity kernel: RIP: 0033:0x45f5f7
Jan 18 01:55:21 gravity kernel: Code: 8b 24 24 48 8b 6c 24 10 48 83 c4 18 c3 cc cc cc cc cc cc 48 8b 7c 24 08 48 8b 74 24 10 8b 54 24 18 48 c7 c0 1c 00 00 00 0f 05 <89> 44 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 24 14
Jan 18 01:55:21 gravity kernel: RSP: 002b:000014e3ef77fb80 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
Jan 18 01:55:21 gravity kernel: RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 000000000045f5f7
Jan 18 01:55:21 gravity kernel: RDX: 0000000000000008 RSI: 0000000000006000 RDI: 000000c0007d2000
Jan 18 01:55:21 gravity kernel: RBP: 000014e3ef77fbc0 R08: 0000000000000000 R09: 0000000000000000
Jan 18 01:55:21 gravity kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcda86e60e
Jan 18 01:55:21 gravity kernel: R13: 00007ffcda86e60f R14: 000014e3ef580000 R15: 0000000000000003
Jan 18 01:56:03 gravity webGUI: Successful login user root from 192.168.178.24
Jan 18 01:56:54 gravity webGUI: Unsuccessful login user kinitzki from 192.168.178.24
Jan 18 01:57:13 gravity webGUI: Unsuccessful login user kinitzki from 192.168.178.24
Jan 18 01:57:48 gravity webGUI: Successful login user root from 192.168.178.24
Jan 18 01:57:58 gravity webGUI: Successful login user root from 192.168.178.24
Jan 18 01:58:00 gravity nginx: 2021/01/18 01:58:00 [error] 6677#6677: *18539 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.178.24, server: , request: "POST /plugins/community.applications/scripts/notices.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "192.168.178.56", referrer: "http://192.168.178.56/Settings/BackupMainV2"
Jan 18 01:58:21 gravity kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jan 18 01:58:21 gravity kernel: rcu: 7-....: (420003 ticks this GP) idle=cf6/1/0x4000000000000002 softirq=95302/95302 fqs=104982 
Jan 18 01:58:21 gravity kernel: rcu: (t=420007 jiffies g=339325 q=10852)
Jan 18 01:58:21 gravity kernel: NMI backtrace for cpu 7
Jan 18 01:58:21 gravity kernel: CPU: 7 PID: 10402 Comm: containerd Not tainted 4.19.107-Unraid #1
Jan 18 01:58:21 gravity kernel: Hardware name: Gigabyte Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F60e 12/09/2020
Jan 18 01:58:21 gravity kernel: Call Trace:
Jan 18 01:58:21 gravity kernel: <IRQ>
Jan 18 01:58:21 gravity kernel: dump_stack+0x67/0x83
Jan 18 01:58:21 gravity kernel: nmi_cpu_backtrace+0x71/0x83
Jan 18 01:58:21 gravity kernel: ? lapic_can_unplug_cpu+0x97/0x97
Jan 18 01:58:21 gravity kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Jan 18 01:58:21 gravity kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Jan 18 01:58:21 gravity kernel: rcu_check_callbacks+0x296/0x5a0
Jan 18 01:58:21 gravity kernel: update_process_times+0x24/0x47
Jan 18 01:58:21 gravity kernel: tick_sched_timer+0x36/0x64
Jan 18 01:58:21 gravity kernel: __hrtimer_run_queues+0xb7/0x10b
Jan 18 01:58:21 gravity kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Jan 18 01:58:21 gravity kernel: hrtimer_interrupt+0xf4/0x20e
Jan 18 01:58:21 gravity kernel: smp_apic_timer_interrupt+0x7b/0x93
Jan 18 01:58:21 gravity kernel: apic_timer_interrupt+0xf/0x20
Jan 18 01:58:21 gravity kernel: </IRQ>
Jan 18 01:58:21 gravity kernel: RIP: 0010:smp_call_function_many+0x1ed/0x200
Jan 18 01:58:21 gravity kernel: Code: 4c 89 ee e8 b0 c4 5b 00 3b 05 32 93 df 00 89 c7 73 1c 48 63 c7 49 8b 14 24 48 03 14 c5 20 37 db 81 8b 42 18 a8 01 74 04 f3 90 <eb> f5 eb d2 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c9 85
Jan 18 01:58:21 gravity kernel: RSP: 0018:ffffc900026b3d08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
Jan 18 01:58:21 gravity kernel: RAX: 0000000000000003 RBX: 0000000000000001 RCX: 0000000000000002
Jan 18 01:58:21 gravity kernel: RDX: ffff88842e6a4b60 RSI: 0000000000000000 RDI: 0000000000000002
Jan 18 01:58:21 gravity kernel: RBP: ffff88842e7e0ce8 R08: ffff88842e7e0cc8 R09: 0000000000000004
Jan 18 01:58:21 gravity kernel: R10: 0000000000000000 R11: ffff888000000000 R12: ffff88842e7e0cc0
Jan 18 01:58:21 gravity kernel: R13: ffff88842e7e0cc8 R14: ffffc900026b3d50 R15: 0000000000000100
Jan 18 01:58:21 gravity kernel: ? smp_call_function_many+0x1cb/0x200
Jan 18 01:58:21 gravity kernel: ? flush_tlb_func_common.constprop.0+0xc2/0xc2
Jan 18 01:58:21 gravity kernel: flush_tlb_mm_range+0xd1/0xf0
Jan 18 01:58:21 gravity kernel: tlb_flush_mmu+0x2f/0xa8
Jan 18 01:58:21 gravity kernel: arch_tlb_finish_mmu+0x35/0x58
Jan 18 01:58:21 gravity kernel: tlb_finish_mmu+0x19/0x25
Jan 18 01:58:21 gravity kernel: madvise_free_single_vma+0x116/0x13e
Jan 18 01:58:21 gravity kernel: ? find_vma+0xe/0x54
Jan 18 01:58:21 gravity kernel: __se_sys_madvise+0x3da/0x6d2
Jan 18 01:58:21 gravity kernel: ? do_syscall_64+0x57/0xf2
Jan 18 01:58:21 gravity kernel: do_syscall_64+0x57/0xf2
Jan 18 01:58:21 gravity kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jan 18 01:58:21 gravity kernel: RIP: 0033:0x45f5f7
Jan 18 01:58:21 gravity kernel: Code: 8b 24 24 48 8b 6c 24 10 48 83 c4 18 c3 cc cc cc cc cc cc 48 8b 7c 24 08 48 8b 74 24 10 8b 54 24 18 48 c7 c0 1c 00 00 00 0f 05 <89> 44 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 24 14
Jan 18 01:58:21 gravity kernel: RSP: 002b:000014e3ef77fb80 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
Jan 18 01:58:21 gravity kernel: RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 000000000045f5f7
Jan 18 01:58:21 gravity kernel: RDX: 0000000000000008 RSI: 0000000000006000 RDI: 000000c0007d2000
Jan 18 01:58:21 gravity kernel: RBP: 000014e3ef77fbc0 R08: 0000000000000000 R09: 0000000000000000
Jan 18 01:58:21 gravity kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcda86e60e
Jan 18 01:58:21 gravity kernel: R13: 00007ffcda86e60f R14: 000014e3ef580000 R15: 0000000000000003
Jan 18 01:59:48 gravity nginx: 2021/01/18 01:59:48 [error] 6677#6677: *18561 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.178.24, server: , request: "GET /Main HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "192.168.178.56", referrer: "http://192.168.178.56/login"
Jan 18 02:00:05 gravity kernel: rcu: INFO: rcu_bh self-detected stall on CPU
Jan 18 02:00:05 gravity kernel: rcu: 7-....: (524805 ticks this GP) idle=cf6/1/0x4000000000000002 softirq=95088/95302 fqs=14992 
Jan 18 02:00:05 gravity kernel: rcu: (t=60000 jiffies g=-1043 q=10)
Jan 18 02:00:05 gravity kernel: NMI backtrace for cpu 7
Jan 18 02:00:05 gravity kernel: CPU: 7 PID: 10402 Comm: containerd Not tainted 4.19.107-Unraid #1
Jan 18 02:00:05 gravity kernel: Hardware name: Gigabyte Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F60e 12/09/2020
Jan 18 02:00:05 gravity kernel: Call Trace:
Jan 18 02:00:05 gravity kernel: <IRQ>
Jan 18 02:00:05 gravity kernel: dump_stack+0x67/0x83
Jan 18 02:00:05 gravity kernel: nmi_cpu_backtrace+0x71/0x83
Jan 18 02:00:05 gravity kernel: ? lapic_can_unplug_cpu+0x97/0x97
Jan 18 02:00:05 gravity kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Jan 18 02:00:05 gravity kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Jan 18 02:00:05 gravity kernel: rcu_check_callbacks+0x296/0x5a0
Jan 18 02:00:05 gravity kernel: update_process_times+0x24/0x47
Jan 18 02:00:05 gravity kernel: tick_sched_timer+0x36/0x64
Jan 18 02:00:05 gravity kernel: __hrtimer_run_queues+0xb7/0x10b
Jan 18 02:00:05 gravity kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Jan 18 02:00:05 gravity kernel: hrtimer_interrupt+0xf4/0x20e
Jan 18 02:00:05 gravity kernel: smp_apic_timer_interrupt+0x7b/0x93
Jan 18 02:00:05 gravity kernel: apic_timer_interrupt+0xf/0x20
Jan 18 02:00:05 gravity kernel: </IRQ>
Jan 18 02:00:05 gravity kernel: RIP: 0010:smp_call_function_many+0x1e9/0x200
Jan 18 02:00:05 gravity kernel: Code: 31 83 cf ff 4c 89 ee e8 b0 c4 5b 00 3b 05 32 93 df 00 89 c7 73 1c 48 63 c7 49 8b 14 24 48 03 14 c5 20 37 db 81 8b 42 18 a8 01 <74> 04 f3 90 eb f5 eb d2 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f
Jan 18 02:00:05 gravity kernel: RSP: 0018:ffffc900026b3d08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
Jan 18 02:00:05 gravity kernel: RAX: 0000000000000003 RBX: 0000000000000001 RCX: 0000000000000002
Jan 18 02:00:05 gravity kernel: RDX: ffff88842e6a4b60 RSI: 0000000000000000 RDI: 0000000000000002
Jan 18 02:00:05 gravity kernel: RBP: ffff88842e7e0ce8 R08: ffff88842e7e0cc8 R09: 0000000000000004
Jan 18 02:00:05 gravity kernel: R10: 0000000000000000 R11: ffff888000000000 R12: ffff88842e7e0cc0
Jan 18 02:00:05 gravity kernel: R13: ffff88842e7e0cc8 R14: ffffc900026b3d50 R15: 0000000000000100
Jan 18 02:00:05 gravity kernel: ? smp_call_function_many+0x1cb/0x200
Jan 18 02:00:05 gravity kernel: ? flush_tlb_func_common.constprop.0+0xc2/0xc2
Jan 18 02:00:05 gravity kernel: flush_tlb_mm_range+0xd1/0xf0
Jan 18 02:00:05 gravity kernel: tlb_flush_mmu+0x2f/0xa8
Jan 18 02:00:05 gravity kernel: arch_tlb_finish_mmu+0x35/0x58
Jan 18 02:00:05 gravity kernel: tlb_finish_mmu+0x19/0x25
Jan 18 02:00:05 gravity kernel: madvise_free_single_vma+0x116/0x13e
Jan 18 02:00:05 gravity kernel: ? find_vma+0xe/0x54
Jan 18 02:00:05 gravity kernel: __se_sys_madvise+0x3da/0x6d2
Jan 18 02:00:05 gravity kernel: ? do_syscall_64+0x57/0xf2
Jan 18 02:00:05 gravity kernel: do_syscall_64+0x57/0xf2
Jan 18 02:00:05 gravity kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jan 18 02:00:05 gravity kernel: RIP: 0033:0x45f5f7
Jan 18 02:00:05 gravity kernel: Code: 8b 24 24 48 8b 6c 24 10 48 83 c4 18 c3 cc cc cc cc cc cc 48 8b 7c 24 08 48 8b 74 24 10 8b 54 24 18 48 c7 c0 1c 00 00 00 0f 05 <89> 44 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 24 14
Jan 18 02:00:05 gravity kernel: RSP: 002b:000014e3ef77fb80 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
Jan 18 02:00:05 gravity kernel: RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 000000000045f5f7
Jan 18 02:00:05 gravity kernel: RDX: 0000000000000008 RSI: 0000000000006000 RDI: 000000c0007d2000
Jan 18 02:00:05 gravity kernel: RBP: 000014e3ef77fbc0 R08: 0000000000000000 R09: 0000000000000000
Jan 18 02:00:05 gravity kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcda86e60e
Jan 18 02:00:05 gravity kernel: R13: 00007ffcda86e60f R14: 000014e3ef580000 R15: 0000000000000003
Jan 18 02:00:05 gravity kernel: Sending NMI from CPU 7 to CPUs 11:
Jan 18 02:00:05 gravity kernel: NMI backtrace for cpu 11
Jan 18 02:00:05 gravity kernel: CPU: 11 PID: 19015 Comm: sshd Not tainted 4.19.107-Unraid #1
Jan 18 02:00:05 gravity kernel: Hardware name: Gigabyte Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F60e 12/09/2020
Jan 18 02:00:05 gravity kernel: RIP: 0010:smp_call_function_many+0x1ed/0x200
Jan 18 02:00:05 gravity kernel: Code: 4c 89 ee e8 b0 c4 5b 00 3b 05 32 93 df 00 89 c7 73 1c 48 63 c7 49 8b 14 24 48 03 14 c5 20 37 db 81 8b 42 18 a8 01 74 04 f3 90 <eb> f5 eb d2 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 31 c9 85
Jan 18 02:00:05 gravity kernel: RSP: 0018:ffffc9001330bc30 EFLAGS: 00000202
Jan 18 02:00:05 gravity kernel: RAX: 0000000000000003 RBX: 0000000000000001 RCX: 0000000000000002
Jan 18 02:00:05 gravity kernel: RDX: ffff88842e6a4be0 RSI: 0000000000000000 RDI: 0000000000000002
Jan 18 02:00:05 gravity kernel: RBP: ffff88842e8e0ce8 R08: ffff88842e8e0cc8 R09: 0000000000000004
Jan 18 02:00:05 gravity kernel: R10: 0000000000000000 R11: ffff88842e007000 R12: ffff88842e8e0cc0
Jan 18 02:00:05 gravity kernel: R13: ffff88842e8e0cc8 R14: 0000000000000000 R15: 0000000000000100
Jan 18 02:00:05 gravity kernel: FS: 0000147846377e00(0000) GS:ffff88842e8c0000(0000) knlGS:0000000000000000
Jan 18 02:00:05 gravity kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 18 02:00:05 gravity kernel: CR2: 00007ffc62f70f0b CR3: 00000003f8ea6000 CR4: 00000000003406e0
Jan 18 02:00:05 gravity kernel: Call Trace:
Jan 18 02:00:05 gravity kernel: ? tlbflush_read_file+0x6b/0x6b
Jan 18 02:00:05 gravity kernel: ? tlbflush_read_file+0x6b/0x6b
Jan 18 02:00:05 gravity kernel: smp_call_function+0x19/0x1c
Jan 18 02:00:05 gravity kernel: on_each_cpu+0x11/0x41
Jan 18 02:00:05 gravity kernel: flush_tlb_kernel_range+0x5e/0x78
Jan 18 02:00:05 gravity kernel: ? __purge_vmap_area_lazy+0x22/0xa9
Jan 18 02:00:05 gravity kernel: __purge_vmap_area_lazy+0x4b/0xa9
Jan 18 02:00:05 gravity kernel: vm_unmap_aliases+0xe4/0x10f
Jan 18 02:00:05 gravity kernel: change_page_attr_set_clr+0xd3/0x35b
Jan 18 02:00:05 gravity kernel: set_memory_ro+0x1e/0x23
Jan 18 02:00:05 gravity kernel: bpf_prog_select_runtime+0x54/0x96
Jan 18 02:00:05 gravity kernel: bpf_prepare_filter+0x44a/0x49a
Jan 18 02:00:05 gravity kernel: ? stop_machine_from_inactive_cpu+0x10d/0x10d
Jan 18 02:00:05 gravity kernel: bpf_prog_create_from_user+0xbf/0xd7
Jan 18 02:00:05 gravity kernel: do_seccomp+0x1e3/0x604
Jan 18 02:00:05 gravity kernel: __se_sys_prctl+0x261/0x4e8
Jan 18 02:00:05 gravity kernel: do_syscall_64+0x57/0xf2
Jan 18 02:00:05 gravity kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jan 18 02:00:05 gravity kernel: RIP: 0033:0x1478464b231a
Jan 18 02:00:05 gravity kernel: Code: 48 8b 0d 79 7b 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 9d 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 46 7b 0c 00 f7 d8 64 89 01 48
Jan 18 02:00:05 gravity kernel: RSP: 002b:00007ffc62f6fa68 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
Jan 18 02:00:05 gravity kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00001478464b231a
Jan 18 02:00:05 gravity kernel: RDX: 000055a8bac2a260 RSI: 0000000000000002 RDI: 0000000000000016
Jan 18 02:00:05 gravity kernel: RBP: 00007ffc62f6fa70 R08: 0000000000000000 R09: 00007ffc62f6f1cb
Jan 18 02:00:05 gravity kernel: R10: 00001478464b231a R11: 0000000000000246 R12: 00007ffc62f6fb20
Jan 18 02:00:05 gravity kernel: R13: 000055a8bac3da10 R14: 000055a8bac4fef0 R15: 000055a8babe5f9a
Jan 18 02:01:21 gravity kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jan 18 02:01:21 gravity kernel: rcu: 7-....: (600006 ticks this GP) idle=cf6/1/0x4000000000000002 softirq=95302/95302 fqs=149973 
Jan 18 02:01:21 gravity kernel: rcu: (t=600010 jiffies g=339325 q=19646)
Jan 18 02:01:21 gravity kernel: NMI backtrace for cpu 7
Jan 18 02:01:21 gravity kernel: CPU: 7 PID: 10402 Comm: containerd Not tainted 4.19.107-Unraid #1
Jan 18 02:01:21 gravity kernel: Hardware name: Gigabyte Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F60e 12/09/2020
Jan 18 02:01:21 gravity kernel: Call Trace:
Jan 18 02:01:21 gravity kernel: <IRQ>
Jan 18 02:01:21 gravity kernel: dump_stack+0x67/0x83
Jan 18 02:01:21 gravity kernel: nmi_cpu_backtrace+0x71/0x83
Jan 18 02:01:21 gravity kernel: ? lapic_can_unplug_cpu+0x97/0x97
Jan 18 02:01:21 gravity kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4
Jan 18 02:01:21 gravity kernel: rcu_dump_cpu_stacks+0x8b/0xb4
Jan 18 02:01:21 gravity kernel: rcu_check_callbacks+0x296/0x5a0
Jan 18 02:01:21 gravity kernel: update_process_times+0x24/0x47
Jan 18 02:01:21 gravity kernel: tick_sched_timer+0x36/0x64
Jan 18 02:01:21 gravity kernel: __hrtimer_run_queues+0xb7/0x10b
Jan 18 02:01:21 gravity kernel: ? tick_sched_handle.isra.0+0x2f/0x2f
Jan 18 02:01:21 gravity kernel: hrtimer_interrupt+0xf4/0x20e
Jan 18 02:01:21 gravity kernel: smp_apic_timer_interrupt+0x7b/0x93
Jan 18 02:01:21 gravity kernel: apic_timer_interrupt+0xf/0x20
Jan 18 02:01:21 gravity kernel: </IRQ>
Jan 18 02:01:21 gravity kernel: RIP: 0010:smp_call_function_many+0x1e9/0x200
Jan 18 02:01:21 gravity kernel: Code: 31 83 cf ff 4c 89 ee e8 b0 c4 5b 00 3b 05 32 93 df 00 89 c7 73 1c 48 63 c7 49 8b 14 24 48 03 14 c5 20 37 db 81 8b 42 18 a8 01 <74> 04 f3 90 eb f5 eb d2 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f
Jan 18 02:01:21 gravity kernel: RSP: 0018:ffffc900026b3d08 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
Jan 18 02:01:21 gravity kernel: RAX: 0000000000000003 RBX: 0000000000000001 RCX: 0000000000000002
Jan 18 02:01:21 gravity kernel: RDX: ffff88842e6a4b60 RSI: 0000000000000000 RDI: 0000000000000002
Jan 18 02:01:21 gravity kernel: RBP: ffff88842e7e0ce8 R08: ffff88842e7e0cc8 R09: 0000000000000004
Jan 18 02:01:21 gravity kernel: R10: 0000000000000000 R11: ffff888000000000 R12: ffff88842e7e0cc0
Jan 18 02:01:21 gravity kernel: R13: ffff88842e7e0cc8 R14: ffffc900026b3d50 R15: 0000000000000100
Jan 18 02:01:21 gravity kernel: ? smp_call_function_many+0x1cb/0x200
Jan 18 02:01:21 gravity kernel: ? flush_tlb_func_common.constprop.0+0xc2/0xc2
Jan 18 02:01:21 gravity kernel: flush_tlb_mm_range+0xd1/0xf0
Jan 18 02:01:21 gravity kernel: tlb_flush_mmu+0x2f/0xa8
Jan 18 02:01:21 gravity kernel: arch_tlb_finish_mmu+0x35/0x58
Jan 18 02:01:21 gravity kernel: tlb_finish_mmu+0x19/0x25
Jan 18 02:01:21 gravity kernel: madvise_free_single_vma+0x116/0x13e
Jan 18 02:01:21 gravity kernel: ? find_vma+0xe/0x54
Jan 18 02:01:21 gravity kernel: __se_sys_madvise+0x3da/0x6d2
Jan 18 02:01:21 gravity kernel: ? do_syscall_64+0x57/0xf2
Jan 18 02:01:21 gravity kernel: do_syscall_64+0x57/0xf2
Jan 18 02:01:21 gravity kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jan 18 02:01:21 gravity kernel: RIP: 0033:0x45f5f7
Jan 18 02:01:21 gravity kernel: Code: 8b 24 24 48 8b 6c 24 10 48 83 c4 18 c3 cc cc cc cc cc cc 48 8b 7c 24 08 48 8b 74 24 10 8b 54 24 18 48 c7 c0 1c 00 00 00 0f 05 <89> 44 24 20 c3 cc cc cc cc 48 8b 7c 24 08 8b 74 24 10 8b 54 24 14
Jan 18 02:01:21 gravity kernel: RSP: 002b:000014e3ef77fb80 EFLAGS: 00000246 ORIG_RAX: 000000000000001c
Jan 18 02:01:21 gravity kernel: RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 000000000045f5f7
Jan 18 02:01:21 gravity kernel: RDX: 0000000000000008 RSI: 0000000000006000 RDI: 000000c0007d2000
Jan 18 02:01:21 gravity kernel: RBP: 000014e3ef77fbc0 R08: 0000000000000000 R09: 0000000000000000
Jan 18 02:01:21 gravity kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcda86e60e
Jan 18 02:01:21 gravity kernel: R13: 00007ffcda86e60f R14: 000014e3ef580000 R15: 0000000000000003

 

After reboot and some further episode of "freezing":

 

Jan 18 02:30:22 gravity avahi-daemon[6695]: Joining mDNS multicast group on interface br0.IPv4 with address 192.168.178.56.
Jan 18 02:30:22 gravity avahi-daemon[6695]: New relevant interface br0.IPv4 for mDNS.
Jan 18 02:30:22 gravity avahi-daemon[6695]: Network interface enumeration completed.
Jan 18 02:30:22 gravity avahi-daemon[6695]: Registering new address record for 192.168.178.56 on br0.IPv4.
Jan 18 02:30:22 gravity emhttpd: shcmd (18): /etc/rc.d/rc.avahidnsconfd start
Jan 18 02:30:22 gravity root: Starting Avahi mDNS/DNS-SD DNS Server Configuration Daemon: /usr/sbin/avahi-dnsconfd -D
Jan 18 02:30:22 gravity avahi-dnsconfd[6706]: Successfully connected to Avahi daemon.
Jan 18 02:30:23 gravity avahi-daemon[6695]: Server startup complete. Host name is gravity.local. Local service cookie is 891107096.
Jan 18 02:30:24 gravity avahi-daemon[6695]: Service "gravity" (/services/ssh.service) successfully established.
Jan 18 02:30:24 gravity avahi-daemon[6695]: Service "gravity" (/services/smb.service) successfully established.
Jan 18 02:30:24 gravity avahi-daemon[6695]: Service "gravity" (/services/sftp-ssh.service) successfully established.
Jan 18 02:30:24 gravity kernel: mdcmd (31): set md_num_stripes 1280
Jan 18 02:30:24 gravity kernel: mdcmd (32): set md_queue_limit 80
Jan 18 02:30:24 gravity kernel: mdcmd (33): set md_sync_limit 5
Jan 18 02:30:24 gravity kernel: mdcmd (34): set md_write_method
Jan 18 02:30:24 gravity kernel: mdcmd (35): set spinup_group 0 0
Jan 18 02:30:24 gravity kernel: mdcmd (36): set spinup_group 1 0
Jan 18 02:30:24 gravity kernel: mdcmd (37): set spinup_group 2 0
Jan 18 02:30:24 gravity kernel: mdcmd (38): set spinup_group 3 0
Jan 18 02:30:24 gravity kernel: mdcmd (39): start STOPPED
Jan 18 02:30:24 gravity kernel: unraid: allocating 25990K for 1280 stripes (5 disks)
Jan 18 02:30:24 gravity kernel: md1: running, size: 7814026532 blocks
Jan 18 02:30:24 gravity kernel: md2: running, size: 3907018532 blocks
Jan 18 02:30:24 gravity kernel: md3: running, size: 3907018532 blocks
Jan 18 02:30:24 gravity emhttpd: shcmd (23): udevadm settle
Jan 18 02:30:24 gravity emhttpd: Autostart disabled
Jan 18 02:30:24 gravity kernel: mdcmd (40): stop 
Jan 18 02:30:24 gravity kernel: md1: stopping
Jan 18 02:30:24 gravity kernel: md2: stopping
Jan 18 02:30:24 gravity kernel: md3: stopping
Jan 18 02:30:25 gravity emhttpd: shcmd (29): /etc/rc.d/rc.php-fpm start
Jan 18 02:30:25 gravity root: Starting php-fpm done
Jan 18 02:30:25 gravity emhttpd: shcmd (30): /etc/rc.d/rc.nginx start
Jan 18 02:30:25 gravity root: Starting Nginx server daemon...
Jan 18 02:30:40 gravity webGUI: Successful login user root from 192.168.178.24
Jan 18 02:31:01 gravity webGUI: Successful login user root from 192.168.178.24
Jan 18 02:32:58 gravity sshd[7625]: Accepted password for root from 192.168.178.24 port 56159 ssh2
Jan 18 02:33:43 gravity webGUI: Successful login user root from 192.168.178.24

 

I added another diagnostics file after the array has been started.

 

gravity-diagnostics-20210118-1130.zip gravity-diagnostics-20210118-1136.zip

Link to comment

Ok, cool. Thanks a lot. So:

 

Bios Setting: Advanced  -->  AMD CBS  -->  Zen Common Options  -->  Global C-state Control

 

Any idea how/when this hard lock happens? As I mentioned, the last day it was running fine. Maybe because the parity checks kept the system busy? :D

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.