System frequently offline

johnmick

By johnmick
in General Support

Recommended Posts

johnmick Noob

johnmick

Posted
Posted

I've run Unraid on my NAS and experienced frequently server offline.

The network was offline, no video output and no respond to keyboard input. Had to hard restart the server everytime and therefore can't get the dignositics logs. Only found a syslog file in the usb boot driver that contains the log before the server offline. 

Aug 16 04:13:09 HP-GEN7 kernel: WARNING: CPU: 0 PID: 815 at fs/inode.c:263 __destroy_inode+0xa8/0xe4
Aug 16 04:13:09 HP-GEN7 kernel: Modules linked in: xt_nat xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_MASQUERADE iptable_nat nf_nat xfs nfsd lockd grace sunrpc md_mod i915 iosf_mbi i2c_algo_bit drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops nct6775 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables bonding x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl intel_cstate i2c_i801 i2c_smbus btusb i2c_core nvme intel_uncore btrtl btbcm btintel e1000e nvme_core bluetooth ahci ecdh_generic ecc libahci video backlight acpi_pad button
Aug 16 04:13:09 HP-GEN7 kernel: CPU: 0 PID: 815 Comm: kswapd0 Not tainted 5.10.21-Unraid #1
Aug 16 04:13:09 HP-GEN7 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H310CM-ITX/ac, BIOS P4.10 05/14/2019
Aug 16 04:13:09 HP-GEN7 kernel: RIP: 0010:__destroy_inode+0xa8/0xe4
Aug 16 04:13:09 HP-GEN7 kernel: Code: 00 00 00 00 48 89 df e8 17 f0 01 00 48 89 df e8 46 76 03 00 83 7b 40 00 75 1a 48 8b 43 28 48 8b 90 48 04 00 00 48 85 d2 75 02 <0f> 0b f0 48 ff 88 48 04 00 00 48 8b 7b 10 48 85 ff 74 0b 40 f6 c7
Aug 16 04:13:09 HP-GEN7 kernel: RSP: 0018:ffffc9000020fb98 EFLAGS: 00010246
Aug 16 04:13:09 HP-GEN7 kernel: RAX: ffff888104514800 RBX: ffff888160801888 RCX: ffffea0009cc4d00
Aug 16 04:13:09 HP-GEN7 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888160801888
Aug 16 04:13:09 HP-GEN7 kernel: RBP: ffffffff81c27ee0 R08: 0000000000000000 R09: ffffffff8122da43
Aug 16 04:13:09 HP-GEN7 kernel: R10: 0000000000000046 R11: ffff888161255c08 R12: ffff888160380958
Aug 16 04:13:09 HP-GEN7 kernel: R13: ffff888160380958 R14: 0000000000001826 R15: 0000000000000000
Aug 16 04:13:09 HP-GEN7 kernel: FS:  0000000000000000(0000) GS:ffff88845f000000(0000) knlGS:0000000000000000
Aug 16 04:13:09 HP-GEN7 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 16 04:13:09 HP-GEN7 kernel: CR2: 000014fb926ae800 CR3: 0000000144de4003 CR4: 00000000003706f0
Aug 16 04:13:09 HP-GEN7 kernel: Call Trace:
Aug 16 04:13:09 HP-GEN7 kernel: destroy_inode+0x27/0x62
Aug 16 04:13:09 HP-GEN7 kernel: __dentry_kill+0xc9/0x12f
Aug 16 04:13:09 HP-GEN7 kernel: shrink_dentry_list+0xa2/0xae
Aug 16 04:13:09 HP-GEN7 kernel: prune_dcache_sb+0x56/0x74
Aug 16 04:13:09 HP-GEN7 kernel: super_cache_scan+0xee/0x16d
Aug 16 04:13:09 HP-GEN7 kernel: do_shrink_slab+0x148/0x1ae
Aug 16 04:13:09 HP-GEN7 kernel: shrink_slab+0x234/0x29e
Aug 16 04:13:09 HP-GEN7 kernel: shrink_node+0x32a/0x578
Aug 16 04:13:09 HP-GEN7 kernel: balance_pgdat+0x25d/0x3dc
Aug 16 04:13:09 HP-GEN7 kernel: kswapd+0x240/0x28c
Aug 16 04:13:09 HP-GEN7 kernel: ? init_wait_entry+0x24/0x24
Aug 16 04:13:09 HP-GEN7 kernel: ? balance_pgdat+0x3dc/0x3dc
Aug 16 04:13:09 HP-GEN7 kernel: kthread+0xe5/0xea
Aug 16 04:13:09 HP-GEN7 kernel: ? __kthread_bind_mask+0x57/0x57
Aug 16 04:13:09 HP-GEN7 kernel: ret_from_fork+0x22/0x30
Aug 16 04:13:09 HP-GEN7 kernel: ---[ end trace e929555722579a7f ]---
Aug 16 04:13:09 HP-GEN7 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000411
Aug 16 04:13:09 HP-GEN7 kernel: #PF: supervisor read access in kernel mode
Aug 16 04:13:09 HP-GEN7 kernel: #PF: error_code(0x0000) - not-present page
Aug 16 04:13:09 HP-GEN7 kernel: PGD 80000001357d2067 P4D 80000001357d2067 PUD 146c11067 PMD 0 
Aug 16 04:13:09 HP-GEN7 kernel: Oops: 0000 [#1] SMP PTI
Aug 16 04:13:09 HP-GEN7 kernel: CPU: 0 PID: 815 Comm: kswapd0 Tainted: G        W         5.10.21-Unraid #1
Aug 16 04:13:09 HP-GEN7 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H310CM-ITX/ac, BIOS P4.10 05/14/2019
Aug 16 04:13:09 HP-GEN7 kernel: RIP: 0010:rb_erase+0x89/0x24e
Aug 16 04:13:09 HP-GEN7 kernel: Code: 10 48 3b 78 10 75 05 48 89 70 10 c3 48 89 70 08 c3 48 89 32 c3 48 8b 41 10 48 89 cf 48 85 c0 75 09 4c 8b 49 08 48 89 c8 eb 29 <48> 8b 70 10 48 85 f6 74 08 48 89 c7 48 89 f0 eb ef 4c 8b 48 08 4c
Aug 16 04:13:09 HP-GEN7 kernel: RSP: 0018:ffffc9000020fb70 EFLAGS: 00010202
Aug 16 04:13:09 HP-GEN7 kernel: RAX: 0000000000000401 RBX: ffff888160801888 RCX: ffff888160801bd8
Aug 16 04:13:09 HP-GEN7 kernel: RDX: ffff888105879bc8 RSI: 0000000000000401 RDI: ffff8881625bc800
Aug 16 04:13:09 HP-GEN7 kernel: RBP: ffff888105879800 R08: ffff888160801760 R09: 0000000000000000
Aug 16 04:13:09 HP-GEN7 kernel: R10: 0000000000000046 R11: ffff8883387a1fa0 R12: ffff888160801658
Aug 16 04:13:09 HP-GEN7 kernel: R13: ffff888105879bc4 R14: ffff888160801760 R15: 0000000000000000
Aug 16 04:13:09 HP-GEN7 kernel: FS:  0000000000000000(0000) GS:ffff88845f000000(0000) knlGS:0000000000000000
Aug 16 04:13:09 HP-GEN7 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 16 04:13:09 HP-GEN7 kernel: CR2: 0000000000000411 CR3: 0000000144de4003 CR4: 00000000003706f0
Aug 16 04:13:09 HP-GEN7 kernel: Call Trace:
Aug 16 04:13:09 HP-GEN7 kernel: btrfs_destroy_inode+0x119/0x1a0
Aug 16 04:13:09 HP-GEN7 kernel: destroy_inode+0x58/0x62
Aug 16 04:13:09 HP-GEN7 kernel: __dentry_kill+0xc9/0x12f
Aug 16 04:13:09 HP-GEN7 kernel: shrink_dentry_list+0xa2/0xae
Aug 16 04:13:09 HP-GEN7 kernel: prune_dcache_sb+0x56/0x74
Aug 16 04:13:09 HP-GEN7 kernel: super_cache_scan+0xee/0x16d
Aug 16 04:13:09 HP-GEN7 kernel: do_shrink_slab+0x148/0x1ae
Aug 16 04:13:09 HP-GEN7 kernel: shrink_slab+0x234/0x29e
Aug 16 04:13:09 HP-GEN7 kernel: shrink_node+0x32a/0x578
Aug 16 04:13:09 HP-GEN7 kernel: balance_pgdat+0x25d/0x3dc
Aug 16 04:13:09 HP-GEN7 kernel: kswapd+0x240/0x28c
Aug 16 04:13:09 HP-GEN7 kernel: ? init_wait_entry+0x24/0x24
Aug 16 04:13:09 HP-GEN7 kernel: ? balance_pgdat+0x3dc/0x3dc
Aug 16 04:13:09 HP-GEN7 kernel: kthread+0xe5/0xea
Aug 16 04:13:09 HP-GEN7 kernel: ? __kthread_bind_mask+0x57/0x57
Aug 16 04:13:09 HP-GEN7 kernel: ret_from_fork+0x22/0x30
Aug 16 04:13:09 HP-GEN7 kernel: Modules linked in: xt_nat xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap veth xt_MASQUERADE iptable_nat nf_nat xfs nfsd lockd grace sunrpc md_mod i915 iosf_mbi i2c_algo_bit drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops nct6775 hwmon_vid ip6table_filter ip6_tables iptable_filter ip_tables bonding x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl intel_cstate i2c_i801 i2c_smbus btusb i2c_core nvme intel_uncore btrtl btbcm btintel e1000e nvme_core bluetooth ahci ecdh_generic ecc libahci video backlight acpi_pad button
Aug 16 04:13:09 HP-GEN7 kernel: CR2: 0000000000000411
Aug 16 04:13:09 HP-GEN7 kernel: ---[ end trace e929555722579a80 ]---
Aug 16 04:13:09 HP-GEN7 kernel: RIP: 0010:rb_erase+0x89/0x24e
Aug 16 04:13:09 HP-GEN7 kernel: Code: 10 48 3b 78 10 75 05 48 89 70 10 c3 48 89 70 08 c3 48 89 32 c3 48 8b 41 10 48 89 cf 48 85 c0 75 09 4c 8b 49 08 48 89 c8 eb 29 <48> 8b 70 10 48 85 f6 74 08 48 89 c7 48 89 f0 eb ef 4c 8b 48 08 4c
Aug 16 04:13:09 HP-GEN7 kernel: RSP: 0018:ffffc9000020fb70 EFLAGS: 00010202
Aug 16 04:13:09 HP-GEN7 kernel: RAX: 0000000000000401 RBX: ffff888160801888 RCX: ffff888160801bd8
Aug 16 04:13:09 HP-GEN7 kernel: RDX: ffff888105879bc8 RSI: 0000000000000401 RDI: ffff8881625bc800
Aug 16 04:13:09 HP-GEN7 kernel: RBP: ffff888105879800 R08: ffff888160801760 R09: 0000000000000000
Aug 16 04:13:09 HP-GEN7 kernel: R10: 0000000000000046 R11: ffff8883387a1fa0 R12: ffff888160801658
Aug 16 04:13:09 HP-GEN7 kernel: R13: ffff888105879bc4 R14: ffff888160801760 R15: 0000000000000000
Aug 16 04:13:09 HP-GEN7 kernel: FS:  0000000000000000(0000) GS:ffff88845f000000(0000) knlGS:0000000000000000
Aug 16 04:13:09 HP-GEN7 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 16 04:13:09 HP-GEN7 kernel: CR2: 0000000000000411 CR3: 0000000144de4003 CR4: 00000000003706f0
Aug 16 04:13:10 HP-GEN7 autofan: Highest disk temp is 38C, adjusting fan speed from: 143 (56% @ 1178rpm) to: 157 (61% @ 1277rpm)
Aug 16 04:14:09 HP-GEN7 kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Aug 16 04:14:09 HP-GEN7 kernel: rcu: 	0-....: (59999 ticks this GP) idle=582/1/0x4000000000000000 softirq=3656541/3656541 fqs=14957 
Aug 16 04:14:09 HP-GEN7 kernel: 	(t=60002 jiffies g=7757353 q=19237)
Aug 16 04:14:09 HP-GEN7 kernel: NMI backtrace for cpu 0
Aug 16 04:14:09 HP-GEN7 kernel: CPU: 0 PID: 10014 Comm: shfs Tainted: G      D W         5.10.21-Unraid #1
Aug 16 04:14:09 HP-GEN7 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H310CM-ITX/ac, BIOS P4.10 05/14/2019
Aug 16 04:14:09 HP-GEN7 kernel: Call Trace:
Aug 16 04:14:09 HP-GEN7 kernel: <IRQ>
Aug 16 04:14:09 HP-GEN7 kernel: dump_stack+0x6b/0x83
Aug 16 04:14:09 HP-GEN7 kernel: ? lapic_can_unplug_cpu+0x8e/0x8e
Aug 16 04:14:09 HP-GEN7 kernel: nmi_cpu_backtrace+0x7d/0x8f
Aug 16 04:14:09 HP-GEN7 kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3
Aug 16 04:14:09 HP-GEN7 kernel: rcu_dump_cpu_stacks+0x9f/0xc6
Aug 16 04:14:09 HP-GEN7 kernel: rcu_sched_clock_irq+0x1ec/0x543
Aug 16 04:14:09 HP-GEN7 kernel: ? trigger_load_balance+0x5a/0x1ca
Aug 16 04:14:09 HP-GEN7 kernel: update_process_times+0x50/0x6e
Aug 16 04:14:09 HP-GEN7 kernel: tick_sched_timer+0x36/0x64
Aug 16 04:14:09 HP-GEN7 kernel: __hrtimer_run_queues+0xb7/0x10b
Aug 16 04:14:09 HP-GEN7 kernel: ? tick_sched_do_timer+0x39/0x39
Aug 16 04:14:09 HP-GEN7 kernel: hrtimer_interrupt+0x8d/0x160
Aug 16 04:14:09 HP-GEN7 kernel: __sysvec_apic_timer_interrupt+0x5d/0x68
Aug 16 04:14:09 HP-GEN7 kernel: asm_call_irq_on_stack+0x12/0x20
Aug 16 04:14:09 HP-GEN7 kernel: </IRQ>
Aug 16 04:14:09 HP-GEN7 kernel: sysvec_apic_timer_interrupt+0x71/0x95
Aug 16 04:14:09 HP-GEN7 kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Aug 16 04:14:09 HP-GEN7 kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x79/0x18a
Aug 16 04:14:09 HP-GEN7 kernel: Code: c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 74 0c 0f ba e0 08 72 1a c6 47 01 00 eb 14 85 c0 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 48 c7 c0 00 30 02 00 65 48 03 05 4c 8f f8
Aug 16 04:14:09 HP-GEN7 kernel: RSP: 0018:ffffc90000623d78 EFLAGS: 00000202
Aug 16 04:14:09 HP-GEN7 kernel: RAX: 0000000000100101 RBX: ffff888142c50478 RCX: 0000000000000001
Aug 16 04:14:09 HP-GEN7 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888105879bc4
Aug 16 04:14:09 HP-GEN7 kernel: RBP: ffff888105879bc4 R08: ffffc90000623da8 R09: 0000000000000018
Aug 16 04:14:09 HP-GEN7 kernel: R10: 000000000000015e R11: 0000000000004000 R12: ffff888105879800
Aug 16 04:14:09 HP-GEN7 kernel: R13: 0000000000000100 R14: ffffc90000623e18 R15: 0000000000000000
Aug 16 04:14:09 HP-GEN7 kernel: queued_spin_lock_slowpath+0x7/0xa
Aug 16 04:14:09 HP-GEN7 kernel: btrfs_get_delayed_node+0x43/0xbe
Aug 16 04:14:09 HP-GEN7 kernel: ? filldir+0x17c/0x17c
Aug 16 04:14:09 HP-GEN7 kernel: btrfs_readdir_get_delayed_items+0x1d/0xdb
Aug 16 04:14:09 HP-GEN7 kernel: ? filldir+0x17c/0x17c
Aug 16 04:14:09 HP-GEN7 kernel: btrfs_real_readdir+0x12d/0x3c7
Aug 16 04:14:09 HP-GEN7 kernel: iterate_dir+0x93/0x131
Aug 16 04:14:09 HP-GEN7 kernel: __do_sys_getdents64+0x6b/0xd4
Aug 16 04:14:09 HP-GEN7 kernel: ? filldir+0x17c/0x17c
Aug 16 04:14:09 HP-GEN7 kernel: do_syscall_64+0x5d/0x6a
Aug 16 04:14:09 HP-GEN7 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Aug 16 04:14:09 HP-GEN7 kernel: RIP: 0033:0x15211f718f97
Aug 16 04:14:09 HP-GEN7 kernel: Code: 0f 1f 00 48 8b 47 20 c3 66 2e 0f 1f 84 00 00 00 00 00 90 48 81 fa ff ff ff 7f b8 ff ff ff 7f 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 c9 6e 10 00 f7 d8 64 89 02 48
Aug 16 04:14:09 HP-GEN7 kernel: RSP: 002b:000015211f21dab8 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Aug 16 04:14:09 HP-GEN7 kernel: RAX: ffffffffffffffda RBX: 0000152118193030 RCX: 000015211f718f97
Aug 16 04:14:09 HP-GEN7 kernel: RDX: 0000000000008000 RSI: 0000152118193030 RDI: 000000000000015e
Aug 16 04:14:09 HP-GEN7 kernel: RBP: ffffffffffffff80 R08: 0000000000000030 R09: 00001521180041d0
Aug 16 04:14:09 HP-GEN7 kernel: R10: 00001521180008d0 R11: 0000000000000293 R12: 0000152118193004
Aug 16 04:14:09 HP-GEN7 kernel: R13: 000000000000003d R14: 0000152118193000 R15: 0000000000000001
Aug 16 04:14:13 HP-GEN7 kernel: rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 0-... } 63087 jiffies s: 11405 root: 0x1/.
Aug 16 04:14:13 HP-GEN7 kernel: rcu: blocking rcu_node structures:
Aug 16 04:14:13 HP-GEN7 kernel: Task dump for CPU 0:
Aug 16 04:14:13 HP-GEN7 kernel: task:shfs            state:R  running task     stack:    0 pid:10014 ppid:     1 flags:0x00000008
Aug 16 04:14:13 HP-GEN7 kernel: Call Trace:
Aug 16 04:14:13 HP-GEN7 kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
Aug 16 04:14:13 HP-GEN7 kernel: ? native_queued_spin_lock_slowpath+0x79/0x18a
Aug 16 04:14:13 HP-GEN7 kernel: ? queued_spin_lock_slowpath+0x7/0xa
Aug 16 04:14:13 HP-GEN7 kernel: ? btrfs_get_delayed_node+0x43/0xbe
Aug 16 04:14:13 HP-GEN7 kernel: ? filldir+0x17c/0x17c
Aug 16 04:14:13 HP-GEN7 kernel: ? btrfs_readdir_get_delayed_items+0x1d/0xdb
Aug 16 04:14:13 HP-GEN7 kernel: ? filldir+0x17c/0x17c
Aug 16 04:14:13 HP-GEN7 kernel: ? btrfs_real_readdir+0x12d/0x3c7
Aug 16 04:14:13 HP-GEN7 kernel: ? iterate_dir+0x93/0x131
Aug 16 04:14:13 HP-GEN7 kernel: ? __do_sys_getdents64+0x6b/0xd4
Aug 16 04:14:13 HP-GEN7 kernel: ? filldir+0x17c/0x17c
Aug 16 04:14:13 HP-GEN7 kernel: ? do_syscall_64+0x5d/0x6a
Aug 16 04:14:13 HP-GEN7 kernel: ? entry_SYSCALL_64_after_hwframe+0x44/0xa9

Does anybody have any idea what causes the issue?

The dignositics logs after the reboot is also attached.

 

hp-gen7-diagnostics-20220816-0739.zip

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing