- Closed
- Urgent
I have had this issue come up before, but could never pull diagnostics. This time I had syslog open, and was able to copy the full log from it before being disconnected from the server.
Dec 2 14:00:02 TheiaHD kernel: BUG: Bad page map in process php7 pte:800000067dff6067 pmd:4f0c94067 Dec 2 14:00:02 TheiaHD kernel: page:ffffea0019f7fd80 count:2 mapcount:-1 mapping:ffff88014f47c370 index:0xce0a Dec 2 14:00:02 TheiaHD kernel: flags: 0xaffff0000000038(uptodate|dirty|lru) Dec 2 14:00:02 TheiaHD kernel: raw: 0affff0000000038 ffffea00180f2b08 ffffea0017369ec8 ffff88014f47c370 Dec 2 14:00:02 TheiaHD kernel: raw: 000000000000ce0a 0000000000000000 00000002fffffffe ffff880109acc000 Dec 2 14:00:02 TheiaHD kernel: page dumped because: bad pte Dec 2 14:00:02 TheiaHD kernel: page->mem_cgroup:ffff880109acc000 Dec 2 14:00:02 TheiaHD kernel: addr:000000002d34a4f5 vm_flags:000000fb anon_vma: (null) mapping:000000008764a8ed index:12ea Dec 2 14:00:02 TheiaHD kernel: file:dev/zero fault:shmem_fault mmap:shmem_mmap readpage: (null) Dec 2 14:00:02 TheiaHD kernel: CPU: 12 PID: 51982 Comm: php7 Not tainted 4.18.17-unRAID #1 Dec 2 14:00:02 TheiaHD kernel: Hardware name: System manufacturer System Product Name/PRIME X399-A, BIOS 0807 08/03/2018 Dec 2 14:00:02 TheiaHD kernel: Call Trace: Dec 2 14:00:02 TheiaHD kernel: dump_stack+0x5d/0x79 Dec 2 14:00:02 TheiaHD kernel: print_bad_pte+0x212/0x22f Dec 2 14:00:02 TheiaHD kernel: unmap_page_range+0x612/0x88a Dec 2 14:00:02 TheiaHD kernel: unmap_vmas+0x4b/0x7f Dec 2 14:00:02 TheiaHD kernel: unmap_region+0xad/0x105 Dec 2 14:00:02 TheiaHD kernel: do_munmap+0x243/0x2b8 Dec 2 14:00:02 TheiaHD kernel: vm_munmap+0x62/0x90 Dec 2 14:00:02 TheiaHD kernel: __x64_sys_munmap+0xd/0x10 Dec 2 14:00:02 TheiaHD kernel: do_syscall_64+0x57/0xe6 Dec 2 14:00:02 TheiaHD kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Dec 2 14:00:02 TheiaHD kernel: RIP: 0033:0x14df749e3318 Dec 2 14:00:02 TheiaHD kernel: Code: c7 e8 6d 9a fe ff 5a c3 48 83 ec 18 48 89 7c 24 08 48 89 34 24 e8 e1 e1 01 00 b8 0b 00 00 00 48 8b 34 24 48 8b 7c 24 08 0f 05 <48> 89 c7 e8 41 9a fe ff 48 83 c4 18 c3 31 c0 83 fa 04 74 0c 48 63 Dec 2 14:00:02 TheiaHD kernel: RSP: 002b:00007ffe1c1bbd10 EFLAGS: 00000246 ORIG_RAX: 000000000000000b Dec 2 14:00:02 TheiaHD kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 000014df749e3318 Dec 2 14:00:02 TheiaHD kernel: RDX: 0000000000000000 RSI: 0000000008000000 RDI: 000014df540c8000 Dec 2 14:00:02 TheiaHD kernel: RBP: 000055a623913260 R08: 0000000000000001 R09: 0000000000000200 Dec 2 14:00:02 TheiaHD kernel: R10: 000014df73a00000 R11: 0000000000000246 R12: 0000000000000030 Dec 2 14:00:02 TheiaHD kernel: R13: 00007ffe1c1bbdf8 R14: 0000000000000000 R15: 0000000000000000 Dec 2 14:00:02 TheiaHD kernel: Disabling lock debugging due to kernel taint Dec 2 14:00:02 TheiaHD kernel: BUG: Bad page state in process php7 pfn:67dff6 Dec 2 14:00:02 TheiaHD kernel: page:ffffea0019f7fd80 count:0 mapcount:-1 mapping:ffff88014f47c370 index:0x1 Dec 2 14:00:02 TheiaHD kernel: flags: 0xaffff0000000000() Dec 2 14:00:02 TheiaHD kernel: raw: 0affff0000000000 dead000000000100 dead000000000200 ffff88014f47c370 Dec 2 14:00:02 TheiaHD kernel: raw: 0000000000000001 0000000000000000 00000000fffffffe 0000000000000000 Dec 2 14:00:02 TheiaHD kernel: page dumped because: non-NULL mapping Dec 2 14:00:02 TheiaHD kernel: Modules linked in: xfs md_mod tun iptable_mangle veth xt_nat ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat tcp_htcp bonding mlx4_en mlx4_core igb i2c_algo_bit edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd mpt3sas glue_helper hid_logitech_hidpp wmi_bmof i2c_piix4 k10temp raid_class mxm_wmi scsi_transport_sas ahci hid_logitech_dj ccp i2c_core pcc_cpufreq libahci wmi button acpi_cpufreq [last unloaded: md_mod] Dec 2 14:00:02 TheiaHD kernel: CPU: 12 PID: 51982 Comm: php7 Tainted: G B 4.18.17-unRAID #1 Dec 2 14:00:02 TheiaHD kernel: Hardware name: System manufacturer System Product Name/PRIME X399-A, BIOS 0807 08/03/2018 Dec 2 14:00:02 TheiaHD kernel: Call Trace: Dec 2 14:00:02 TheiaHD kernel: dump_stack+0x5d/0x79 Dec 2 14:00:02 TheiaHD kernel: bad_page+0xec/0x106 Dec 2 14:00:02 TheiaHD kernel: free_pcppages_bulk+0x123/0x3d4 Dec 2 14:00:02 TheiaHD kernel: free_unref_page_list+0xc9/0x10d Dec 2 14:00:02 TheiaHD kernel: release_pages+0x2ea/0x30e Dec 2 14:00:02 TheiaHD kernel: __pagevec_release+0x1f/0x24 Dec 2 14:00:02 TheiaHD kernel: shmem_undo_range+0x225/0x61f Dec 2 14:00:02 TheiaHD kernel: shmem_truncate_range+0xf/0x2a Dec 2 14:00:02 TheiaHD kernel: shmem_evict_inode+0x6c/0x148 Dec 2 14:00:02 TheiaHD kernel: evict+0xb9/0x16d Dec 2 14:00:02 TheiaHD kernel: __dentry_kill+0xc8/0x130 Dec 2 14:00:02 TheiaHD kernel: dentry_kill+0x11f/0x15a Dec 2 14:00:02 TheiaHD kernel: dput.part.5+0xdf/0xea Dec 2 14:00:02 TheiaHD kernel: __fput+0x178/0x18b Dec 2 14:00:02 TheiaHD kernel: task_work_run+0x77/0x8b Dec 2 14:00:02 TheiaHD kernel: exit_to_usermode_loop+0x46/0x96 Dec 2 14:00:02 TheiaHD kernel: do_syscall_64+0xdf/0xe6 Dec 2 14:00:02 TheiaHD kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Dec 2 14:00:02 TheiaHD kernel: RIP: 0033:0x14df749e3318 Dec 2 14:00:02 TheiaHD kernel: Code: c7 e8 6d 9a fe ff 5a c3 48 83 ec 18 48 89 7c 24 08 48 89 34 24 e8 e1 e1 01 00 b8 0b 00 00 00 48 8b 34 24 48 8b 7c 24 08 0f 05 <48> 89 c7 e8 41 9a fe ff 48 83 c4 18 c3 31 c0 83 fa 04 74 0c 48 63 Dec 2 14:00:02 TheiaHD kernel: RSP: 002b:00007ffe1c1bbd10 EFLAGS: 00000246 ORIG_RAX: 000000000000000b Dec 2 14:00:02 TheiaHD kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 000014df749e3318 Dec 2 14:00:02 TheiaHD kernel: RDX: 0000000000000000 RSI: 0000000008000000 RDI: 000014df540c8000 Dec 2 14:00:02 TheiaHD kernel: RBP: 000055a623913260 R08: 0000000000000001 R09: 0000000000000200 Dec 2 14:00:02 TheiaHD kernel: R10: 000014df73a00000 R11: 0000000000000246 R12: 0000000000000030 Dec 2 14:00:02 TheiaHD kernel: R13: 00007ffe1c1bbdf8 R14: 0000000000000000 R15: 0000000000000000 Dec 2 14:00:02 TheiaHD kernel: BUG: Bad page cache in process php7 pfn:67dff2 Dec 2 14:00:02 TheiaHD kernel: page:ffffea0019f7fc80 count:3 mapcount:1 mapping:ffff880532c42d58 index:0x12ea Dec 2 14:00:02 TheiaHD kernel: flags: 0xaffff0000040029(locked|uptodate|lru|swapbacked) Dec 2 14:00:02 TheiaHD kernel: raw: 0affff0000040029 ffffea0019c13248 ffffea001e04ae88 ffff880532c42d58 Dec 2 14:00:02 TheiaHD kernel: raw: 00000000000012ea 0000000000000000 0000000300000000 ffff88018c4ed000 Dec 2 14:00:02 TheiaHD kernel: page dumped because: still mapped when deleted Dec 2 14:00:02 TheiaHD kernel: page->mem_cgroup:ffff88018c4ed000 Dec 2 14:00:02 TheiaHD kernel: CPU: 12 PID: 51982 Comm: php7 Tainted: G B 4.18.17-unRAID #1 Dec 2 14:00:02 TheiaHD kernel: Hardware name: System manufacturer System Product Name/PRIME X399-A, BIOS 0807 08/03/2018 Dec 2 14:00:02 TheiaHD kernel: Call Trace: Dec 2 14:00:02 TheiaHD kernel: dump_stack+0x5d/0x79 Dec 2 14:00:02 TheiaHD kernel: unaccount_page_cache_page+0xc4/0x1e2 Dec 2 14:00:02 TheiaHD kernel: __delete_from_page_cache+0x33/0xf3 Dec 2 14:00:02 TheiaHD kernel: ? _cond_resched+0x1b/0x1e Dec 2 14:00:02 TheiaHD kernel: ? down_write+0x9/0x33 Dec 2 14:00:02 TheiaHD kernel: delete_from_page_cache+0x43/0x60 Dec 2 14:00:02 TheiaHD kernel: truncate_inode_page+0x17/0x21 Dec 2 14:00:02 TheiaHD kernel: shmem_undo_range+0x1ec/0x61f Dec 2 14:00:02 TheiaHD kernel: shmem_truncate_range+0xf/0x2a Dec 2 14:00:02 TheiaHD kernel: shmem_evict_inode+0x6c/0x148 Dec 2 14:00:02 TheiaHD kernel: evict+0xb9/0x16d Dec 2 14:00:02 TheiaHD kernel: __dentry_kill+0xc8/0x130 Dec 2 14:00:02 TheiaHD kernel: dentry_kill+0x11f/0x15a Dec 2 14:00:02 TheiaHD kernel: dput.part.5+0xdf/0xea Dec 2 14:00:02 TheiaHD kernel: __fput+0x178/0x18b Dec 2 14:00:02 TheiaHD kernel: task_work_run+0x77/0x8b Dec 2 14:00:02 TheiaHD kernel: exit_to_usermode_loop+0x46/0x96 Dec 2 14:00:02 TheiaHD kernel: do_syscall_64+0xdf/0xe6 Dec 2 14:00:02 TheiaHD kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Dec 2 14:00:02 TheiaHD kernel: RIP: 0033:0x14df749e3318 Dec 2 14:00:02 TheiaHD kernel: Code: c7 e8 6d 9a fe ff 5a c3 48 83 ec 18 48 89 7c 24 08 48 89 34 24 e8 e1 e1 01 00 b8 0b 00 00 00 48 8b 34 24 48 8b 7c 24 08 0f 05 <48> 89 c7 e8 41 9a fe ff 48 83 c4 18 c3 31 c0 83 fa 04 74 0c 48 63 Dec 2 14:00:02 TheiaHD kernel: RSP: 002b:00007ffe1c1bbd10 EFLAGS: 00000246 ORIG_RAX: 000000000000000b Dec 2 14:00:02 TheiaHD kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 000014df749e3318 Dec 2 14:00:02 TheiaHD kernel: RDX: 0000000000000000 RSI: 0000000008000000 RDI: 000014df540c8000 Dec 2 14:00:02 TheiaHD kernel: RBP: 000055a623913260 R08: 0000000000000001 R09: 0000000000000200 Dec 2 14:00:02 TheiaHD kernel: R10: 000014df73a00000 R11: 0000000000000246 R12: 0000000000000030 Dec 2 14:00:02 TheiaHD kernel: R13: 00007ffe1c1bbdf8 R14: 0000000000000000 R15: 0000000000000000 Dec 2 14:00:02 TheiaHD kernel: BUG: Bad rss-counter state mm:00000000e320d7b0 idx:0 val:-1 Dec 2 14:00:02 TheiaHD kernel: BUG: Bad rss-counter state mm:00000000e320d7b0 idx:3 val:1 Dec 2 14:01:37 TheiaHD kernel: INFO: rcu_sched self-detected stall on CPU Dec 2 14:01:37 TheiaHD kernel: 38-....: (59999 ticks this GP) idle=022/1/4611686018427387906 softirq=36560218/36560218 fqs=14824 Dec 2 14:01:37 TheiaHD kernel: (t=60000 jiffies g=44797580 c=44797579 q=899812) Dec 2 14:01:37 TheiaHD kernel: NMI backtrace for cpu 38 Dec 2 14:01:37 TheiaHD kernel: CPU: 38 PID: 60513 Comm: kworker/u257:11 Tainted: G B 4.18.17-unRAID #1 Dec 2 14:01:37 TheiaHD kernel: Hardware name: System manufacturer System Product Name/PRIME X399-A, BIOS 0807 08/03/2018 Dec 2 14:01:37 TheiaHD kernel: Workqueue: writeback wb_workfn (flush-0:99) Dec 2 14:01:37 TheiaHD kernel: Call Trace: Dec 2 14:01:37 TheiaHD kernel: <IRQ> Dec 2 14:01:37 TheiaHD kernel: dump_stack+0x5d/0x79 Dec 2 14:01:37 TheiaHD kernel: nmi_cpu_backtrace+0x71/0x83 Dec 2 14:01:37 TheiaHD kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Dec 2 14:01:37 TheiaHD kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7 Dec 2 14:01:37 TheiaHD kernel: rcu_dump_cpu_stacks+0x91/0xbb Dec 2 14:01:37 TheiaHD kernel: rcu_check_callbacks+0x23f/0x5ca Dec 2 14:01:37 TheiaHD kernel: ? tick_sched_handle.isra.5+0x2f/0x2f Dec 2 14:01:37 TheiaHD kernel: update_process_times+0x23/0x45 Dec 2 14:01:37 TheiaHD kernel: tick_sched_timer+0x36/0x64 Dec 2 14:01:37 TheiaHD kernel: __hrtimer_run_queues+0xb1/0x105 Dec 2 14:01:37 TheiaHD kernel: hrtimer_interrupt+0xf4/0x20d Dec 2 14:01:37 TheiaHD kernel: smp_apic_timer_interrupt+0x79/0x89 Dec 2 14:01:37 TheiaHD kernel: apic_timer_interrupt+0xf/0x20 Dec 2 14:01:37 TheiaHD kernel: </IRQ> Dec 2 14:01:37 TheiaHD kernel: RIP: 0010:find_get_pages_range_tag+0x6a/0x1ec Dec 2 14:01:37 TheiaHD kernel: Code: 44 24 10 00 00 00 00 4d 89 cf 45 31 ed 48 89 44 24 18 4d 85 ed 0f 84 38 01 00 00 48 39 6c 24 10 0f 87 53 01 00 00 49 8b 55 00 <48> 85 d2 0f 84 a1 00 00 00 48 89 d0 83 e0 03 74 21 48 ff c8 0f 85 Dec 2 14:01:37 TheiaHD kernel: RSP: 0018:ffffc9001a14fa68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 Dec 2 14:01:37 TheiaHD kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000000a Dec 2 14:01:37 TheiaHD kernel: RDX: ffffea0019f7fd80 RSI: 000000000000000a RDI: ffffea0019f7fd80 Dec 2 14:01:37 TheiaHD kernel: RBP: ffffffffffffffff R08: ffff88049de25468 R09: ffffc9001a14fa78 Dec 2 14:01:37 TheiaHD kernel: R10: 000000000000ce40 R11: 0000000000000000 R12: ffffc9001a14fb28 Dec 2 14:01:37 TheiaHD kernel: R13: ffff88049de254e0 R14: 000000000000000f R15: ffffc9001a14fb38 Dec 2 14:01:37 TheiaHD kernel: ? find_get_pages_range_tag+0x1ad/0x1ec Dec 2 14:01:37 TheiaHD kernel: pagevec_lookup_range_tag+0x1f/0x26 Dec 2 14:01:37 TheiaHD kernel: write_cache_pages+0x11f/0x319 Dec 2 14:01:37 TheiaHD kernel: ? fuse_writepages+0xe0/0xe0 Dec 2 14:01:37 TheiaHD kernel: ? try_to_grab_pending+0x10b/0x115 Dec 2 14:01:37 TheiaHD kernel: ? update_blocked_averages+0x1d5/0x774 Dec 2 14:01:37 TheiaHD kernel: ? release_stripe+0x25/0x3b [md_mod] Dec 2 14:01:37 TheiaHD kernel: fuse_writepages+0x77/0xe0 Dec 2 14:01:37 TheiaHD kernel: do_writepages+0x28/0x51 Dec 2 14:01:37 TheiaHD kernel: __writeback_single_inode+0x2a/0x153 Dec 2 14:01:37 TheiaHD kernel: writeback_sb_inodes+0x1d8/0x368 Dec 2 14:01:37 TheiaHD kernel: __writeback_inodes_wb+0x63/0x9a Dec 2 14:01:37 TheiaHD kernel: wb_writeback+0x11c/0x1b6 Dec 2 14:01:37 TheiaHD kernel: ? get_nr_inodes+0x1c/0x47 Dec 2 14:01:37 TheiaHD kernel: ? cpumask_next+0x16/0x17 Dec 2 14:01:37 TheiaHD kernel: wb_workfn+0x19d/0x251 Dec 2 14:01:37 TheiaHD kernel: process_one_work+0x16e/0x243 Dec 2 14:01:37 TheiaHD kernel: ? cancel_delayed_work_sync+0xa/0xa Dec 2 14:01:37 TheiaHD kernel: worker_thread+0x1dc/0x2ac Dec 2 14:01:37 TheiaHD kernel: kthread+0x10b/0x113 Dec 2 14:01:37 TheiaHD kernel: ? kthread_flush_work_fn+0x9/0x9 Dec 2 14:01:37 TheiaHD kernel: ret_from_fork+0x22/0x40 Dec 2 14:02:41 TheiaHD nginx: 2018/12/02 14:02:41 [error] 17633#17633: *1765692 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.0.74, server: , request: "POST /webGui/include/Notify.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: ".unraid.net:444", referrer: "https://.unraid.net:444/Dashboard" Dec 2 14:02:45 TheiaHD nginx: 2018/12/02 14:02:45 [error] 17633#17633: *1765692 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.0.74, server: , request: "POST /webGui/include/DashUpdate.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: ".unraid.net:444", referrer: "https://.unraid.net:444/Dashboard"