eds Posted April 29, 2019 Share Posted April 29, 2019 For, perhaps the fourth time this month, my unraid system has become unresponsive while streaming video either with Kodi or with Plex running on a local machine. It seems to happen after the system has been running for about six days. In addition to the video becoming unresponsive, I notice the memory hits about 91 to 92%. When I try to shutdown using unraid, the system hangs when trying to shutdown php-fpm. At that point I have to do a warm reboot. I am not sure what the problem is. Any help would be appreciated. Thanks. unraid-diagnostics-20190428-2327.zip Quote Link to comment
sylinen Posted May 1, 2019 Share Posted May 1, 2019 I've had a similar issue, and it usually triggers for me also at around 6 days. The crash usually coincides with fast-forwarding or rewinding in Plex. Quote Link to comment
eds Posted May 1, 2019 Author Share Posted May 1, 2019 Yes. Plex or Kodi for me. Did it stop for you? Do you have vms running when it happens to you? I have a deepin, windows server and windows 7 vm. The windows 7 vm was not running, so I've crossed that off the list. Also the issue with php-fpm seems to be linked to docker. Quote Link to comment
sylinen Posted May 3, 2019 Share Posted May 3, 2019 2 VMs, an Ubuntu server for pihole and one general purpose Manjaro VM. Haven't tried disabling them. I'm running the official Plex Inc. docker if that matters. Ryzen 1700X for CPU. Quote Link to comment
eds Posted May 16, 2019 Author Share Posted May 16, 2019 OK, I think I have narrowed what may be a problem. The windows server vm appears to be the problem of one (or perhaps all) of my crashes. I was able to catch the log as it was being written when I started the vm and it crashes unraid. Here is the relevant section, for what I was able to copy: Quote ay 16 15:46:48 Unraid kernel: NETDEV WATCHDOG: eth2 (igb): transmit queue 2 timed out May 16 15:46:48 Unraid kernel: WARNING: CPU: 5 PID: 30117 at net/sched/sch_generic.c:461 dev_watchdog+0x150/0x1a8 May 16 15:46:48 Unraid kernel: Modules linked in: veth nvidia_uvm(O) macvlan xt_CHECKSUM iptable_mangle ipt_REJECT ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap xt_nat ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod it87 hwmon_vid be2net igb i2c_algo_bit nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel drm_kms_helper aes_x86_64 crypto_simd cryptd drm glue_helper intel_cstate intel_uncore i2c_i801 ipmi_ssif agpgart intel_rapl_perf i2c_core ahci libahci syscopyarea sysfillrect sysimgblt fb_sys_fops wmi acpi_power_meter ipmi_si acpi_pad pcc_cpufreq button [last unloaded: be2net] May 16 15:46:48 Unraid kernel: CPU: 5 PID: 30117 Comm: CPU 4/KVM Tainted: P D W O 4.18.20-unRAID #1 May 16 15:46:48 Unraid kernel: Hardware name: GIGABYTE MD30-RS0-XX/MD30-RS0-XX, BIOS R08 06/15/2018 May 16 15:46:48 Unraid kernel: RIP: 0010:dev_watchdog+0x150/0x1a8 May 16 15:46:48 Unraid kernel: Code: 69 f8 97 00 00 75 36 4c 89 ef c6 05 5d f8 97 00 01 e8 29 c5 fd ff 89 e9 4c 89 ee 48 c7 c7 66 0f d9 81 48 89 c2 e8 f9 bb b2 ff <0f> 0b eb 0f ff c5 48 81 c2 40 01 00 00 39 cd 75 98 eb 13 48 8b 83 May 16 15:46:48 Unraid kernel: RSP: 0018:ffff88056b943ea0 EFLAGS: 00010286 May 16 15:46:48 Unraid kernel: RAX: 0000000000000000 RBX: ffff8805497ec3b0 RCX: 0000000000000007 May 16 15:46:48 Unraid kernel: RDX: 0000000000000006 RSI: ffff88056b956470 RDI: ffff88056b956470 May 16 15:46:48 Unraid kernel: RBP: 0000000000000002 R08: 000000000000000f R09: ffff8800000bef00 May 16 15:46:48 Unraid kernel: R10: 0000000000000b17 R11: 0000000000036ac8 R12: ffff8805497ec39c May 16 15:46:48 Unraid kernel: R13: ffff8805497ec000 R14: ffff880546f4c940 R15: 0000000000000005 May 16 15:46:48 Unraid kernel: FS: 000014cb1c8f3700(0000) GS:ffff88056b940000(0000) knlGS:00000000fff6e000 May 16 15:46:48 Unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 16 15:46:48 Unraid kernel: CR2: 000000003d4dc000 CR3: 000000086758e006 CR4: 00000000001626e0 May 16 15:46:48 Unraid kernel: Call Trace: May 16 15:46:48 Unraid kernel: <IRQ> May 16 15:46:48 Unraid kernel: call_timer_fn+0x18/0x7b May 16 15:46:48 Unraid kernel: ? qdisc_reset+0xc0/0xc0 May 16 15:46:48 Unraid kernel: expire_timers+0x7f/0x8e May 16 15:46:48 Unraid kernel: run_timer_softirq+0x72/0x120 May 16 15:46:48 Unraid kernel: ? enqueue_hrtimer.isra.3+0x23/0x27 May 16 15:46:48 Unraid kernel: ? __hrtimer_run_queues+0xd7/0x105 May 16 15:46:48 Unraid kernel: ? recalibrate_cpu_khz+0x1/0x1 May 16 15:46:48 Unraid kernel: ? ktime_get+0x3a/0x8d May 16 15:46:48 Unraid kernel: __do_softirq+0xce/0x1e2 May 16 15:46:48 Unraid kernel: irq_exit+0x5e/0x9d May 16 15:46:48 Unraid kernel: smp_apic_timer_interrupt+0x7e/0x91 May 16 15:46:48 Unraid kernel: apic_timer_interrupt+0xf/0x20 May 16 15:46:48 Unraid kernel: </IRQ> May 16 15:46:48 Unraid kernel: RIP: 0010:vmx_handle_external_intr+0x65/0x66 [kvm_intel] May 16 15:46:48 Unraid kernel: Code: 0f b7 50 06 8b 48 08 0f b7 00 48 c1 e2 10 48 c1 e1 20 48 09 ca 48 09 d0 48 89 e2 48 83 e4 f0 6a 18 52 9c 6a 10 e8 4a f6 1d e1 <c3> 31 c0 f6 05 7e 96 02 00 80 74 0a 0f b7 05 79 96 02 00 83 e0 01 May 16 15:46:48 Unraid kernel: RSP: 0018:ffffc90006e0fd58 EFLAGS: 00000086 ORIG_RAX: ffffffffffffff13 May 16 15:46:48 Unraid kernel: RAX: ffffffff81801730 RBX: ffff880865c68000 RCX: ffffffff00000000 May 16 15:46:48 Unraid kernel: RDX: ffffc90006e0fd58 RSI: 0009c6c068bde7f4 RDI: ffff880865c68000 May 16 15:46:48 Unraid kernel: RBP: 0000000000000000 R08: fff7625b2df1fdb2 R09: 0000000000000000 May 16 15:46:48 Unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 May 16 15:46:48 Unraid kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffff880865c68058 May 16 15:46:48 Unraid kernel: ? __entry_text_end+0x7/0x7 May 16 15:46:48 Unraid kernel: ? vmx_get_exit_info+0x47/0x47 [kvm_intel] May 16 15:46:48 Unraid kernel: ? kvm_arch_vcpu_ioctl_run+0xf85/0x1298 [kvm] May 16 15:46:48 Unraid kernel: ? kvm_vcpu_ioctl+0x187/0x4ba [kvm] May 16 15:46:48 Unraid kernel: ? __seccomp_filter+0x39/0x1ed May 16 15:46:48 Unraid kernel: ? vfs_ioctl+0x19/0x26 May 16 15:46:48 Unraid kernel: ? do_vfs_ioctl+0x518/0x540 May 16 15:46:48 Unraid kernel: ? ksys_ioctl+0x39/0x58 May 16 15:46:48 Unraid kernel: ? __x64_sys_ioctl+0x11/0x14 May 16 15:46:48 Unraid kernel: ? do_syscall_64+0x57/0xe6 May 16 15:46:48 Unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 May 16 15:46:48 Unraid kernel: ---[ end trace b90117603784f00c ]--- May 16 15:46:48 Unraid kernel: igb 0000:07:00.0 eth2: Reset adapter May 16 15:46:48 Unraid kernel: igb 0000:07:00.0 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX May 16 15:47:35 Unraid kernel: INFO: rcu_sched self-detected stall on CPU May 16 15:47:35 Unraid kernel: 35-....: (59999 ticks this GP) idle=b4a/1/4611686018427387906 softirq=14417731/14417731 fqs=14946 May 16 15:47:35 Unraid kernel: (t=60000 jiffies g=16469846 c=16469845 q=69705) May 16 15:47:35 Unraid kernel: NMI backtrace for cpu 35 May 16 15:47:35 Unraid kernel: CPU: 35 PID: 37690 Comm: worker Tainted: P D W O 4.18.20-unRAID #1 May 16 15:47:35 Unraid kernel: Hardware name: GIGABYTE MD30-RS0-XX/MD30-RS0-XX, BIOS R08 06/15/2018 May 16 15:47:35 Unraid kernel: Call Trace: May 16 15:47:35 Unraid kernel: <IRQ> May 16 15:47:35 Unraid kernel: dump_stack+0x5d/0x79 May 16 15:47:35 Unraid kernel: nmi_cpu_backtrace+0x71/0x83 May 16 15:47:35 Unraid kernel: ? lapic_can_unplug_cpu+0x8e/0x8e May 16 15:47:35 Unraid kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7 May 16 15:47:35 Unraid kernel: rcu_dump_cpu_stacks+0x91/0xbb May 16 15:47:35 Unraid kernel: rcu_check_callbacks+0x23f/0x5ca May 16 15:47:35 Unraid kernel: ? tick_sched_handle.isra.5+0x2f/0x2f May 16 15:47:35 Unraid kernel: update_process_times+0x23/0x45 May 16 15:47:35 Unraid kernel: tick_sched_timer+0x36/0x64 May 16 15:47:35 Unraid kernel: __hrtimer_run_queues+0xb1/0x105 May 16 15:47:35 Unraid kernel: hrtimer_interrupt+0xf4/0x20d May 16 15:47:35 Unraid kernel: smp_apic_timer_interrupt+0x79/0x91 May 16 15:47:35 Unraid kernel: apic_timer_interrupt+0xf/0x20 May 16 15:47:35 Unraid kernel: </IRQ> May 16 15:47:35 Unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x165/0x16d May 16 15:47:35 Unraid kernel: Code: eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d ba d9 f9 7e c3 85 d2 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 c8 61 b9 40 00 00 May 16 15:47:35 Unraid kernel: RSP: 0018:ffffc9000ed135e0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 May 16 15:47:35 Unraid kernel: RAX: 0000000000000101 RBX: ffffc9000ed13658 RCX: 0000000000000101 May 16 15:47:35 Unraid kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffea000f943628 May 16 15:47:35 Unraid kernel: RBP: ffffea000e9d83c0 R08: ffff880000000000 R09: ffffea000e9d8400 May 16 15:47:35 Unraid kernel: R10: ffff88057fffc000 R11: 0000000000025098 R12: ffff88054d20e600 May 16 15:47:35 Unraid kernel: R13: ffffc9000ed13720 R14: ffff8804bdf53980 R15: ffff8803efe65c80 May 16 15:47:35 Unraid kernel: _raw_spin_lock+0x16/0x19 May 16 15:47:35 Unraid kernel: page_vma_mapped_walk+0x3d9/0x5e1 May 16 15:47:35 Unraid kernel: try_to_unmap_one+0xfe/0x62c May 16 15:47:35 Unraid kernel: rmap_walk_anon+0x163/0x1d3 May 16 15:47:35 Unraid kernel: ? __ClearPageMovable+0x6/0x6 May 16 15:47:35 Unraid kernel: try_to_unmap+0x9e/0xde May 16 15:47:35 Unraid kernel: ? page_remove_rmap+0x29c/0x29c May 16 15:47:35 Unraid kernel: ? page_not_mapped+0xc/0xc May 16 15:47:35 Unraid kernel: ? page_get_anon_vma+0x5c/0x5c May 16 15:47:35 Unraid kernel: ? is_vma_temporary_stack+0x1d/0x1d May 16 15:47:35 Unraid kernel: migrate_pages+0x307/0x635 May 16 15:47:35 Unraid kernel: ? isolate_migratepages_block+0x634/0x634 May 16 15:47:35 Unraid kernel: compact_zone+0x353/0x534 May 16 15:47:35 Unraid kernel: compact_zone_order+0x92/0xac May 16 15:47:35 Unraid kernel: try_to_compact_pages+0x9b/0x12e May 16 15:47:35 Unraid kernel: __alloc_pages_direct_compact+0x46/0xd8 May 16 15:47:35 Unraid kernel: __alloc_pages_nodemask+0x598/0xa8f May 16 15:47:35 Unraid kernel: ? alloc_pages_vma+0xe0/0x173 May 16 15:47:35 Unraid kernel: do_huge_pmd_anonymous_page+0x267/0x4d0 May 16 15:47:35 Unraid kernel: __handle_mm_fault+0x2c0/0x10aa May 16 15:47:35 Unraid kernel: handle_mm_fault+0x158/0x1a7 May 16 15:47:35 Unraid kernel: __do_page_fault+0x271/0x40b May 16 15:47:35 Unraid kernel: page_fault+0x1e/0x30 May 16 15:47:35 Unraid kernel: RIP: 0010:copy_user_enhanced_fast_string+0xe/0x20 May 16 15:47:35 Unraid kernel: Code: 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 83 fa 40 0f 82 70 ff ff ff 89 d1 <f3> a4 31 c0 0f 1f 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 83 May 16 15:47:35 Unraid kernel: RSP: 0018:ffffc9000ed13c80 EFLAGS: 00010206 May 16 15:47:35 Unraid kernel: RAX: 0000146fc4810000 RBX: 0000000000001000 RCX: 0000000000001000 May 16 15:47:35 Unraid kernel: RDX: 0000000000001000 RSI: ffff880432e2f000 RDI: 0000146fc480f000 May 16 15:47:35 Unraid kernel: RBP: ffffc9000ed13e28 R08: ffffea0010cb8bc0 R09: ffffc9000ed13c68 May 16 15:47:35 Unraid kernel: R10: ffff88044c3007b8 R11: 0000000000000000 R12: ffff880041059e70 May 16 15:47:35 Unraid kernel: R13: ffff880432e2f000 R14: 0000000000001000 R15: 0000000000001000 May 16 15:47:35 Unraid kernel: copyout+0x22/0x27 May 16 15:47:35 Unraid kernel: copy_page_to_iter+0x122/0x275 May 16 15:47:35 Unraid kernel: generic_file_read_iter+0x31b/0x6c8 May 16 15:47:35 Unraid kernel: xfs_file_buffered_aio_read+0x4a/0x64 [xfs] May 16 15:47:35 Unraid kernel: xfs_file_read_iter+0x6f/0xb6 [xfs] May 16 15:47:35 Unraid kernel: do_iter_readv_writev+0x10d/0x143 May 16 15:47:35 Unraid kernel: do_iter_read+0x86/0x15e May 16 15:47:35 Unraid kernel: vfs_readv+0x6a/0xa1 May 16 15:47:35 Unraid kernel: ? __seccomp_filter+0x39/0x1ed May 16 15:47:35 Unraid kernel: ? syscall_trace_enter+0x164/0x1a3 May 16 15:47:35 Unraid kernel: do_preadv+0x69/0xab May 16 15:47:35 Unraid kernel: do_syscall_64+0x57/0xe6 May 16 15:47:35 Unraid kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 May 16 15:47:35 Unraid kernel: RIP: 0033:0x146fd3be75ea May 16 15:47:35 Unraid kernel: Code: 89 d4 55 48 89 f5 53 89 fb 48 83 ec 18 e8 7e 81 01 00 45 31 c0 4d 89 ea 44 89 e2 41 89 c1 48 89 ee 89 df b8 27 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 32 44 89 cf 48 89 44 24 08 e8 b1 81 01 00 48 May 16 15:47:35 Unraid kernel: RSP: 002b:0000146eb9bc1ed0 EFLAGS: 00000246 ORIG_RAX: 0000000000000127 May 16 15:47:35 Unraid kernel: RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 0000146fd3be75ea May 16 15:47:35 Unraid kernel: RDX: 0000000000000020 RSI: 0000146ebb89be00 RDI: 0000000000000014 May 16 15:47:35 Unraid kernel: RBP: 0000146ebb89be00 R08: 0000000000000000 R09: 0000000000000000 May 16 15:47:35 Unraid kernel: R10: 0000000213d90000 R11: 0000000000000246 R12: 0000000000000020 May 16 15:47:35 Unraid kernel: R13: 0000000213d90000 R14: 0000146fcf222680 R15: 0000000000000000 Not sure if I am reading it right, but one of the cpu cores begins to cause a problem which causes unraid to become unstable. Unraid seems to use the term "tainted". Am I right? Do I need to delete and rebuild this vm? Any help would be appreciated. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.