System becomes unresponsive while streaming video

[eds]

For, perhaps the fourth time this month, my unraid system has become unresponsive while streaming video either with Kodi or with Plex running on a local machine.

 

It seems to happen after the system has been running for about six days. 

 

In addition to the video becoming unresponsive, I notice the memory hits about 91 to 92%.   When I try to shutdown using unraid, the system hangs when trying to shutdown php-fpm.  

At that point I have to do a warm reboot. 

 

I am not sure what the problem is.  Any help would be appreciated. 

 

Thanks.

unraid-diagnostics-20190428-2327.zip

[eds]

Any help??

[sylinen]

I've had a similar issue, and it usually triggers for me also at around 6 days. The crash usually coincides with fast-forwarding or rewinding in Plex.

[eds]

Yes.   Plex or Kodi for me.   Did it stop for you?

Do you have vms running when it happens to you?

I have a deepin, windows server and windows 7 vm.

The windows 7 vm  was not running, so I've crossed that off the list. 

Also the issue with php-fpm seems to be linked to docker.    

 

[sylinen]

2 VMs, an Ubuntu server for pihole and one general purpose Manjaro VM. Haven't tried disabling them. I'm running the official Plex Inc. docker if that matters. Ryzen 1700X for CPU. 

[eds]

OK, I think I have narrowed what may be a problem. 

 

The windows server vm appears to be the problem of one (or perhaps all) of my crashes. 

 

I was able to catch the log as it was being written when I started the vm and it crashes unraid. 

 

Here is the relevant section, for what I was able to copy:

 

Quote

ay 16 15:46:48 Unraid kernel: NETDEV WATCHDOG: eth2 (igb): transmit queue 2 timed out
May 16 15:46:48 Unraid kernel: WARNING: CPU: 5 PID: 30117 at net/sched/sch_generic.c:461 dev_watchdog+0x150/0x1a8
May 16 15:46:48 Unraid kernel: Modules linked in: veth nvidia_uvm(O) macvlan xt_CHECKSUM iptable_mangle ipt_REJECT ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap xt_nat ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs nfsd lockd grace sunrpc md_mod it87 hwmon_vid be2net igb i2c_algo_bit nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel drm_kms_helper aes_x86_64 crypto_simd cryptd drm glue_helper intel_cstate intel_uncore i2c_i801 ipmi_ssif agpgart intel_rapl_perf i2c_core ahci libahci syscopyarea sysfillrect sysimgblt fb_sys_fops wmi acpi_power_meter ipmi_si acpi_pad pcc_cpufreq button [last unloaded: be2net]
May 16 15:46:48 Unraid kernel: CPU: 5 PID: 30117 Comm: CPU 4/KVM Tainted: P D W O 4.18.20-unRAID #1
May 16 15:46:48 Unraid kernel: Hardware name: GIGABYTE MD30-RS0-XX/MD30-RS0-XX, BIOS R08 06/15/2018
May 16 15:46:48 Unraid kernel: RIP: 0010:dev_watchdog+0x150/0x1a8
May 16 15:46:48 Unraid kernel: Code: 69 f8 97 00 00 75 36 4c 89 ef c6 05 5d f8 97 00 01 e8 29 c5 fd ff 89 e9 4c 89 ee 48 c7 c7 66 0f d9 81 48 89 c2 e8 f9 bb b2 ff <0f> 0b eb 0f ff c5 48 81 c2 40 01 00 00 39 cd 75 98 eb 13 48 8b 83 
May 16 15:46:48 Unraid kernel: RSP: 0018:ffff88056b943ea0 EFLAGS: 00010286
May 16 15:46:48 Unraid kernel: RAX: 0000000000000000 RBX: ffff8805497ec3b0 RCX: 0000000000000007
May 16 15:46:48 Unraid kernel: RDX: 0000000000000006 RSI: ffff88056b956470 RDI: ffff88056b956470
May 16 15:46:48 Unraid kernel: RBP: 0000000000000002 R08: 000000000000000f R09: ffff8800000bef00
May 16 15:46:48 Unraid kernel: R10: 0000000000000b17 R11: 0000000000036ac8 R12: ffff8805497ec39c
May 16 15:46:48 Unraid kernel: R13: ffff8805497ec000 R14: ffff880546f4c940 R15: 0000000000000005
May 16 15:46:48 Unraid kernel: FS: 000014cb1c8f3700(0000) GS:ffff88056b940000(0000) knlGS:00000000fff6e000
May 16 15:46:48 Unraid kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 15:46:48 Unraid kernel: CR2: 000000003d4dc000 CR3: 000000086758e006 CR4: 00000000001626e0
May 16 15:46:48 Unraid kernel: Call Trace:
May 16 15:46:48 Unraid kernel: <IRQ>
May 16 15:46:48 Unraid kernel: call_timer_fn+0x18/0x7b
May 16 15:46:48 Unraid kernel: ? qdisc_reset+0xc0/0xc0
May 16 15:46:48 Unraid kernel: expire_timers+0x7f/0x8e
May 16 15:46:48 Unraid kernel: run_timer_softirq+0x72/0x120
May 16 15:46:48 Unraid kernel: ? enqueue_hrtimer.isra.3+0x23/0x27
May 16 15:46:48 Unraid kernel: ? __hrtimer_run_queues+0xd7/0x105
May 16 15:46:48 Unraid kernel: ? recalibrate_cpu_khz+0x1/0x1
May 16 15:46:48 Unraid kernel: ? ktime_get+0x3a/0x8d
May 16 15:46:48 Unraid kernel: __do_softirq+0xce/0x1e2
May 16 15:46:48 Unraid kernel: irq_exit+0x5e/0x9d
May 16 15:46:48 Unraid kernel: smp_apic_timer_interrupt+0x7e/0x91
May 16 15:46:48 Unraid kernel: apic_timer_interrupt+0xf/0x20
May 16 15:46:48 Unraid kernel: </IRQ>
May 16 15:46:48 Unraid kernel: RIP: 0010:vmx_handle_external_intr+0x65/0x66 [kvm_intel]
May 16 15:46:48 Unraid kernel: Code: 0f b7 50 06 8b 48 08 0f b7 00 48 c1 e2 10 48 c1 e1 20 48 09 ca 48 09 d0 48 89 e2 48 83 e4 f0 6a 18 52 9c 6a 10 e8 4a f6 1d e1 <c3> 31 c0 f6 05 7e 96 02 00 80 74 0a 0f b7 05 79 96 02 00 83 e0 01 
May 16 15:46:48 Unraid kernel: RSP: 0018:ffffc90006e0fd58 EFLAGS: 00000086 ORIG_RAX: ffffffffffffff13
May 16 15:46:48 Unraid kernel: RAX: ffffffff81801730 RBX: ffff880865c68000 RCX: ffffffff00000000
May 16 15:46:48 Unraid kernel: RDX: ffffc90006e0fd58 RSI: 0009c6c068bde7f4 RDI: ffff880865c68000
May 16 15:46:48 Unraid kernel: RBP: 0000000000000000 R08: fff7625b2df1fdb2 R09: 0000000000000000
May 16 15:46:48 Unraid kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
May 16 15:46:48 Unraid kernel: R13: 0000000000000000 R14: 0000000000000000 R15: ffff880865c68058
May 16 15:46:48 Unraid kernel: ? __entry_text_end+0x7/0x7
May 16 15:46:48 Unraid kernel: ? vmx_get_exit_info+0x47/0x47 [kvm_intel]
May 16 15:46:48 Unraid kernel: ? kvm_arch_vcpu_ioctl_run+0xf85/0x1298 [kvm]
May 16 15:46:48 Unraid kernel: ? kvm_vcpu_ioctl+0x187/0x4ba [kvm]
May 16 15:46:48 Unraid kernel: ? __seccomp_filter+0x39/0x1ed
May 16 15:46:48 Unraid kernel: ? vfs_ioctl+0x19/0x26
May 16 15:46:48 Unraid kernel: ? do_vfs_ioctl+0x518/0x540
May 16 15:46:48 Unraid kernel: ? ksys_ioctl+0x39/0x58
May 16 15:46:48 Unraid kernel: ? __x64_sys_ioctl+0x11/0x14
May 16 15:46:48 Unraid kernel: ? do_syscall_64+0x57/0xe6
May 16 15:46:48 Unraid kernel: ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 16 15:46:48 Unraid kernel: ---[ end trace b90117603784f00c ]---
May 16 15:46:48 Unraid kernel: igb 0000:07:00.0 eth2: Reset adapter
May 16 15:46:48 Unraid kernel: igb 0000:07:00.0 eth2: igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
May 16 15:47:35 Unraid kernel: INFO: rcu_sched self-detected stall on CPU
May 16 15:47:35 Unraid kernel: 35-....: (59999 ticks this GP) idle=b4a/1/4611686018427387906 softirq=14417731/14417731 fqs=14946 
May 16 15:47:35 Unraid kernel: (t=60000 jiffies g=16469846 c=16469845 q=69705)
May 16 15:47:35 Unraid kernel: NMI backtrace for cpu 35
May 16 15:47:35 Unraid kernel: CPU: 35 PID: 37690 Comm: worker Tainted: P D W O 4.18.20-unRAID #1
May 16 15:47:35 Unraid kernel: Hardware name: GIGABYTE MD30-RS0-XX/MD30-RS0-XX, BIOS R08 06/15/2018
May 16 15:47:35 Unraid kernel: Call Trace:
May 16 15:47:35 Unraid kernel: <IRQ>
May 16 15:47:35 Unraid kernel: dump_stack+0x5d/0x79
May 16 15:47:35 Unraid kernel: nmi_cpu_backtrace+0x71/0x83
May 16 15:47:35 Unraid kernel: ? lapic_can_unplug_cpu+0x8e/0x8e
May 16 15:47:35 Unraid kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7
May 16 15:47:35 Unraid kernel: rcu_dump_cpu_stacks+0x91/0xbb
May 16 15:47:35 Unraid kernel: rcu_check_callbacks+0x23f/0x5ca
May 16 15:47:35 Unraid kernel: ? tick_sched_handle.isra.5+0x2f/0x2f
May 16 15:47:35 Unraid kernel: update_process_times+0x23/0x45
May 16 15:47:35 Unraid kernel: tick_sched_timer+0x36/0x64
May 16 15:47:35 Unraid kernel: __hrtimer_run_queues+0xb1/0x105
May 16 15:47:35 Unraid kernel: hrtimer_interrupt+0xf4/0x20d
May 16 15:47:35 Unraid kernel: smp_apic_timer_interrupt+0x79/0x91
May 16 15:47:35 Unraid kernel: apic_timer_interrupt+0xf/0x20
May 16 15:47:35 Unraid kernel: </IRQ>
May 16 15:47:35 Unraid kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x165/0x16d
May 16 15:47:35 Unraid kernel: Code: eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d ba d9 f9 7e c3 85 d2 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 c8 61 b9 40 00 00 
May 16 15:47:35 Unraid kernel: RSP: 0018:ffffc9000ed135e0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
May 16 15:47:35 Unraid kernel: RAX: 0000000000000101 RBX: ffffc9000ed13658 RCX: 0000000000000101
May 16 15:47:35 Unraid kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffea000f943628
May 16 15:47:35 Unraid kernel: RBP: ffffea000e9d83c0 R08: ffff880000000000 R09: ffffea000e9d8400
May 16 15:47:35 Unraid kernel: R10: ffff88057fffc000 R11: 0000000000025098 R12: ffff88054d20e600
May 16 15:47:35 Unraid kernel: R13: ffffc9000ed13720 R14: ffff8804bdf53980 R15: ffff8803efe65c80
May 16 15:47:35 Unraid kernel: _raw_spin_lock+0x16/0x19
May 16 15:47:35 Unraid kernel: page_vma_mapped_walk+0x3d9/0x5e1
May 16 15:47:35 Unraid kernel: try_to_unmap_one+0xfe/0x62c
May 16 15:47:35 Unraid kernel: rmap_walk_anon+0x163/0x1d3
May 16 15:47:35 Unraid kernel: ? __ClearPageMovable+0x6/0x6
May 16 15:47:35 Unraid kernel: try_to_unmap+0x9e/0xde
May 16 15:47:35 Unraid kernel: ? page_remove_rmap+0x29c/0x29c
May 16 15:47:35 Unraid kernel: ? page_not_mapped+0xc/0xc
May 16 15:47:35 Unraid kernel: ? page_get_anon_vma+0x5c/0x5c
May 16 15:47:35 Unraid kernel: ? is_vma_temporary_stack+0x1d/0x1d
May 16 15:47:35 Unraid kernel: migrate_pages+0x307/0x635
May 16 15:47:35 Unraid kernel: ? isolate_migratepages_block+0x634/0x634
May 16 15:47:35 Unraid kernel: compact_zone+0x353/0x534
May 16 15:47:35 Unraid kernel: compact_zone_order+0x92/0xac
May 16 15:47:35 Unraid kernel: try_to_compact_pages+0x9b/0x12e
May 16 15:47:35 Unraid kernel: __alloc_pages_direct_compact+0x46/0xd8
May 16 15:47:35 Unraid kernel: __alloc_pages_nodemask+0x598/0xa8f
May 16 15:47:35 Unraid kernel: ? alloc_pages_vma+0xe0/0x173
May 16 15:47:35 Unraid kernel: do_huge_pmd_anonymous_page+0x267/0x4d0
May 16 15:47:35 Unraid kernel: __handle_mm_fault+0x2c0/0x10aa
May 16 15:47:35 Unraid kernel: handle_mm_fault+0x158/0x1a7
May 16 15:47:35 Unraid kernel: __do_page_fault+0x271/0x40b
May 16 15:47:35 Unraid kernel: page_fault+0x1e/0x30
May 16 15:47:35 Unraid kernel: RIP: 0010:copy_user_enhanced_fast_string+0xe/0x20
May 16 15:47:35 Unraid kernel: Code: 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 83 fa 40 0f 82 70 ff ff ff 89 d1 <f3> a4 31 c0 0f 1f 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 83 
May 16 15:47:35 Unraid kernel: RSP: 0018:ffffc9000ed13c80 EFLAGS: 00010206
May 16 15:47:35 Unraid kernel: RAX: 0000146fc4810000 RBX: 0000000000001000 RCX: 0000000000001000
May 16 15:47:35 Unraid kernel: RDX: 0000000000001000 RSI: ffff880432e2f000 RDI: 0000146fc480f000
May 16 15:47:35 Unraid kernel: RBP: ffffc9000ed13e28 R08: ffffea0010cb8bc0 R09: ffffc9000ed13c68
May 16 15:47:35 Unraid kernel: R10: ffff88044c3007b8 R11: 0000000000000000 R12: ffff880041059e70
May 16 15:47:35 Unraid kernel: R13: ffff880432e2f000 R14: 0000000000001000 R15: 0000000000001000
May 16 15:47:35 Unraid kernel: copyout+0x22/0x27
May 16 15:47:35 Unraid kernel: copy_page_to_iter+0x122/0x275
May 16 15:47:35 Unraid kernel: generic_file_read_iter+0x31b/0x6c8
May 16 15:47:35 Unraid kernel: xfs_file_buffered_aio_read+0x4a/0x64 [xfs]
May 16 15:47:35 Unraid kernel: xfs_file_read_iter+0x6f/0xb6 [xfs]
May 16 15:47:35 Unraid kernel: do_iter_readv_writev+0x10d/0x143
May 16 15:47:35 Unraid kernel: do_iter_read+0x86/0x15e
May 16 15:47:35 Unraid kernel: vfs_readv+0x6a/0xa1
May 16 15:47:35 Unraid kernel: ? __seccomp_filter+0x39/0x1ed
May 16 15:47:35 Unraid kernel: ? syscall_trace_enter+0x164/0x1a3
May 16 15:47:35 Unraid kernel: do_preadv+0x69/0xab
May 16 15:47:35 Unraid kernel: do_syscall_64+0x57/0xe6
May 16 15:47:35 Unraid kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 16 15:47:35 Unraid kernel: RIP: 0033:0x146fd3be75ea
May 16 15:47:35 Unraid kernel: Code: 89 d4 55 48 89 f5 53 89 fb 48 83 ec 18 e8 7e 81 01 00 45 31 c0 4d 89 ea 44 89 e2 41 89 c1 48 89 ee 89 df b8 27 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 32 44 89 cf 48 89 44 24 08 e8 b1 81 01 00 48 
May 16 15:47:35 Unraid kernel: RSP: 002b:0000146eb9bc1ed0 EFLAGS: 00000246 ORIG_RAX: 0000000000000127
May 16 15:47:35 Unraid kernel: RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 0000146fd3be75ea
May 16 15:47:35 Unraid kernel: RDX: 0000000000000020 RSI: 0000146ebb89be00 RDI: 0000000000000014
May 16 15:47:35 Unraid kernel: RBP: 0000146ebb89be00 R08: 0000000000000000 R09: 0000000000000000
May 16 15:47:35 Unraid kernel: R10: 0000000213d90000 R11: 0000000000000246 R12: 0000000000000020
May 16 15:47:35 Unraid kernel: R13: 0000000213d90000 R14: 0000146fcf222680 R15: 0000000000000000

 

Not sure if I am reading it right, but one of the cpu cores begins to cause a problem which causes unraid to become unstable.  Unraid seems to use the term "tainted".   Am I right?  Do I need to delete and rebuild this vm?

 

Any help would be appreciated.