Jerky_san
Members
-
Joined
Everything posted by Jerky_san
-
It does it simply doesn't work on threadripper. Yet another thing that is broken in QMEU. There is a patch for that as well to make SMT work properly.
-
Change the below into <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='4' threads='1'/> </cpu> This but remember to change the cores to half(if your using SMT) of what you assigned. So 16 cores(8 SMT) would be 8 cores 2 threads. Also if you ever change anything on your config you'll have to re-apply this as unraid will alter it back to the above. <cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>EPYC-IBPB</model> <topology sockets='1' cores='3' threads='2'/> <feature policy='require' name='topoext'/> </cpu>
-
A guy on reddit told me this code would fix it. I also requested the fix in bug fixes and posted a few posts. Basically we just need the option to turn on this patch.
-
You have to manually adjust the cores/threads and even then I don't believe QEMU plays nicely with SMT cores on threadripper yet but I've not noticed any ill effects.
-
For me the EYPC tweaks helped a lot especially under high load.. If the VM thinks the cache is much bigger than expected it might cause issues. Anyways all you can do is try and see.
-
Mine wasn't called numa. Mine was auto, die, and channel. Channel is best on mine. From a guide I used In my case, under Advanced -> CBS -> DF there is a Memory model item that has choices like auto | distribute | channel. In our case, we are interested in channel mode, as it will expose NUMA information to the host once more.
-
Would like to add I no longer know when people are using my plex server.. I hope that this patch is added in or QEMU gets updated and then I can span numas but damn it's great so far. Can't believe what cache does for me
-
how about your VM feel any better?
-
Don't know why it wouldn't start but since you have your VM's split between the nodes it looks like its using the other node's ram. Still strange though.
-
Thats strange should of started then. Run sudo dmidecode -t memory | grep -i size and post
-
How much ram do you have?
-
numastat -c qemu
-
So the "node set" function tells it where to pull memory from. It should only pull memory from the node its assigned to die wise. It always pulls a a little ram from the other side. 64081 is my main gaming VM. Also is your ram populated across at least 4 dims? Per-node process memory usage (in MBs) PID Node 0 Node 1 Node 2 Node 3 Total --------------- ------ ------ ------ ------ ----- 52912 (qemu-syst 4 0 1510 0 1514 64081 (qemu-syst 26427 0 1833 0 28260 --------------- ------ ------ ------ ------ ----- Total 26431 0 3343 0 29773
-
So I probably should of set that to "strict" instead of "interleave" as that is for when I am spanning two numas.
-
The biggest change for me was L3. The memory latency was something in the VM itself and some tweaks.(I purged the whole VM) The L3 was a 6x decrease in latency. L1 for me was 3ns now 1ns and thats massive for L1 as its used non stop but the biggest thing is that the cache is properly allocated. Before if you look i had 5x16 on my L3 which is literally impossible and L1 was 2x larger than it was supposed to be and only 2 way instead of 8 way. Also with this change you can't span numa. So if you have any procs allocated outside a single numa it will cause more latency on memory. If we can get it to identify threadripper properly then we can do the NUMA cross and be fine. Should also state I have 2990wx and I have my memory set to channel from "auto" in the bios that provided a large increase in performance as well. "In my case, under Advanced -> CBS -> DF there is a Memory model item that has choices like auto | distribute | channel. In our case, we are interested in channel mode, as it will expose NUMA information to the host once more." <vcpu placement='static'>14</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='33'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='34'/> <vcpupin vcpu='4' cpuset='3'/> <vcpupin vcpu='5' cpuset='35'/> <vcpupin vcpu='6' cpuset='4'/> <vcpupin vcpu='7' cpuset='36'/> <vcpupin vcpu='8' cpuset='5'/> <vcpupin vcpu='9' cpuset='37'/> <vcpupin vcpu='10' cpuset='6'/> <vcpupin vcpu='11' cpuset='38'/> <vcpupin vcpu='12' cpuset='7'/> <vcpupin vcpu='13' cpuset='39'/> <emulatorpin cpuset='1-7'/> </cputune> <numatune> <memory mode='interleave' nodeset='0'/> </numatune>
-
A person on reddit told me the answer to my problem. If you do the below QEMU provides EPYC instead and all the cache is right. It dropped latency accrossed the board. L3 is down to 13ns and l1-1ns l2-2-3ns. Machine seems MUCH more responsive. Should also mention they said they had updated their kernel and a patch on QEMU made theirs see it properly without this code so hopefully we will see it on unraid as well. <cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>EPYC-IBPB</model> <topology sockets='1' cores='8' threads='2'/> <feature policy='require' name='topoext'/> </cpu> To give an idea of how much of a change. If your wondering why the read/write/copy of the Ls is "higher" on the old its because I had cores bounding from multiple numa in an attempt to make things faster. New is from numa 0 with 1-7 and its SMT cores bound. Edit: An update to this is it decreased latency across the board but you cannot have multiple NUMA. So far stating numa topology fails and the OS is unaware of when you cross NUMA. Working on that part since once I fix that I will be able to raise my read/write/copy speeds a lot. Edit2: I can report substantial increases in FPS for my games and the stutter thus far has been eliminated. Old New Physical Baremetal performance CPUZ Baremetal Virtual Old Virtual New Another user posted this to me So the Qemu code does not allow the passing of the cache topology. Here is a patch for 3.0.0 that allows chache topology to be passed through: diff --git a/target/i386/cpu.c b/target/i386/cpu.c index 723e02221e..2912867872 100644 --- a/target/i386/cpu.c +++ b/target/i386/cpu.c @@ -4221,6 +4221,10 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, break; case 0x8000001D: *eax = 0; + if (cpu->cache_info_passthrough) { + host_cpuid(index, count, eax, ebx, ecx, edx); + break; + } switch (count) { case 0: /* L1 dcache info */ encode_cache_cpuid8000001d(env->cache_info_amd.l1d_cache, cs, Use at your own risk.
-
Even if I don't 100% trust CPUZ or ignore all my experience with Hyper-V and VMware(over 10 years and going now) and it being accurate in those instances. Even Linux reads the wrong cache size with LSCPU Ubuntu VM L1d cache: 64K L1i cache: 64K L2 cache: 512K L3 cache: 16384K L1d cache is wrong and l3 cache is double the size that it should be. Unraid: L1d cache: 32K L1i cache: 64K L2 cache: 512K L3 cache: 8192K
-
So I noticed something today.. It finally hit me. Why is latency so high? Because the VM isn't getting the correct information. Look at the pics below. The left is what a 2990wx looks like in windows bare metal. The right is my VM. Now granted I'm only passing 20 cores so I expect certain things but look at the cache sizes. L1 is double and only 2 way instead of 8-way. L1 inst is 2 instead of 4 way, l2 is the right size but 16 way instead of 8 way and l3 is 5x 16MB instead of of 8x8 mbytes. Can anyone confirm this?
-
I tried doing this with my x399 board (Asus Zenith) and it will let you pass the wifi but the bluetooth is mia.. never figured out how to get bluetooth working.
-
I turned off SMT for an experiment. The even/odd when SMT is off are physical cores but they are very weirdly broken up. There is a bios programmer over in overclocker's forum that works on my board so I asked him if he'd look at it but so far not much of an answer.
-
Received a bunch of bug checks after rebooting. Wish I had more to really add-on to that statement but just restarted and saw it in the error log. The VM page doesn't seem to load now either. Oct 8 16:12:50 Tower kernel: INFO: rcu_sched self-detected stall on CPU Oct 8 16:12:50 Tower kernel: 25-....: (59999 ticks this GP) idle=c3a/1/4611686018427387906 softirq=36453/36462 fqs=14915 Oct 8 16:12:50 Tower kernel: (t=60000 jiffies g=24287 c=24286 q=300188) Oct 8 16:12:50 Tower kernel: NMI backtrace for cpu 25 Oct 8 16:12:50 Tower kernel: CPU: 25 PID: 18182 Comm: apps.plugin Not tainted 4.18.10-unRAID #2 Oct 8 16:12:50 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:12:50 Tower kernel: Call Trace: Oct 8 16:12:50 Tower kernel: <IRQ> Oct 8 16:12:50 Tower kernel: dump_stack+0x5d/0x79 Oct 8 16:12:50 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Oct 8 16:12:50 Tower kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Oct 8 16:12:50 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7 Oct 8 16:12:50 Tower kernel: rcu_dump_cpu_stacks+0x91/0xbb Oct 8 16:12:50 Tower kernel: rcu_check_callbacks+0x23f/0x5ca Oct 8 16:12:50 Tower kernel: ? tick_sched_handle.isra.5+0x2f/0x2f Oct 8 16:12:50 Tower kernel: update_process_times+0x23/0x45 Oct 8 16:12:50 Tower kernel: tick_sched_timer+0x36/0x64 Oct 8 16:12:50 Tower kernel: __hrtimer_run_queues+0xb1/0x105 Oct 8 16:12:50 Tower kernel: hrtimer_interrupt+0xf4/0x20d Oct 8 16:12:50 Tower kernel: smp_apic_timer_interrupt+0x79/0x89 Oct 8 16:12:50 Tower kernel: apic_timer_interrupt+0xf/0x20 Oct 8 16:12:50 Tower kernel: </IRQ> Oct 8 16:12:50 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x165/0x16d Oct 8 16:12:50 Tower kernel: Code: eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d 1a de f9 7e c3 85 d2 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 c8 61 b9 40 00 00 Oct 8 16:12:50 Tower kernel: RSP: 0018:ffffc90011c33c70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 Oct 8 16:12:50 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000102 Oct 8 16:12:50 Tower kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8807c1aff858 Oct 8 16:12:50 Tower kernel: RBP: ffff880f5dae2900 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:12:50 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:12:50 Tower kernel: R13: ffffffffffffffff R14: ffffc90011c33df0 R15: ffff8807c18dfa40 Oct 8 16:12:50 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:12:50 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:12:50 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:12:50 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:12:50 Tower kernel: ? __bpf_prog_run32+0x2e/0x48 Oct 8 16:12:50 Tower kernel: ? do_io_accounting+0x18b/0x1d4 Oct 8 16:12:50 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:12:50 Tower kernel: ? mutex_lock+0xa/0x25 Oct 8 16:12:50 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:12:50 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:12:50 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:12:50 Tower kernel: RIP: 0033:0x14864c1c6820 Oct 8 16:12:50 Tower kernel: Code: 73 01 c3 48 8b 0d 88 77 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d a9 bb 20 00 00 75 10 b8 02 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f6 ff ff 48 89 04 24 Oct 8 16:12:50 Tower kernel: RSP: 002b:00007ffc45b829f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002 Oct 8 16:12:50 Tower kernel: RAX: ffffffffffffffda RBX: 00005594a68036e0 RCX: 000014864c1c6820 Oct 8 16:12:50 Tower kernel: RDX: 00000000000001b6 RSI: 0000000000020000 RDI: 00005594a6c01920 Oct 8 16:12:50 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffd0 Oct 8 16:12:50 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005594a6c01920 Oct 8 16:12:50 Tower kernel: R13: 00005594a6804d0c R14: 000000000000001e R15: 00005594a6c012f0 Oct 8 16:13:31 Tower emhttpd: req (1): cmd=/plugins/dynamix.plugin.manager/scripts/plugin&arg1=update&arg2=community.applications.plg&csrf_token=**************** Oct 8 16:13:31 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugin update community.applications.plg Oct 8 16:13:31 Tower root: plugin: running: anonymous Oct 8 16:13:31 Tower root: plugin: running: anonymous Oct 8 16:13:31 Tower root: plugin: creating: /boot/config/plugins/community.applications/community.applications-2018.10.08.txz - downloading from URL https://raw.github.com/Squidly271/community.applications/master/archive/community.applications-2018.10.08.txz Oct 8 16:13:32 Tower root: plugin: checking: /boot/config/plugins/community.applications/community.applications-2018.10.08.txz - MD5 Oct 8 16:13:32 Tower root: plugin: running: /boot/config/plugins/community.applications/community.applications-2018.10.08.txz Oct 8 16:13:32 Tower root: plugin: running: anonymous Oct 8 16:14:02 Tower ntpd[2710]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Oct 8 16:14:20 Tower kernel: INFO: rcu_bh self-detected stall on CPU Oct 8 16:14:20 Tower kernel: 19-....: (1 GPs behind) idle=0ee/1/4611686018427387906 softirq=0/22976 fqs=14736 Oct 8 16:14:20 Tower kernel: (t=60001 jiffies g=-299 c=-300 q=4) Oct 8 16:14:20 Tower kernel: NMI backtrace for cpu 19 Oct 8 16:14:20 Tower kernel: CPU: 19 PID: 38544 Comm: pgrep Not tainted 4.18.10-unRAID #2 Oct 8 16:14:20 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:14:20 Tower kernel: Call Trace: Oct 8 16:14:20 Tower kernel: <IRQ> Oct 8 16:14:20 Tower kernel: dump_stack+0x5d/0x79 Oct 8 16:14:20 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Oct 8 16:14:20 Tower kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Oct 8 16:14:20 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7 Oct 8 16:14:20 Tower kernel: rcu_dump_cpu_stacks+0x91/0xbb Oct 8 16:14:20 Tower kernel: rcu_check_callbacks+0x23f/0x5ca Oct 8 16:14:20 Tower kernel: ? tick_sched_handle.isra.5+0x2f/0x2f Oct 8 16:14:20 Tower kernel: update_process_times+0x23/0x45 Oct 8 16:14:20 Tower kernel: tick_sched_timer+0x36/0x64 Oct 8 16:14:20 Tower kernel: __hrtimer_run_queues+0xb1/0x105 Oct 8 16:14:20 Tower kernel: hrtimer_interrupt+0xf4/0x20d Oct 8 16:14:20 Tower kernel: smp_apic_timer_interrupt+0x79/0x89 Oct 8 16:14:20 Tower kernel: apic_timer_interrupt+0xf/0x20 Oct 8 16:14:20 Tower kernel: </IRQ> Oct 8 16:14:20 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x115/0x16d Oct 8 16:14:20 Tower kernel: Code: c0 17 02 00 48 03 04 cd 00 17 da 81 48 89 10 8b 42 08 85 c0 75 04 f3 90 eb f5 48 8b 0a 48 85 c9 74 c9 0f 0d 09 8b 07 66 85 c0 <74> 04 f3 90 eb f5 41 89 c0 66 45 31 c0 44 39 c6 74 0a 48 85 c9 c6 Oct 8 16:14:20 Tower kernel: RSP: 0018:ffffc900174f3c70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 Oct 8 16:14:20 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000000 Oct 8 16:14:20 Tower kernel: RDX: ffff88085f1e17c0 RSI: 0000000000500000 RDI: ffff8807c1aff858 Oct 8 16:14:20 Tower kernel: RBP: ffff880667174c00 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:14:20 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:14:20 Tower kernel: R13: ffffffffffffffff R14: ffffc900174f3df0 R15: ffff8807c18dfa40 Oct 8 16:14:20 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:14:20 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:14:20 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:14:20 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:14:20 Tower kernel: ? filename_lookup.part.16+0xa5/0xcc Oct 8 16:14:20 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:14:20 Tower kernel: ? _copy_to_user+0x22/0x28 Oct 8 16:14:20 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:14:20 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:14:20 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:14:20 Tower kernel: RIP: 0033:0x14f0a712bff0 Oct 8 16:14:20 Tower kernel: Code: 25 00 00 41 00 3d 00 00 41 00 74 36 48 8d 05 7f d9 2d 00 8b 00 85 c0 75 5a 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 84 00 00 00 48 83 c4 68 5b 5d c3 0f 1f 44 Oct 8 16:14:20 Tower kernel: RSP: 002b:00007ffc98266830 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 Oct 8 16:14:20 Tower kernel: RAX: ffffffffffffffda RBX: 00007ffc982668c0 RCX: 000014f0a712bff0 Oct 8 16:14:20 Tower kernel: RDX: 0000000000000000 RSI: 00007ffc982668c0 RDI: 00000000ffffff9c Oct 8 16:14:20 Tower kernel: RBP: 00007ffc98266ae0 R08: 000014f0a761b946 R09: 0000000000000000 Oct 8 16:14:20 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014f0a7820950 Oct 8 16:14:20 Tower kernel: R13: 00000000006074e0 R14: 0000000000000020 R15: 0000000000000001 Oct 8 16:14:20 Tower kernel: Sending NMI from CPU 19 to CPUs 25: Oct 8 16:14:20 Tower kernel: NMI backtrace for cpu 25 Oct 8 16:14:20 Tower kernel: CPU: 25 PID: 18182 Comm: apps.plugin Not tainted 4.18.10-unRAID #2 Oct 8 16:14:20 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:14:20 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x165/0x16d Oct 8 16:14:20 Tower kernel: Code: eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d 1a de f9 7e c3 85 d2 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 c8 61 b9 40 00 00 Oct 8 16:14:20 Tower kernel: RSP: 0018:ffffc90011c33c70 EFLAGS: 00000202 Oct 8 16:14:20 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000102 Oct 8 16:14:20 Tower kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8807c1aff858 Oct 8 16:14:20 Tower kernel: RBP: ffff880f5dae2900 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:14:20 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:14:20 Tower kernel: R13: ffffffffffffffff R14: ffffc90011c33df0 R15: ffff8807c18dfa40 Oct 8 16:14:20 Tower kernel: FS: 000014864c8f2700(0000) GS:ffff88107ef40000(0000) knlGS:0000000000000000 Oct 8 16:14:20 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 8 16:14:20 Tower kernel: CR2: 00005594a6c01000 CR3: 00000007bddf6000 CR4: 00000000003406e0 Oct 8 16:14:20 Tower kernel: Call Trace: Oct 8 16:14:20 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:14:20 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:14:20 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:14:20 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:14:20 Tower kernel: ? __bpf_prog_run32+0x2e/0x48 Oct 8 16:14:20 Tower kernel: ? do_io_accounting+0x18b/0x1d4 Oct 8 16:14:20 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:14:20 Tower kernel: ? mutex_lock+0xa/0x25 Oct 8 16:14:20 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:14:20 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:14:20 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:14:20 Tower kernel: RIP: 0033:0x14864c1c6820 Oct 8 16:14:20 Tower kernel: Code: 73 01 c3 48 8b 0d 88 77 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d a9 bb 20 00 00 75 10 b8 02 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f6 ff ff 48 89 04 24 Oct 8 16:14:20 Tower kernel: RSP: 002b:00007ffc45b829f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002 Oct 8 16:14:20 Tower kernel: RAX: ffffffffffffffda RBX: 00005594a68036e0 RCX: 000014864c1c6820 Oct 8 16:14:20 Tower kernel: RDX: 00000000000001b6 RSI: 0000000000020000 RDI: 00005594a6c01920 Oct 8 16:14:20 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffd0 Oct 8 16:14:20 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005594a6c01920 Oct 8 16:14:20 Tower kernel: R13: 00005594a6804d0c R14: 000000000000001e R15: 00005594a6c012f0 Oct 8 16:14:20 Tower kernel: INFO: rcu_bh detected stalls on CPUs/tasks: Oct 8 16:14:20 Tower kernel: 19-....: (1 GPs behind) idle=0ee/1/4611686018427387906 softirq=0/22976 fqs=14736 Oct 8 16:14:20 Tower kernel: 25-....: (1 GPs behind) idle=c3a/1/4611686018427387904 softirq=0/36462 fqs=14736 Oct 8 16:14:20 Tower kernel: (detected by 1, t=60002 jiffies, g=-299, c=-300, q=4) Oct 8 16:14:20 Tower kernel: Sending NMI from CPU 1 to CPUs 19: Oct 8 16:14:20 Tower kernel: NMI backtrace for cpu 19 Oct 8 16:14:20 Tower kernel: CPU: 19 PID: 38544 Comm: pgrep Not tainted 4.18.10-unRAID #2 Oct 8 16:14:20 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:14:20 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x119/0x16d Oct 8 16:14:20 Tower kernel: Code: 48 03 04 cd 00 17 da 81 48 89 10 8b 42 08 85 c0 75 04 f3 90 eb f5 48 8b 0a 48 85 c9 74 c9 0f 0d 09 8b 07 66 85 c0 74 04 f3 90 <eb> f5 41 89 c0 66 45 31 c0 44 39 c6 74 0a 48 85 c9 c6 07 01 75 1b Oct 8 16:14:20 Tower kernel: RSP: 0018:ffffc900174f3c70 EFLAGS: 00000202 Oct 8 16:14:20 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000000 Oct 8 16:14:20 Tower kernel: RDX: ffff88085f1e17c0 RSI: 0000000000500000 RDI: ffff8807c1aff858 Oct 8 16:14:20 Tower kernel: RBP: ffff880667174c00 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:14:20 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:14:20 Tower kernel: R13: ffffffffffffffff R14: ffffc900174f3df0 R15: ffff8807c18dfa40 Oct 8 16:14:20 Tower kernel: FS: 000014f0a7a48740(0000) GS:ffff88085f1c0000(0000) knlGS:0000000000000000 Oct 8 16:14:20 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 8 16:14:20 Tower kernel: CR2: 0000000000617198 CR3: 0000000667124000 CR4: 00000000003406e0 Oct 8 16:14:20 Tower kernel: Call Trace: Oct 8 16:14:20 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:14:20 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:14:20 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:14:20 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:14:20 Tower kernel: ? filename_lookup.part.16+0xa5/0xcc Oct 8 16:14:20 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:14:20 Tower kernel: ? _copy_to_user+0x22/0x28 Oct 8 16:14:20 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:14:20 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:14:20 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:14:20 Tower kernel: RIP: 0033:0x14f0a712bff0 Oct 8 16:14:20 Tower kernel: Code: 25 00 00 41 00 3d 00 00 41 00 74 36 48 8d 05 7f d9 2d 00 8b 00 85 c0 75 5a 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 84 00 00 00 48 83 c4 68 5b 5d c3 0f 1f 44 Oct 8 16:14:20 Tower kernel: RSP: 002b:00007ffc98266830 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 Oct 8 16:14:20 Tower kernel: RAX: ffffffffffffffda RBX: 00007ffc982668c0 RCX: 000014f0a712bff0 Oct 8 16:14:20 Tower kernel: RDX: 0000000000000000 RSI: 00007ffc982668c0 RDI: 00000000ffffff9c Oct 8 16:14:20 Tower kernel: RBP: 00007ffc98266ae0 R08: 000014f0a761b946 R09: 0000000000000000 Oct 8 16:14:20 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014f0a7820950 Oct 8 16:14:20 Tower kernel: R13: 00000000006074e0 R14: 0000000000000020 R15: 0000000000000001 Oct 8 16:14:20 Tower kernel: Sending NMI from CPU 1 to CPUs 25: Oct 8 16:14:20 Tower kernel: NMI backtrace for cpu 25 Oct 8 16:14:20 Tower kernel: CPU: 25 PID: 18182 Comm: apps.plugin Not tainted 4.18.10-unRAID #2 Oct 8 16:14:20 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:14:20 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x165/0x16d Oct 8 16:14:20 Tower kernel: Code: eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d 1a de f9 7e c3 85 d2 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 c8 61 b9 40 00 00 Oct 8 16:14:20 Tower kernel: RSP: 0018:ffffc90011c33c70 EFLAGS: 00000202 Oct 8 16:14:20 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000102 Oct 8 16:14:20 Tower kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8807c1aff858 Oct 8 16:14:20 Tower kernel: RBP: ffff880f5dae2900 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:14:20 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:14:20 Tower kernel: R13: ffffffffffffffff R14: ffffc90011c33df0 R15: ffff8807c18dfa40 Oct 8 16:14:20 Tower kernel: FS: 000014864c8f2700(0000) GS:ffff88107ef40000(0000) knlGS:0000000000000000 Oct 8 16:14:20 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 8 16:14:20 Tower kernel: CR2: 00005594a6c01000 CR3: 00000007bddf6000 CR4: 00000000003406e0 Oct 8 16:14:20 Tower kernel: Call Trace: Oct 8 16:14:20 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:14:20 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:14:20 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:14:20 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:14:20 Tower kernel: ? __bpf_prog_run32+0x2e/0x48 Oct 8 16:14:20 Tower kernel: ? do_io_accounting+0x18b/0x1d4 Oct 8 16:14:20 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:14:20 Tower kernel: ? mutex_lock+0xa/0x25 Oct 8 16:14:20 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:14:20 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:14:20 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:14:20 Tower kernel: RIP: 0033:0x14864c1c6820 Oct 8 16:14:20 Tower kernel: Code: 73 01 c3 48 8b 0d 88 77 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d a9 bb 20 00 00 75 10 b8 02 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f6 ff ff 48 89 04 24 Oct 8 16:14:20 Tower kernel: RSP: 002b:00007ffc45b829f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002 Oct 8 16:14:20 Tower kernel: RAX: ffffffffffffffda RBX: 00005594a68036e0 RCX: 000014864c1c6820 Oct 8 16:14:20 Tower kernel: RDX: 00000000000001b6 RSI: 0000000000020000 RDI: 00005594a6c01920 Oct 8 16:14:20 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffd0 Oct 8 16:14:20 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005594a6c01920 Oct 8 16:14:20 Tower kernel: R13: 00005594a6804d0c R14: 000000000000001e R15: 00005594a6c012f0 Oct 8 16:15:50 Tower kernel: INFO: rcu_sched self-detected stall on CPU Oct 8 16:15:50 Tower kernel: 25-....: (240002 ticks this GP) idle=c3a/1/4611686018427387906 softirq=36453/36462 fqs=59232 Oct 8 16:15:50 Tower kernel: (t=240003 jiffies g=24287 c=24286 q=1449740) Oct 8 16:15:50 Tower kernel: NMI backtrace for cpu 25 Oct 8 16:15:50 Tower kernel: CPU: 25 PID: 18182 Comm: apps.plugin Not tainted 4.18.10-unRAID #2 Oct 8 16:15:50 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:15:50 Tower kernel: Call Trace: Oct 8 16:15:50 Tower kernel: <IRQ> Oct 8 16:15:50 Tower kernel: dump_stack+0x5d/0x79 Oct 8 16:15:50 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Oct 8 16:15:50 Tower kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Oct 8 16:15:50 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7 Oct 8 16:15:50 Tower kernel: rcu_dump_cpu_stacks+0x91/0xbb Oct 8 16:15:50 Tower kernel: rcu_check_callbacks+0x23f/0x5ca Oct 8 16:15:50 Tower kernel: ? tick_sched_handle.isra.5+0x2f/0x2f Oct 8 16:15:50 Tower kernel: update_process_times+0x23/0x45 Oct 8 16:15:50 Tower kernel: tick_sched_timer+0x36/0x64 Oct 8 16:15:50 Tower kernel: __hrtimer_run_queues+0xb1/0x105 Oct 8 16:15:50 Tower kernel: hrtimer_interrupt+0xf4/0x20d Oct 8 16:15:50 Tower kernel: smp_apic_timer_interrupt+0x79/0x89 Oct 8 16:15:50 Tower kernel: apic_timer_interrupt+0xf/0x20 Oct 8 16:15:50 Tower kernel: </IRQ> Oct 8 16:15:50 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x165/0x16d Oct 8 16:15:50 Tower kernel: Code: eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d 1a de f9 7e c3 85 d2 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 c8 61 b9 40 00 00 Oct 8 16:15:50 Tower kernel: RSP: 0018:ffffc90011c33c70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 Oct 8 16:15:50 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000102 Oct 8 16:15:50 Tower kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8807c1aff858 Oct 8 16:15:50 Tower kernel: RBP: ffff880f5dae2900 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:15:50 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:15:50 Tower kernel: R13: ffffffffffffffff R14: ffffc90011c33df0 R15: ffff8807c18dfa40 Oct 8 16:15:50 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:15:50 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:15:50 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:15:50 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:15:50 Tower kernel: ? __bpf_prog_run32+0x2e/0x48 Oct 8 16:15:50 Tower kernel: ? do_io_accounting+0x18b/0x1d4 Oct 8 16:15:50 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:15:50 Tower kernel: ? mutex_lock+0xa/0x25 Oct 8 16:15:50 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:15:50 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:15:50 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:15:50 Tower kernel: RIP: 0033:0x14864c1c6820 Oct 8 16:15:50 Tower kernel: Code: 73 01 c3 48 8b 0d 88 77 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d a9 bb 20 00 00 75 10 b8 02 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f6 ff ff 48 89 04 24 Oct 8 16:15:50 Tower kernel: RSP: 002b:00007ffc45b829f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002 Oct 8 16:15:50 Tower kernel: RAX: ffffffffffffffda RBX: 00005594a68036e0 RCX: 000014864c1c6820 Oct 8 16:15:50 Tower kernel: RDX: 00000000000001b6 RSI: 0000000000020000 RDI: 00005594a6c01920 Oct 8 16:15:50 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffd0 Oct 8 16:15:50 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005594a6c01920 Oct 8 16:15:50 Tower kernel: R13: 00005594a6804d0c R14: 000000000000001e R15: 00005594a6c012f0 Oct 8 16:16:39 Tower nginx: 2018/10/08 16:16:39 [error] 10970#10970: *622 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.2.87, server: , request: "POST /webGui/include/Download.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "0202276e11922ac8ddccb963142bcada38e303c1.unraid.net", referrer: "https://0202276e11922ac8ddccb963142bcada38e303c1.unraid.net/Tools/Diagnostics" Oct 8 16:17:20 Tower kernel: INFO: rcu_bh self-detected stall on CPU Oct 8 16:17:20 Tower kernel: 19-....: (1 GPs behind) idle=0ee/1/4611686018427387906 softirq=0/22976 fqs=59053 Oct 8 16:17:20 Tower kernel: (t=240006 jiffies g=-299 c=-300 q=14) Oct 8 16:17:20 Tower kernel: NMI backtrace for cpu 19 Oct 8 16:17:20 Tower kernel: CPU: 19 PID: 38544 Comm: pgrep Not tainted 4.18.10-unRAID #2 Oct 8 16:17:20 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:17:20 Tower kernel: Call Trace: Oct 8 16:17:20 Tower kernel: <IRQ> Oct 8 16:17:20 Tower kernel: dump_stack+0x5d/0x79 Oct 8 16:17:20 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Oct 8 16:17:20 Tower kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Oct 8 16:17:20 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7 Oct 8 16:17:20 Tower kernel: rcu_dump_cpu_stacks+0x91/0xbb Oct 8 16:17:20 Tower kernel: rcu_check_callbacks+0x23f/0x5ca Oct 8 16:17:20 Tower kernel: ? tick_sched_handle.isra.5+0x2f/0x2f Oct 8 16:17:20 Tower kernel: update_process_times+0x23/0x45 Oct 8 16:17:20 Tower kernel: tick_sched_timer+0x36/0x64 Oct 8 16:17:20 Tower kernel: __hrtimer_run_queues+0xb1/0x105 Oct 8 16:17:20 Tower kernel: hrtimer_interrupt+0xf4/0x20d Oct 8 16:17:20 Tower kernel: smp_apic_timer_interrupt+0x79/0x89 Oct 8 16:17:20 Tower kernel: apic_timer_interrupt+0xf/0x20 Oct 8 16:17:20 Tower kernel: </IRQ> Oct 8 16:17:20 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x115/0x16d Oct 8 16:17:20 Tower kernel: Code: c0 17 02 00 48 03 04 cd 00 17 da 81 48 89 10 8b 42 08 85 c0 75 04 f3 90 eb f5 48 8b 0a 48 85 c9 74 c9 0f 0d 09 8b 07 66 85 c0 <74> 04 f3 90 eb f5 41 89 c0 66 45 31 c0 44 39 c6 74 0a 48 85 c9 c6 Oct 8 16:17:20 Tower kernel: RSP: 0018:ffffc900174f3c70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 Oct 8 16:17:20 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000000 Oct 8 16:17:20 Tower kernel: RDX: ffff88085f1e17c0 RSI: 0000000000500000 RDI: ffff8807c1aff858 Oct 8 16:17:20 Tower kernel: RBP: ffff880667174c00 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:17:20 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:17:20 Tower kernel: R13: ffffffffffffffff R14: ffffc900174f3df0 R15: ffff8807c18dfa40 Oct 8 16:17:20 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:17:20 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:17:20 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:17:20 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:17:20 Tower kernel: ? filename_lookup.part.16+0xa5/0xcc Oct 8 16:17:20 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:17:20 Tower kernel: ? _copy_to_user+0x22/0x28 Oct 8 16:17:20 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:17:20 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:17:20 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:17:20 Tower kernel: RIP: 0033:0x14f0a712bff0 Oct 8 16:17:20 Tower kernel: Code: 25 00 00 41 00 3d 00 00 41 00 74 36 48 8d 05 7f d9 2d 00 8b 00 85 c0 75 5a 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 84 00 00 00 48 83 c4 68 5b 5d c3 0f 1f 44 Oct 8 16:17:20 Tower kernel: RSP: 002b:00007ffc98266830 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 Oct 8 16:17:20 Tower kernel: RAX: ffffffffffffffda RBX: 00007ffc982668c0 RCX: 000014f0a712bff0 Oct 8 16:17:20 Tower kernel: RDX: 0000000000000000 RSI: 00007ffc982668c0 RDI: 00000000ffffff9c Oct 8 16:17:20 Tower kernel: RBP: 00007ffc98266ae0 R08: 000014f0a761b946 R09: 0000000000000000 Oct 8 16:17:20 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014f0a7820950 Oct 8 16:17:20 Tower kernel: R13: 00000000006074e0 R14: 0000000000000020 R15: 0000000000000001 Oct 8 16:17:20 Tower kernel: Sending NMI from CPU 19 to CPUs 25: Oct 8 16:17:20 Tower kernel: NMI backtrace for cpu 25 Oct 8 16:17:20 Tower kernel: CPU: 25 PID: 18182 Comm: apps.plugin Not tainted 4.18.10-unRAID #2 Oct 8 16:17:20 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:17:20 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x15f/0x16d Oct 8 16:17:20 Tower kernel: Code: 00 f0 0f b1 37 75 eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d 1a de f9 7e c3 85 d2 74 0a 8b 07 <84> c0 74 04 f3 90 eb f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 Oct 8 16:17:20 Tower kernel: RSP: 0018:ffffc90011c33c70 EFLAGS: 00000202 Oct 8 16:17:20 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000102 Oct 8 16:17:20 Tower kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8807c1aff858 Oct 8 16:17:20 Tower kernel: RBP: ffff880f5dae2900 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:17:20 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:17:20 Tower kernel: R13: ffffffffffffffff R14: ffffc90011c33df0 R15: ffff8807c18dfa40 Oct 8 16:17:20 Tower kernel: FS: 000014864c8f2700(0000) GS:ffff88107ef40000(0000) knlGS:0000000000000000 Oct 8 16:17:20 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 8 16:17:20 Tower kernel: CR2: 00005594a6c01000 CR3: 00000007bddf6000 CR4: 00000000003406e0 Oct 8 16:17:20 Tower kernel: Call Trace: Oct 8 16:17:20 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:17:20 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:17:20 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:17:20 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:17:20 Tower kernel: ? __bpf_prog_run32+0x2e/0x48 Oct 8 16:17:20 Tower kernel: ? do_io_accounting+0x18b/0x1d4 Oct 8 16:17:20 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:17:20 Tower kernel: ? mutex_lock+0xa/0x25 Oct 8 16:17:20 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:17:20 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:17:20 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:17:20 Tower kernel: RIP: 0033:0x14864c1c6820 Oct 8 16:17:20 Tower kernel: Code: 73 01 c3 48 8b 0d 88 77 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d a9 bb 20 00 00 75 10 b8 02 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f6 ff ff 48 89 04 24 Oct 8 16:17:20 Tower kernel: RSP: 002b:00007ffc45b829f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002 Oct 8 16:17:20 Tower kernel: RAX: ffffffffffffffda RBX: 00005594a68036e0 RCX: 000014864c1c6820 Oct 8 16:17:20 Tower kernel: RDX: 00000000000001b6 RSI: 0000000000020000 RDI: 00005594a6c01920 Oct 8 16:17:20 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffd0 Oct 8 16:17:20 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005594a6c01920 Oct 8 16:17:20 Tower kernel: R13: 00005594a6804d0c R14: 000000000000001e R15: 00005594a6c012f0 Oct 8 16:17:20 Tower kernel: INFO: rcu_bh detected stalls on CPUs/tasks: Oct 8 16:17:20 Tower kernel: 19-....: (1 GPs behind) idle=0ee/1/4611686018427387906 softirq=0/22976 fqs=59054 Oct 8 16:17:20 Tower kernel: 25-....: (1 GPs behind) idle=c3a/1/4611686018427387904 softirq=0/36462 fqs=59054 Oct 8 16:17:20 Tower kernel: (detected by 27, t=240007 jiffies, g=-299, c=-300, q=14) Oct 8 16:17:20 Tower kernel: Sending NMI from CPU 27 to CPUs 19: Oct 8 16:17:20 Tower kernel: NMI backtrace for cpu 19 Oct 8 16:17:20 Tower kernel: CPU: 19 PID: 38544 Comm: pgrep Not tainted 4.18.10-unRAID #2 Oct 8 16:17:20 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:17:20 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x119/0x16d Oct 8 16:17:20 Tower kernel: Code: 48 03 04 cd 00 17 da 81 48 89 10 8b 42 08 85 c0 75 04 f3 90 eb f5 48 8b 0a 48 85 c9 74 c9 0f 0d 09 8b 07 66 85 c0 74 04 f3 90 <eb> f5 41 89 c0 66 45 31 c0 44 39 c6 74 0a 48 85 c9 c6 07 01 75 1b Oct 8 16:17:20 Tower kernel: RSP: 0018:ffffc900174f3c70 EFLAGS: 00000202 Oct 8 16:17:20 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000000 Oct 8 16:17:20 Tower kernel: RDX: ffff88085f1e17c0 RSI: 0000000000500000 RDI: ffff8807c1aff858 Oct 8 16:17:20 Tower kernel: RBP: ffff880667174c00 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:17:20 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:17:20 Tower kernel: R13: ffffffffffffffff R14: ffffc900174f3df0 R15: ffff8807c18dfa40 Oct 8 16:17:20 Tower kernel: FS: 000014f0a7a48740(0000) GS:ffff88085f1c0000(0000) knlGS:0000000000000000 Oct 8 16:17:20 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 8 16:17:20 Tower kernel: CR2: 0000000000617198 CR3: 0000000667124000 CR4: 00000000003406e0 Oct 8 16:17:20 Tower kernel: Call Trace: Oct 8 16:17:20 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:17:20 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:17:20 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:17:20 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:17:20 Tower kernel: ? filename_lookup.part.16+0xa5/0xcc Oct 8 16:17:20 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:17:20 Tower kernel: ? _copy_to_user+0x22/0x28 Oct 8 16:17:20 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:17:20 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:17:20 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:17:20 Tower kernel: RIP: 0033:0x14f0a712bff0 Oct 8 16:17:20 Tower kernel: Code: 25 00 00 41 00 3d 00 00 41 00 74 36 48 8d 05 7f d9 2d 00 8b 00 85 c0 75 5a 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 84 00 00 00 48 83 c4 68 5b 5d c3 0f 1f 44 Oct 8 16:17:20 Tower kernel: RSP: 002b:00007ffc98266830 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 Oct 8 16:17:20 Tower kernel: RAX: ffffffffffffffda RBX: 00007ffc982668c0 RCX: 000014f0a712bff0 Oct 8 16:17:20 Tower kernel: RDX: 0000000000000000 RSI: 00007ffc982668c0 RDI: 00000000ffffff9c Oct 8 16:17:20 Tower kernel: RBP: 00007ffc98266ae0 R08: 000014f0a761b946 R09: 0000000000000000 Oct 8 16:17:20 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000014f0a7820950 Oct 8 16:17:20 Tower kernel: R13: 00000000006074e0 R14: 0000000000000020 R15: 0000000000000001 Oct 8 16:17:20 Tower kernel: Sending NMI from CPU 27 to CPUs 25: Oct 8 16:17:20 Tower kernel: NMI backtrace for cpu 25 Oct 8 16:17:20 Tower kernel: CPU: 25 PID: 18182 Comm: apps.plugin Not tainted 4.18.10-unRAID #2 Oct 8 16:17:20 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:17:20 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x165/0x16d Oct 8 16:17:20 Tower kernel: Code: eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d 1a de f9 7e c3 85 d2 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 c8 61 b9 40 00 00 Oct 8 16:17:20 Tower kernel: RSP: 0018:ffffc90011c33c70 EFLAGS: 00000202 Oct 8 16:17:20 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000102 Oct 8 16:17:20 Tower kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8807c1aff858 Oct 8 16:17:20 Tower kernel: RBP: ffff880f5dae2900 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:17:20 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:17:20 Tower kernel: R13: ffffffffffffffff R14: ffffc90011c33df0 R15: ffff8807c18dfa40 Oct 8 16:17:20 Tower kernel: FS: 000014864c8f2700(0000) GS:ffff88107ef40000(0000) knlGS:0000000000000000 Oct 8 16:17:20 Tower kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 8 16:17:20 Tower kernel: CR2: 00005594a6c01000 CR3: 00000007bddf6000 CR4: 00000000003406e0 Oct 8 16:17:20 Tower kernel: Call Trace: Oct 8 16:17:20 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:17:20 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:17:20 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:17:20 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:17:20 Tower kernel: ? __bpf_prog_run32+0x2e/0x48 Oct 8 16:17:20 Tower kernel: ? do_io_accounting+0x18b/0x1d4 Oct 8 16:17:20 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:17:20 Tower kernel: ? mutex_lock+0xa/0x25 Oct 8 16:17:20 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:17:20 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:17:20 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:17:20 Tower kernel: RIP: 0033:0x14864c1c6820 Oct 8 16:17:20 Tower kernel: Code: 73 01 c3 48 8b 0d 88 77 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d a9 bb 20 00 00 75 10 b8 02 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f6 ff ff 48 89 04 24 Oct 8 16:17:20 Tower kernel: RSP: 002b:00007ffc45b829f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002 Oct 8 16:17:20 Tower kernel: RAX: ffffffffffffffda RBX: 00005594a68036e0 RCX: 000014864c1c6820 Oct 8 16:17:20 Tower kernel: RDX: 00000000000001b6 RSI: 0000000000020000 RDI: 00005594a6c01920 Oct 8 16:17:20 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffd0 Oct 8 16:17:20 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005594a6c01920 Oct 8 16:17:20 Tower kernel: R13: 00005594a6804d0c R14: 000000000000001e R15: 00005594a6c012f0 Oct 8 16:18:50 Tower kernel: INFO: rcu_sched self-detected stall on CPU Oct 8 16:18:50 Tower kernel: 25-....: (420005 ticks this GP) idle=c3a/1/4611686018427387906 softirq=36453/36462 fqs=103463 Oct 8 16:18:50 Tower kernel: (t=420006 jiffies g=24287 c=24286 q=2269190) Oct 8 16:18:50 Tower kernel: NMI backtrace for cpu 25 Oct 8 16:18:50 Tower kernel: CPU: 25 PID: 18182 Comm: apps.plugin Not tainted 4.18.10-unRAID #2 Oct 8 16:18:50 Tower kernel: Hardware name: System manufacturer System Product Name/ROG ZENITH EXTREME, BIOS 1402 08/03/2018 Oct 8 16:18:50 Tower kernel: Call Trace: Oct 8 16:18:50 Tower kernel: <IRQ> Oct 8 16:18:50 Tower kernel: dump_stack+0x5d/0x79 Oct 8 16:18:50 Tower kernel: nmi_cpu_backtrace+0x71/0x83 Oct 8 16:18:50 Tower kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Oct 8 16:18:50 Tower kernel: nmi_trigger_cpumask_backtrace+0x57/0xd7 Oct 8 16:18:50 Tower kernel: rcu_dump_cpu_stacks+0x91/0xbb Oct 8 16:18:50 Tower kernel: rcu_check_callbacks+0x23f/0x5ca Oct 8 16:18:50 Tower kernel: ? tick_sched_handle.isra.5+0x2f/0x2f Oct 8 16:18:50 Tower kernel: update_process_times+0x23/0x45 Oct 8 16:18:50 Tower kernel: tick_sched_timer+0x36/0x64 Oct 8 16:18:50 Tower kernel: __hrtimer_run_queues+0xb1/0x105 Oct 8 16:18:50 Tower kernel: hrtimer_interrupt+0xf4/0x20d Oct 8 16:18:50 Tower kernel: smp_apic_timer_interrupt+0x79/0x89 Oct 8 16:18:50 Tower kernel: apic_timer_interrupt+0xf/0x20 Oct 8 16:18:50 Tower kernel: </IRQ> Oct 8 16:18:50 Tower kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x161/0x16d Oct 8 16:18:50 Tower kernel: Code: 0f b1 37 75 eb eb 13 48 8b 0a 48 85 c9 75 04 f3 90 eb f4 c7 41 08 01 00 00 00 65 ff 0d 1a de f9 7e c3 85 d2 74 0a 8b 07 84 c0 <74> 04 f3 90 eb f6 66 c7 07 01 00 c3 49 b8 eb 83 b5 80 46 86 c8 61 Oct 8 16:18:50 Tower kernel: RSP: 0018:ffffc90011c33c70 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 Oct 8 16:18:50 Tower kernel: RAX: 00000000007c0102 RBX: ffff8807c1aff808 RCX: 0000000000000102 Oct 8 16:18:50 Tower kernel: RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8807c1aff858 Oct 8 16:18:50 Tower kernel: RBP: ffff880f5dae2900 R08: 61c8864680b583eb R09: ffff8807c18dfa40 Oct 8 16:18:50 Tower kernel: R10: 0000000000000000 R11: 8080808080808080 R12: 00000000661e377c Oct 8 16:18:50 Tower kernel: R13: ffffffffffffffff R14: ffffc90011c33df0 R15: ffff8807c18dfa40 Oct 8 16:18:50 Tower kernel: _raw_spin_lock+0x16/0x19 Oct 8 16:18:50 Tower kernel: __d_lookup+0x58/0x12b Oct 8 16:18:50 Tower kernel: lookup_fast+0x1d2/0x27a Oct 8 16:18:50 Tower kernel: path_openat+0x2b6/0xc07 Oct 8 16:18:50 Tower kernel: ? __bpf_prog_run32+0x2e/0x48 Oct 8 16:18:50 Tower kernel: ? do_io_accounting+0x18b/0x1d4 Oct 8 16:18:50 Tower kernel: do_filp_open+0x4c/0xa9 Oct 8 16:18:50 Tower kernel: ? mutex_lock+0xa/0x25 Oct 8 16:18:50 Tower kernel: do_sys_open+0x132/0x1ce Oct 8 16:18:50 Tower kernel: do_syscall_64+0x57/0xe6 Oct 8 16:18:50 Tower kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Oct 8 16:18:50 Tower kernel: RIP: 0033:0x14864c1c6820 Oct 8 16:18:50 Tower kernel: Code: 73 01 c3 48 8b 0d 88 77 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d a9 bb 20 00 00 75 10 b8 02 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f6 ff ff 48 89 04 24 Oct 8 16:18:50 Tower kernel: RSP: 002b:00007ffc45b829f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002 Oct 8 16:18:50 Tower kernel: RAX: ffffffffffffffda RBX: 00005594a68036e0 RCX: 000014864c1c6820 Oct 8 16:18:50 Tower kernel: RDX: 00000000000001b6 RSI: 0000000000020000 RDI: 00005594a6c01920 Oct 8 16:18:50 Tower kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000ffffffd0 Oct 8 16:18:50 Tower kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00005594a6c01920 Oct 8 16:18:50 Tower kernel: R13: 00005594a6804d0c R14: 000000000000001e R15: 00005594a6c012f0 tower-diagnostics-20181008-1614.zip
-
I don't know if this is and AMD x399 Zenith board bug or Unraid so posting this to to verify. Was using my machine and got this below. I force stopped the VM and it killed unraid completely. 100% unresponsive so had to hard reset. This isn't the exact one it got but its very similar. It was the only thing logged in the syslog before it occured. Sep 26 15:08:57 Tower kernel: iommu ivhd2: AMD-Vi: Event logged [ Sep 26 15:08:57 Tower kernel: iommu ivhd2: INVALID_DEVICE_REQUEST device=00:00.0 pasid=0x00000 address=0xfffffffdf8000000 flags=0x0a00] tower-syslog-20180926-1518.zip
-
Didn't think it would route through it unless you called it but also assumed everytime I update the docker it would wipe the module and I would have to redo it again.
-
are you using the proxy-conf example or did you create your own?
-
@linuxserver.io any chance you could do this?
