Gammawolf

I'm having a few issues and not sure where else to look.. i've had it working fine and recently have gone through an ISP and router/firewall change. My current situation is i cannot get the VPN to work on my city fibre based connection, over LTE it is working fine. I have both connections configured in sdwan and simply enabling a rule that steers traffic originating from my unraid server to the VPN servers IP over LTE, makes the VPN and client download correctly. Disabling this rule on my firewall so that it takes the default connection (city fibre) results in the docker starting up but items are forever stalled. No changes made to the docker settings between working and not working besides enabling/disabling the rule. The tunnels seem both up and connected as far as i can see through the interfaces although when its not working i get alot of TX dropped. Attached the log when its not working on city fibre. Any help or pointers greatly apprieciated. When running on LTE and working normally sh-5.1# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.17.0.3 netmask 255.255.0.0 broadcast 172.17.255.255 ether 02:42:ac:11:00:03 txqueuelen 0 (Ethernet) RX packets 22264 bytes 3781648 (3.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 22869 bytes 27389758 (26.1 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 loop txqueuelen 1000 (Local Loopback) RX packets 1004 bytes 145782 (142.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1004 bytes 145782 (142.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1500 inet 10.7.2.5 netmask 255.255.255.0 destination 10.7.2.5 unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC) RX packets 21061 bytes 1730496 (1.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 39657 bytes 24770884 (23.6 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 When running on city fibre and not working sh-5.1# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.17.0.3 netmask 255.255.0.0 broadcast 172.17.255.255 ether 02:42:ac:11:00:03 txqueuelen 0 (Ethernet) RX packets 926 bytes 270200 (263.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 871 bytes 127203 (124.2 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 loop txqueuelen 1000 (Local Loopback) RX packets 9 bytes 1235 (1.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 9 bytes 1235 (1.2 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1500 inet 10.7.2.5 netmask 255.255.255.0 destination 10.7.2.5 unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC) RX packets 723 bytes 161632 (157.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 980 bytes 73342 (71.6 KiB) TX errors 0 dropped 2089 overruns 0 carrier 0 collisions 0 cityfibre log.txt

Anyone have any ideas on this? Its happening every 2 to 3 days and i've wiped the flash drive and started over since this post. I didn't import any old configs (started from scratch) just attached the array and re-registered my two dockers (plex and pihole).

I've been running unraid for a while, nearly 2 years on this hardware - not had many problems expect a usb drive go on me. Since about 2 weeks ago unraid has been crashing on me every few days. Everything becomes unresponsive, no web gui, no ssh, all services/vms/dockers dead. I've been on the latest version 6.8.3 since a few days after release. Immediately prior to crashing the cpu load just keeps steadily increasing (it just starts to max each core until you can't access anything). I've tried to look at the syslogs but it doesn't mean much to me... i've attached the tail end when the issue started happening. It repeats that variation of error until it locks up in roughly 45 minutes. Jul 19 04:00:01 SKJ-UNRAID Docker Auto Update: Community Applications Docker Autoupdate running Jul 19 04:00:01 SKJ-UNRAID Docker Auto Update: Checking for available updates Jul 19 04:00:07 SKJ-UNRAID Docker Auto Update: Installing Updates for netdata Jul 19 04:00:17 SKJ-UNRAID Docker Auto Update: Community Applications Docker Autoupdate finished Jul 19 08:27:04 SKJ-UNRAID kernel: WARNING: CPU: 8 PID: 1819 at kernel/workqueue.c:952 worker_thread+0x6a/0x2b8 Jul 19 08:27:04 SKJ-UNRAID kernel: Modules linked in: veth xt_CHECKSUM ipt_REJECT ip6table_mangle ip6table_nat nf_nat_ipv6 iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost tap xt_nat macvlan ipt_MASQUERADE iptable_filter iptable_nat nf_nat_ipv4 nf_nat ip_tables xfs md_mod bonding edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd i2c_piix4 cryptd i2c_core mxm_wmi wmi_bmof igb(O) ccp nvme ahci glue_helper pcc_cpufreq nvme_core libahci wmi button acpi_cpufreq Jul 19 08:27:04 SKJ-UNRAID kernel: CPU: 8 PID: 1819 Comm: kworker/8:2 Tainted: G O 4.19.107-Unraid #1 Jul 19 08:27:04 SKJ-UNRAID kernel: Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5220 09/12/2019 Jul 19 08:27:04 SKJ-UNRAID kernel: Workqueue: (null) (events) Jul 19 08:27:04 SKJ-UNRAID kernel: RIP: 0010:worker_thread+0x6a/0x2b8 Jul 19 08:27:04 SKJ-UNRAID kernel: Code: 1f 44 00 00 48 8b 45 00 48 39 c5 0f 84 21 02 00 00 0f 0b e9 1a 02 00 00 a8 04 48 8b 55 40 75 04 0f 0b eb 28 48 39 5d 38 74 02 <0f> 0b 83 e0 fb 89 45 60 ff 4a 34 48 8b 55 00 48 8b 45 08 48 89 42 Jul 19 08:27:04 SKJ-UNRAID kernel: RSP: 0018:ffffc900084bfee0 EFLAGS: 00010086 Jul 19 08:27:04 SKJ-UNRAID kernel: RAX: 000000000000000c RBX: 0000000000000000 RCX: 0000000000000000 Jul 19 08:27:04 SKJ-UNRAID kernel: RDX: ffff88881e81f440 RSI: ffff88850d159b00 RDI: ffff88881e81f440 Jul 19 08:27:04 SKJ-UNRAID kernel: RBP: ffff888288ce1740 R08: 0000000000000000 R09: 0000000000000000 Jul 19 08:27:04 SKJ-UNRAID kernel: R10: 000000000024e2d4 R11: ffff88881e81fb40 R12: ffff88881e81f440 Jul 19 08:27:04 SKJ-UNRAID kernel: R13: ffff88881e81f440 R14: ffff88881e81f460 R15: ffff888288ce1768 Jul 19 08:27:04 SKJ-UNRAID kernel: FS: 0000000000000000(0000) GS:ffff88881e800000(0000) knlGS:0000000000000000 Jul 19 08:27:04 SKJ-UNRAID kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 19 08:27:04 SKJ-UNRAID kernel: CR2: 00000000000000b0 CR3: 00000007e632c000 CR4: 0000000000340ee0 Jul 19 08:27:04 SKJ-UNRAID kernel: Call Trace: Jul 19 08:27:04 SKJ-UNRAID kernel: ? rescuer_thread+0x2a7/0x2a7 Jul 19 08:27:04 SKJ-UNRAID kernel: kthread+0x10c/0x114 Jul 19 08:27:04 SKJ-UNRAID kernel: ? kthread_park+0x89/0x89 Jul 19 08:27:04 SKJ-UNRAID kernel: ret_from_fork+0x22/0x40 Jul 19 08:27:04 SKJ-UNRAID kernel: ---[ end trace 27415ddf8e1b6616 ]--- Jul 19 08:59:06 SKJ-UNRAID kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: Jul 19 08:59:06 SKJ-UNRAID kernel: rcu: 0-...0: (1 GPs behind) idle=646/0/0x1 softirq=38974706/38974707 fqs=13984 Jul 19 08:59:06 SKJ-UNRAID kernel: rcu: (detected by 3, t=60002 jiffies, g=39217977, q=8516) Jul 19 08:59:06 SKJ-UNRAID kernel: Sending NMI from CPU 3 to CPUs 0: Jul 19 08:59:06 SKJ-UNRAID kernel: NMI backtrace for cpu 0 Jul 19 08:59:06 SKJ-UNRAID kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O 4.19.107-Unraid #1 Jul 19 08:59:06 SKJ-UNRAID kernel: Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5220 09/12/2019 Jul 19 08:59:06 SKJ-UNRAID kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x6b/0x171 Jul 19 08:59:06 SKJ-UNRAID kernel: Code: 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 74 0e 81 e6 00 ff 00 00 75 1a c6 47 01 00 eb 14 85 f6 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 48 c7 c2 40 07 02 00 65 48 03 15 80 6a f8 Jul 19 08:59:06 SKJ-UNRAID kernel: RSP: 0018:ffff88881e603f70 EFLAGS: 00000002 Jul 19 08:59:06 SKJ-UNRAID kernel: RAX: 0000000000000101 RBX: 7fffffffffffffff RCX: 0000000000000000 Jul 19 08:59:06 SKJ-UNRAID kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88881e61cac0 Jul 19 08:59:06 SKJ-UNRAID kernel: RBP: ffff88881e615f40 R08: 0000000000000001 R09: 0000000000000000 Jul 19 08:59:06 SKJ-UNRAID kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000006 Jul 19 08:59:06 SKJ-UNRAID kernel: R13: ffff88881e61cac0 R14: ffff88881e61cbb8 R15: ffff88881e61cbf8 Jul 19 08:59:06 SKJ-UNRAID kernel: FS: 0000000000000000(0000) GS:ffff88881e600000(0000) knlGS:0000000000000000 Jul 19 08:59:06 SKJ-UNRAID kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 19 08:59:06 SKJ-UNRAID kernel: CR2: 00007f6287df5bc8 CR3: 00000007b076a000 CR4: 0000000000340ef0 Jul 19 08:59:06 SKJ-UNRAID kernel: Call Trace: Jul 19 08:59:06 SKJ-UNRAID kernel: <IRQ> Jul 19 08:59:06 SKJ-UNRAID kernel: _raw_spin_lock_irqsave+0x29/0x31 Jul 19 08:59:06 SKJ-UNRAID kernel: hrtimer_interrupt+0x4e/0x20e Jul 19 08:59:06 SKJ-UNRAID kernel: ? update_ts_time_stats+0x21/0x57 Jul 19 08:59:06 SKJ-UNRAID kernel: smp_apic_timer_interrupt+0x7b/0x93 Jul 19 08:59:06 SKJ-UNRAID kernel: apic_timer_interrupt+0xf/0x20 Jul 19 08:59:06 SKJ-UNRAID kernel: </IRQ> Jul 19 08:59:06 SKJ-UNRAID kernel: RIP: 0010:cpuidle_enter_state+0xe8/0x141 Jul 19 08:59:06 SKJ-UNRAID kernel: Code: ff 45 84 f6 74 1d 9c 58 0f 1f 44 00 00 0f ba e0 09 73 09 0f 0b fa 66 0f 1f 44 00 00 31 ff e8 7a 8d bb ff fb 66 0f 1f 44 00 00 <48> 2b 2c 24 b8 ff ff ff 7f 48 b9 ff ff ff ff f3 01 00 00 48 39 cd Jul 19 08:59:06 SKJ-UNRAID kernel: RSP: 0018:ffffffff81e03e80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 Jul 19 08:59:06 SKJ-UNRAID kernel: RAX: ffff88881e61fac0 RBX: ffff88881692d000 RCX: 000000000000001f Jul 19 08:59:06 SKJ-UNRAID kernel: RDX: 0000000000000000 RSI: 0000000020e1d7a0 RDI: 0000000000000000 Jul 19 08:59:06 SKJ-UNRAID kernel: RBP: 0001380f2dba03a5 R08: 0001380f2dba03a5 R09: 00000000000002ea Jul 19 08:59:06 SKJ-UNRAID kernel: R10: 0000000000282e04 R11: 071c71c71c71c71c R12: 0000000000000001 Jul 19 08:59:06 SKJ-UNRAID kernel: R13: ffffffff81e5e1e0 R14: 0000000000000000 R15: ffffffff81e5e258 Jul 19 08:59:06 SKJ-UNRAID kernel: do_idle+0x17e/0x1fc Jul 19 08:59:06 SKJ-UNRAID kernel: cpu_startup_entry+0x6a/0x6c Jul 19 08:59:06 SKJ-UNRAID kernel: start_kernel+0x44e/0x46c Jul 19 08:59:06 SKJ-UNRAID kernel: secondary_startup_64+0xa4/0xb0 Jul 19 09:02:06 SKJ-UNRAID kernel: rcu: INFO: rcu_sched detected stalls on CPUs/tasks: Jul 19 09:02:06 SKJ-UNRAID kernel: rcu: 0-...0: (1 GPs behind) idle=646/0/0x1 softirq=38974706/38974707 fqs=56150 Jul 19 09:02:06 SKJ-UNRAID kernel: rcu: (detected by 2, t=240007 jiffies, g=39217977, q=34675) Jul 19 09:02:06 SKJ-UNRAID kernel: Sending NMI from CPU 2 to CPUs 0: Jul 19 09:02:06 SKJ-UNRAID kernel: NMI backtrace for cpu 0 Jul 19 09:02:06 SKJ-UNRAID kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W O 4.19.107-Unraid #1 Jul 19 09:02:06 SKJ-UNRAID kernel: Hardware name: System manufacturer System Product Name/PRIME X370-PRO, BIOS 5220 09/12/2019 Jul 19 09:02:06 SKJ-UNRAID kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x6b/0x171 Jul 19 09:02:06 SKJ-UNRAID kernel: Code: 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 74 0e 81 e6 00 ff 00 00 75 1a c6 47 01 00 eb 14 85 f6 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 48 c7 c2 40 07 02 00 65 48 03 15 80 6a f8 Jul 19 09:02:06 SKJ-UNRAID kernel: RSP: 0018:ffff88881e603f70 EFLAGS: 00000002 Jul 19 09:02:06 SKJ-UNRAID kernel: RAX: 0000000000000101 RBX: 7fffffffffffffff RCX: 0000000000000000 Jul 19 09:02:06 SKJ-UNRAID kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88881e61cac0 Jul 19 09:02:06 SKJ-UNRAID kernel: RBP: ffff88881e615f40 R08: 0000000000000001 R09: 0000000000000000 Jul 19 09:02:06 SKJ-UNRAID kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000006 Jul 19 09:02:06 SKJ-UNRAID kernel: R13: ffff88881e61cac0 R14: ffff88881e61cbb8 R15: ffff88881e61cbf8 Jul 19 09:02:06 SKJ-UNRAID kernel: FS: 0000000000000000(0000) GS:ffff88881e600000(0000) knlGS:0000000000000000 Jul 19 09:02:06 SKJ-UNRAID kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 19 09:02:06 SKJ-UNRAID kernel: CR2: 00007f6287df5bc8 CR3: 00000007b076a000 CR4: 0000000000340ef0 Jul 19 09:02:06 SKJ-UNRAID kernel: Call Trace: