gdeyoung Posted January 31, 2021 Author Share Posted January 31, 2021 So a quick update to close this out as fixed. - Ran all three servers with issues in safe mode and had no kernel panics and were stable. - Began trying to troubleshoot down which docker container or plugin was causing the instability - Found out it was the Technitum DNS docker container in community apps. With one wrinkle, on a 1G connection it was stable. On a 10G connection it generated kernel panics and spin lock errors. The stock install was on BR0 so wonder if it was in the dockers config for networking. - Once I removed the Technitum DNS server docker from the three servers they have been completely stable 2 Quote Link to comment
juchong Posted February 18, 2021 Share Posted February 18, 2021 Hey! I'm seeing a similar issue on my server as well. Here's the relevant syslog: Feb 17 19:46:21 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:25 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:29 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:33 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:37 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:41 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:45 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 21:09:50 thecloud vsftpd[24107]: connect from 10.0.10.1 (10.0.10.1) Feb 17 21:09:50 thecloud in.telnetd[24108]: connect from 10.0.10.1 (10.0.10.1) Feb 17 21:09:50 thecloud sshd[24109]: Connection from 10.0.10.1 port 36967 on 10.0.10.4 port 22 rdomain "" Feb 17 21:09:50 thecloud sshd[24109]: error: kex_exchange_identification: Connection closed by remote host Feb 17 21:09:50 thecloud sshd[24109]: Connection closed by 10.0.10.1 port 36967 Feb 17 21:09:50 thecloud vsftpd[24113]: connect from 10.0.10.1 (10.0.10.1) Feb 17 21:09:55 thecloud telnetd[24108]: ttloop: peer died: EOF Feb 17 21:10:01 thecloud smbd[24110]: [2021/02/17 21:10:01.582271, 0] ../../source3/smbd/process.c:341(read_packet_remainder) Feb 17 21:10:01 thecloud smbd[24110]: read_fd_with_timeout failed for client 10.0.10.1 read error = NT_STATUS_END_OF_FILE. Feb 17 21:53:56 thecloud kernel: Linux version 5.10.1-Unraid (root@Develop) (gcc (GCC) 9.3.0, GNU ld version 2.33.1-slack15) #1 SMP Thu Dec 17 11:41:39 PST 2020 Feb 17 21:53:56 thecloud kernel: Command line: BOOT_IMAGE=/bzimage initrd=/bzroot console=ttyS0,115200 console=tty0 Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers' Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR' Feb 17 21:53:56 thecloud kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 Feb 17 21:53:56 thecloud kernel: x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64 Feb 17 21:53:56 thecloud kernel: x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64 Feb 17 21:53:56 thecloud kernel: x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format. Feb 17 21:53:56 thecloud kernel: BIOS-provided physical RAM map: Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ebff] usable Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x000000000009ec00-0x000000000009ffff] reserved Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x0000000040000000-0x00000000403fffff] reserved @gdeyoung do you happen to have a Ubiquiti device acting as your gateway/router? The reason I ask is because my USG Pro was actively trying to log into the server via ssh/telnet when it hung. Quote Link to comment
Hoopie Posted October 13, 2021 Share Posted October 13, 2021 (edited) On 2/17/2021 at 9:33 PM, juchong said: Hey! I'm seeing a similar issue on my server as well. Here's the relevant syslog: Feb 17 19:46:21 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:25 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:29 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:33 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:37 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:41 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 19:46:45 thecloud root: error: /plugins/unassigned.devices/UnassignedDevices.php: wrong csrf_token Feb 17 21:09:50 thecloud vsftpd[24107]: connect from 10.0.10.1 (10.0.10.1) Feb 17 21:09:50 thecloud in.telnetd[24108]: connect from 10.0.10.1 (10.0.10.1) Feb 17 21:09:50 thecloud sshd[24109]: Connection from 10.0.10.1 port 36967 on 10.0.10.4 port 22 rdomain "" Feb 17 21:09:50 thecloud sshd[24109]: error: kex_exchange_identification: Connection closed by remote host Feb 17 21:09:50 thecloud sshd[24109]: Connection closed by 10.0.10.1 port 36967 Feb 17 21:09:50 thecloud vsftpd[24113]: connect from 10.0.10.1 (10.0.10.1) Feb 17 21:09:55 thecloud telnetd[24108]: ttloop: peer died: EOF Feb 17 21:10:01 thecloud smbd[24110]: [2021/02/17 21:10:01.582271, 0] ../../source3/smbd/process.c:341(read_packet_remainder) Feb 17 21:10:01 thecloud smbd[24110]: read_fd_with_timeout failed for client 10.0.10.1 read error = NT_STATUS_END_OF_FILE. Feb 17 21:53:56 thecloud kernel: Linux version 5.10.1-Unraid (root@Develop) (gcc (GCC) 9.3.0, GNU ld version 2.33.1-slack15) #1 SMP Thu Dec 17 11:41:39 PST 2020 Feb 17 21:53:56 thecloud kernel: Command line: BOOT_IMAGE=/bzimage initrd=/bzroot console=ttyS0,115200 console=tty0 Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers' Feb 17 21:53:56 thecloud kernel: x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR' Feb 17 21:53:56 thecloud kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 Feb 17 21:53:56 thecloud kernel: x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64 Feb 17 21:53:56 thecloud kernel: x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64 Feb 17 21:53:56 thecloud kernel: x86/fpu: Enabled xstate features 0x1f, context size is 960 bytes, using 'compacted' format. Feb 17 21:53:56 thecloud kernel: BIOS-provided physical RAM map: Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ebff] usable Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x000000000009ec00-0x000000000009ffff] reserved Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable Feb 17 21:53:56 thecloud kernel: BIOS-e820: [mem 0x0000000040000000-0x00000000403fffff] reserved @gdeyoung do you happen to have a Ubiquiti device acting as your gateway/router? The reason I ask is because my USG Pro was actively trying to log into the server via ssh/telnet when it hung. Did you ever get your issue solved? I have a UDM pro that's also trying to log in via SSH. It always appears to be the last thing logged before crashing. Edited October 13, 2021 by Hoopie Quote Link to comment
gdeyoung Posted October 14, 2021 Author Share Posted October 14, 2021 It turned out not to be the UDM. It was the DNS Docker causing panics on the network stack that was my root issue. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.