gdeyoung

Members
  • Posts

    30
  • Joined

  • Last visited

Everything posted by gdeyoung

  1. So a quick update to close this out as fixed. - Ran all three servers with issues in safe mode and had no kernel panics and were stable. - Began trying to troubleshoot down which docker container or plugin was causing the instability - Found out it was the Technitum DNS docker container in community apps. With one wrinkle, on a 1G connection it was stable. On a 10G connection it generated kernel panics and spin lock errors. The stock install was on BR0 so wonder if it was in the dockers config for networking. - Once I removed the Technitum DNS server docker from the three servers they have been completely stable
  2. The 3rd server is in safe mode and still going I switched the 2nd server back to 10G in normal mode with no file copies and it panic'd in 20 minutes. diag's attached mediaserver-diagnostics-20210122-1520.zip
  3. Ok, I put one of the servers in safe mode on 10G and doing some file copies.
  4. They have the default 1500 MTU on the 10G nic. Should I be using 9000 for the jumbo frames?
  5. Yes, the Intel NIC seem to be more stable. I was also having issues with some of my Windows PC with Aquantia 10G and transfers, so I switched to Intel 10G across the board. Yes, rolled back to 6.8.3 and had the same issues for both nics. My other observation is the panics are happening on the ingest servers where I copy files to more often.
  6. @JorgeB Thank for continuing to engage, I really appreciate it. I have completed troubleshooting to try and localize down the issues. I have completely rebuilt two of the four servers with new components the only remaining thing is the Drives and still get the panic issues. I have swapped out all of the all of the network gear, three different 10G switches, new cables. I have removed all external items or replaced several times with new and still get the panics. In the last couple of days I switched two of the servers back to 1G and they are rock solid with no issues and are not chatty in the logs. Where with the 10G I was getting a variety of things pop up in logs every hour. This is not my first post on this. In my previous post I did post full diags and got NO replies. I DM'd @limetech for help and still silence. So I am trying, I really would like to get this working. All I have are the panic traces to go on now and don't have the knowledge to trouble shoot at that level. Here are the two types of NIC's I have used that are supposed to be fully supported. TRENDnet - TRENDnet TEG-10GECSFP - SFP+ Aquantia chipset Supermicro AOC-STGF-i2S - Dual SFP+ Intel chipset
  7. So 2 days ago I switched the my 2nd server from 10g to 1G. 1 day ago I switched my 3rd server to 1G from 10G. These are all different hardware machines Intel & Ryzen running a combo of 6.8.3 and 6.9rc2. All of my servers on 10G (all on their swapped out/2nd 10G NIC) kernel panics under heavy/sustained file copy within 24hrs. Without heavy file load they will panic under 72hrs. I have reworked network and simplified network configs. I have up to date bios on mobo's. It all comes down to sustained load on the 10G Intel and Aquantia nics. I have even three 3 different 10G switches, new 10G DAC cables, 10000base-T transcievers with Cat-7. It all comes back to there is something in the kernel that isn't right with heavy 10G network loads and causes panics. One thing I'm seeing is native_queued_spin_lock_slowpath errors before the full panic but I'm not seeing high CPU loads. Found these two articles/posts that might have some relevance. High CPU load by native_queued_spin_lock_slowpath (linuxquestions.org) The need for speed and the kernel datapath - recent improvements in UDP packets processing - Red Hat Developer What can be done to get 10G working in a stable fashion with sustained file copy loads? The whole reason for 10G... @limetech @JorgeB
  8. Server 3 just panic'd again. again this is a 10G server. also on it's second 10G Intel nic. It appears the panics happen more under large file copy loads on the 10G connection. Will move it back to 1G to see if it makes a difference. Jan 19 16:27:05 Homeserver kernel: Call Trace: Jan 19 16:27:05 Homeserver kernel: <IRQ> Jan 19 16:27:05 Homeserver kernel: dump_stack+0x67/0x83 Jan 19 16:27:05 Homeserver kernel: nmi_cpu_backtrace+0x71/0x83 Jan 19 16:27:05 Homeserver kernel: ? lapic_can_unplug_cpu+0x97/0x97 Jan 19 16:27:05 Homeserver kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4 Jan 19 16:27:05 Homeserver kernel: rcu_dump_cpu_stacks+0x8b/0xb4 Jan 19 16:27:05 Homeserver kernel: rcu_check_callbacks+0x296/0x5a0 Jan 19 16:27:05 Homeserver kernel: update_process_times+0x24/0x47 Jan 19 16:27:05 Homeserver kernel: tick_sched_timer+0x36/0x64 Jan 19 16:27:05 Homeserver kernel: __hrtimer_run_queues+0xb7/0x10b Jan 19 16:27:05 Homeserver kernel: ? tick_sched_handle.isra.0+0x2f/0x2f Jan 19 16:27:05 Homeserver kernel: hrtimer_interrupt+0xf4/0x20e Jan 19 16:27:05 Homeserver kernel: smp_apic_timer_interrupt+0x7b/0x93 Jan 19 16:27:05 Homeserver kernel: apic_timer_interrupt+0xf/0x20 Jan 19 16:27:05 Homeserver kernel: </IRQ> Jan 19 16:27:05 Homeserver kernel: RIP: 0010:gc_worker+0xad/0x270 Jan 19 16:27:05 Homeserver kernel: Code: f6 c6 01 0f 85 4a 01 00 00 41 0f b6 46 37 49 c7 c0 f0 ff ff ff 41 ff c5 48 6b c0 38 49 29 c0 4f 8d 3c 06 49 8b 97 80 00 00 00 <41> 8b 87 88 00 00 00 0f ba e2 0e 73 2c 48 8b 15 ce dc 88 00 29 d0 Jan 19 16:27:05 Homeserver kernel: RSP: 0018:ffffc9001683fe60 EFLAGS: 00000296 ORIG_RAX: ffffffffffffff13 Jan 19 16:27:05 Homeserver kernel: RAX: 0000000000000038 RBX: 0000000000000000 RCX: 0000000000010000 Jan 19 16:27:05 Homeserver kernel: RDX: 0000000000000188 RSI: 00000000000000ad RDI: ffff8887f610d500 Jan 19 16:27:05 Homeserver kernel: RBP: 0000000000005aae R08: ffffffffffffffb8 R09: ffffffff81574c00 Jan 19 16:27:05 Homeserver kernel: R10: ffffea000edc5700 R11: ffff8887f610d501 R12: ffffffff822aa760 Jan 19 16:27:05 Homeserver kernel: R13: 00000000dba74d6c R14: ffff8887abf8ca48 R15: ffff8887abf8ca00 Jan 19 16:27:05 Homeserver kernel: ? nf_ct_get_id+0x80/0xb7 Jan 19 16:27:05 Homeserver kernel: process_one_work+0x16e/0x24f Jan 19 16:27:05 Homeserver kernel: worker_thread+0x1e2/0x2b8 Jan 19 16:27:05 Homeserver kernel: ? rescuer_thread+0x2a7/0x2a7 Jan 19 16:27:05 Homeserver kernel: kthread+0x10c/0x114 Jan 19 16:27:05 Homeserver kernel: ? kthread_park+0x89/0x89 Jan 19 16:27:05 Homeserver kernel: ret_from_fork+0x22/0x40
  9. Ok to update this thread. I tried going back to 6.8.3 on the 2nd and 3rd of my 4 servers that are kernel panicking and they still having panics and crashes daily. My only server that is not experiencing any issues is my 4thone that is 1G connected one. All of my 10G are panicking, and I have replaced the nics to intel server class 10g nics. I finally took my 2nd server back to a 1G connection to see if that stays stable. I have more log snippets from the 10G servers. It looks like they are also having a native_queued_spin_lock_slowpath error in the panic. Call Trace: Jan 19 12:52:28 Mediaserver kernel: <IRQ> Jan 19 12:52:28 Mediaserver kernel: dump_stack+0x67/0x83 Jan 19 12:52:28 Mediaserver kernel: nmi_cpu_backtrace+0x71/0x83 Jan 19 12:52:28 Mediaserver kernel: ? lapic_can_unplug_cpu+0x97/0x97 Jan 19 12:52:28 Mediaserver kernel: nmi_trigger_cpumask_backtrace+0x57/0xd4 Jan 19 12:52:28 Mediaserver kernel: rcu_dump_cpu_stacks+0x8b/0xb4 Jan 19 12:52:28 Mediaserver kernel: rcu_check_callbacks+0x296/0x5a0 Jan 19 12:52:28 Mediaserver kernel: update_process_times+0x24/0x47 Jan 19 12:52:28 Mediaserver kernel: tick_sched_timer+0x36/0x64 Jan 19 12:52:28 Mediaserver kernel: __hrtimer_run_queues+0xb7/0x10b Jan 19 12:52:28 Mediaserver kernel: ? tick_sched_handle.isra.0+0x2f/0x2f Jan 19 12:52:28 Mediaserver kernel: hrtimer_interrupt+0xf4/0x20e Jan 19 12:52:28 Mediaserver kernel: smp_apic_timer_interrupt+0x7b/0x93 Jan 19 12:52:28 Mediaserver kernel: apic_timer_interrupt+0xf/0x20 Jan 19 12:52:28 Mediaserver kernel: </IRQ> RIP: 0010:native_queued_spin_lock_slowpath+0x6b/0x171 Jan 19 12:52:28 Mediaserver kernel: Code: 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 74 0e 81 e6 00 ff 00 00 75 1a c6 47 01 00 eb 14 85 f6 74 0a 8b 07 84 c0 74 04 f3 90 <eb> f6 66 c7 07 01 00 c3 48 c7 c2 40 07 02 00 65 48 03 15 80 6a f8 Jan 19 12:52:28 Mediaserver kernel: RSP: 0018:ffffc90003ce3b88 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 Jan 19 12:52:28 Mediaserver kernel: RAX: 00000000001c0101 RBX: ffffc90003ce3c10 RCX: 000ffffffffff000 Jan 19 12:52:28 Mediaserver kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffea002085d368 Jan 19 12:52:28 Mediaserver kernel: RBP: ffffea0004d77200 R08: ffff888000000000 R09: ffffea0004d77240 Jan 19 12:52:28 Mediaserver kernel: R10: 0000000000000008 R11: 0000000000023eb8 R12: ffffea0004d77200 Jan 19 12:52:28 Mediaserver kernel: R13: ffff8882ed6dc400 R14: ffffea0004d77200 R15: ffff888114684600
  10. So my second server just crashed with a kernel panic, all three are having panics and they are all different hardware. Any idea from this trace? Jan 15 22:36:42 Mediaserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU Jan 15 22:36:42 Mediaserver kernel: rcu: #0110-....: (59999 ticks this GP) idle=e7a/1/0x4000000000000000 softirq=11770626/11770626 fqs=14993 Jan 15 22:36:42 Mediaserver kernel: #011(t=60000 jiffies g=13660245 q=3404623) Jan 15 22:36:42 Mediaserver kernel: NMI backtrace for cpu 0 Jan 15 22:36:42 Mediaserver kernel: CPU: 0 PID: 28592 Comm: kworker/u24:0 Tainted: P O 5.10.1-Unraid #1 Jan 15 22:36:42 Mediaserver kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z390 Extreme4, BIOS P2.30 12/25/2018 Jan 15 22:36:42 Mediaserver kernel: Workqueue: events_power_efficient gc_worker Jan 15 22:36:42 Mediaserver kernel: Call Trace: Jan 15 22:36:42 Mediaserver kernel: <IRQ> Jan 15 22:36:42 Mediaserver kernel: dump_stack+0x6b/0x83 Jan 15 22:36:42 Mediaserver kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Jan 15 22:36:42 Mediaserver kernel: nmi_cpu_backtrace+0x7d/0x8f Jan 15 22:36:42 Mediaserver kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3 Jan 15 22:36:42 Mediaserver kernel: rcu_dump_cpu_stacks+0x9f/0xc6 Jan 15 22:36:42 Mediaserver kernel: rcu_sched_clock_irq+0x1ec/0x543 Jan 15 22:36:42 Mediaserver kernel: ? _raw_spin_unlock_irqrestore+0xd/0xe Jan 15 22:36:42 Mediaserver kernel: update_process_times+0x50/0x6e Jan 15 22:36:42 Mediaserver kernel: tick_sched_timer+0x36/0x64 Jan 15 22:36:42 Mediaserver kernel: __hrtimer_run_queues+0xb7/0x10b Jan 15 22:36:42 Mediaserver kernel: ? tick_sched_do_timer+0x39/0x39 Jan 15 22:36:42 Mediaserver kernel: hrtimer_interrupt+0x8d/0x160 Jan 15 22:36:42 Mediaserver kernel: __sysvec_apic_timer_interrupt+0x5d/0x68 Jan 15 22:36:42 Mediaserver kernel: asm_call_irq_on_stack+0x12/0x20 Jan 15 22:36:42 Mediaserver kernel: </IRQ> Jan 15 22:36:42 Mediaserver kernel: sysvec_apic_timer_interrupt+0x71/0x95 Jan 15 22:36:42 Mediaserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20 Jan 15 22:36:42 Mediaserver kernel: RIP: 0010:gc_worker+0xf4/0x240 Jan 15 22:36:42 Mediaserver kernel: Code: 5c 26 05 41 89 47 08 e9 bc 00 00 00 48 8b 15 ec 05 a4 00 29 d0 85 c0 7f 11 4c 89 ff e8 10 f0 ff ff ff 44 24 08 e9 9e 00 00 00 <85> db 0f 84 96 00 00 00 49 8b 87 80 00 00 00 a8 08 0f 84 87 00 00 Jan 15 22:36:42 Mediaserver kernel: RSP: 0018:ffffc9000525fe48 EFLAGS: 00000206 Jan 15 22:36:42 Mediaserver kernel: RAX: 0000000001447690 RBX: 0000000000000000 RCX: ffff888103000000 Jan 15 22:36:42 Mediaserver kernel: RDX: 00000001014602fe RSI: ffffc9000525fe5c RDI: ffff88840da28548 Jan 15 22:36:42 Mediaserver kernel: RBP: 000000000000c386 R08: 0000000000000000 R09: ffffffff815c56ac Jan 15 22:36:42 Mediaserver kernel: R10: 8080808080808080 R11: ffff88830e1fa780 R12: ffffffff82547ec0 Jan 15 22:36:42 Mediaserver kernel: R13: 000000009fd57c44 R14: ffff88840da28548 R15: ffff88840da28500 Jan 15 22:36:42 Mediaserver kernel: ? nf_conntrack_free+0x2b/0x35 Jan 15 22:36:42 Mediaserver kernel: ? gc_worker+0x9a/0x240 Jan 15 22:36:42 Mediaserver kernel: process_one_work+0x13c/0x1d5 Jan 15 22:36:42 Mediaserver kernel: worker_thread+0x18b/0x22f Jan 15 22:36:42 Mediaserver kernel: ? process_scheduled_works+0x27/0x27 Jan 15 22:36:42 Mediaserver kernel: kthread+0xe5/0xea Jan 15 22:36:42 Mediaserver kernel: ? kthread_unpark+0x52/0x52 Jan 15 22:36:42 Mediaserver kernel: ret_from_fork+0x22/0x30
  11. Ok, I figured it out. I was going about it backwards. In the network settings you can arrange the MAC addresses of the NIC's to what Eth port you want to assign them to. I just rearranged the port 0 MAC address to the Eth0 configuration To simplify the networking I turned off the bond for Eth 0-2 that was set to active-passive (that was the unraid default BTW) . I'm betting it was bouncing since I only had 10G port 1 (Eth1) plugged in. I will report back on the stability
  12. So I have a MB with a integrated 1G ethernet that is mounted as Eth0 I have a Intel 10G 2port SFP+ card that is mounted at Eth1 and Eth2 I have a single DAC cable in Eth1 of the 10G card It is configured as a active bridge on br0 for Eth0, Eth1, Eth2 This is all the default config I went into the BIOS and turned off the built in 1G card mounted as Eth0 I wanted the system to default to the port 0 of the 10G as Eth0 On bootup it has an error that Eth0 can't be found. How do I make the server forget the disabled 1G port and make the 10G port 0 as Eth0?
  13. Yes, I have the GPU stat plugin installed. Any insight on the kernel panic trace above?
  14. Does the above traces connect with the Nvidia driver at all? I'm seeing this in the log this morning after a reboot repeated a lot. Jan 15 10:12:30 Homeserver kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window] Jan 15 10:12:30 Homeserver kernel: caller _nv000709rm+0x1af/0x200 [nvidia] mapping multiple BARs Jan 15 10:12:32 Homeserver kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window] Jan 15 10:12:32 Homeserver kernel: caller _nv000709rm+0x1af/0x200 [nvidia] mapping multiple BARs Jan 15 10:12:34 Homeserver kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window] Jan 15 10:12:34 Homeserver kernel: caller _nv000709rm+0x1af/0x200 [nvidia] mapping multiple BARs Jan 15 10:12:35 Homeserver kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window] Jan 15 10:12:35 Homeserver kernel: caller _nv000709rm+0x1af/0x200 [nvidia] mapping multiple BARs Jan 15 10:12:36 Homeserver kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window] Jan 15 10:12:36 Homeserver kernel: caller _nv000709rm+0x1af/0x200 [nvidia] mapping multiple BARs Jan 15 10:12:38 Homeserver kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window] Jan 15 10:12:38 Homeserver kernel: caller _nv000709rm+0x1af/0x200 [nvidia] mapping multiple BARs Jan 15 10:12:39 Homeserver kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window] Jan 15 10:12:39 Homeserver kernel: caller _nv000709rm+0x1af/0x200 [nvidia] mapping multiple BARs
  15. Happened again 24hrs later. Page faulted: Jan 11 08:30:00 Homeserver kernel: BUG: unable to handle page fault for address: 00000000000053d8 Whole trace: an 11 04:07:30 Homeserver kernel: br0: port 1(bond0) entered forwarding state Jan 11 04:08:25 Homeserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 11 04:15:05 Homeserver kernel: br0: received packet on bond0 with own address as source address (addr:30:9c:23:af:51:e0, vlan:0) Jan 11 04:15:05 Homeserver kernel: br0: received packet on bond0 with own address as source address (addr:30:9c:23:af:51:e0, vlan:0) Jan 11 04:15:06 Homeserver kernel: br0: received packet on bond0 with own address as source address (addr:30:9c:23:af:51:e0, vlan:0) Jan 11 04:15:07 Homeserver kernel: br0: received packet on bond0 with own address as source address (addr:30:9c:23:af:51:e0, vlan:0) Jan 11 04:15:07 Homeserver kernel: br0: received packet on bond0 with own address as source address (addr:30:9c:23:af:51:e0, vlan:0) Jan 11 04:15:06 Homeserver kernel: br0: received packet on bond0 with own address as source address (addr:30:9c:23:af:51:e0, vlan:0) Jan 11 05:08:32 Homeserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 11 06:08:38 Homeserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 11 07:04:35 Homeserver vsftpd[1787]: connect from 192.168.1.1 (192.168.1.1) Jan 11 07:04:35 Homeserver sshd[1788]: Connection from 192.168.1.1 port 50482 on 192.168.1.251 port 22 rdomain "" Jan 11 07:04:35 Homeserver sshd[1788]: error: kex_exchange_identification: Connection closed by remote host Jan 11 07:04:35 Homeserver sshd[1788]: Connection closed by 192.168.1.1 port 50482 Jan 11 07:04:35 Homeserver vsftpd[1794]: connect from 192.168.1.1 (192.168.1.1) Jan 11 07:04:35 Homeserver vsftpd[1796]: connect from 192.168.1.1 (192.168.1.1) Jan 11 07:04:35 Homeserver vsftpd[1798]: connect from 192.168.1.1 (192.168.1.1) Jan 11 07:04:35 Homeserver vsftpd[1800]: connect from 192.168.1.1 (192.168.1.1) Jan 11 07:04:35 Homeserver vsftpd[1802]: connect from 192.168.1.1 (192.168.1.1) Jan 11 07:04:46 Homeserver smbd[1790]: [2021/01/11 07:04:46.002732, 0] ../../source3/smbd/process.c:341(read_packet_remainder) Jan 11 07:04:46 Homeserver smbd[1790]: read_fd_with_timeout failed for client 192.168.1.1 read error = NT_STATUS_END_OF_FILE. Jan 11 07:08:45 Homeserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 11 08:08:51 Homeserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 11 08:30:00 Homeserver kernel: BUG: unable to handle page fault for address: 00000000000053d8 Jan 11 08:30:00 Homeserver kernel: #PF: supervisor write access in kernel mode Jan 11 08:30:00 Homeserver kernel: #PF: error_code(0x0002) - not-present page Jan 11 08:30:00 Homeserver kernel: PGD 0 P4D 0 Jan 11 08:30:00 Homeserver kernel: Oops: 0002 [#1] SMP NOPTI Jan 11 08:30:00 Homeserver kernel: CPU: 6 PID: 11516 Comm: php7 Tainted: P S W O 5.10.1-Unraid #1 Jan 11 08:30:00 Homeserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B78/X470 GAMING PRO CARBON (MS-7B78), BIOS 2.20 04/24/2018 Jan 11 08:30:00 Homeserver kernel: RIP: 0010:slab_post_alloc_hook+0xe1/0x14a Jan 11 08:30:00 Homeserver kernel: Code: 0b 48 8b 55 08 f0 48 83 02 01 eb 04 65 48 ff 02 48 89 44 24 08 e8 da 4b f6 ff 49 8b 57 38 48 89 ef 48 8b 44 24 08 48 83 e2 fe <48> 89 2c c2 49 8b 07 8b 53 08 8b 4b 18 48 c1 e8 3a 48 8b 34 c5 a0 Jan 11 08:30:00 Homeserver kernel: RSP: 0000:ffffc90008cbfd90 EFLAGS: 00010202 Jan 11 08:30:00 Homeserver kernel: RAX: 0000000000000a79 RBX: ffff888100045500 RCX: 0000000000000005 Jan 11 08:30:00 Homeserver kernel: RDX: 0000000000000010 RSI: ffff888187c80000 RDI: ffff888164310f40 Jan 11 08:30:00 Homeserver kernel: RBP: ffff888164310f40 R08: ffffc90008cbfdd8 R09: 0000000080000000 Jan 11 08:30:00 Homeserver kernel: R10: ffffea00061f2000 R11: 000000000000ffff R12: 0000000000000000 Jan 11 08:30:00 Homeserver kernel: R13: 0000000000000cc0 R14: ffffc90008cbfdd8 R15: ffffea00061f2000 Jan 11 08:30:00 Homeserver kernel: FS: 000014d90dc12d48(0000) GS:ffff8887fed80000(0000) knlGS:0000000000000000 Jan 11 08:30:00 Homeserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 11 08:30:00 Homeserver kernel: CR2: 00000000000053d8 CR3: 0000000272046000 CR4: 00000000003506e0 Jan 11 08:30:00 Homeserver kernel: Call Trace: Jan 11 08:30:00 Homeserver kernel: ? __anon_vma_prepare+0x2b/0x110 Jan 11 08:30:00 Homeserver kernel: kmem_cache_alloc+0x108/0x130 Jan 11 08:30:00 Homeserver kernel: __anon_vma_prepare+0x2b/0x110 Jan 11 08:30:00 Homeserver kernel: handle_mm_fault+0xd5d/0xec3 Jan 11 08:30:00 Homeserver kernel: exc_page_fault+0x253/0x36d Jan 11 08:30:00 Homeserver kernel: ? asm_exc_page_fault+0x8/0x30 Jan 11 08:30:00 Homeserver kernel: asm_exc_page_fault+0x1e/0x30 Jan 11 08:30:00 Homeserver kernel: RIP: 0033:0x14d90dbcddc0 Jan 11 08:30:00 Homeserver kernel: Code: 27 48 89 47 2f 48 89 47 37 48 89 44 17 c1 48 89 44 17 c9 48 89 44 17 d1 48 89 44 17 d9 48 89 f8 c3 f7 c7 0f 00 00 00 49 89 f8 <48> 89 44 17 f8 48 89 d1 75 0b 48 c1 e9 03 f3 48 ab 4c 89 c0 c3 31 Jan 11 08:30:00 Homeserver kernel: RSP: 002b:00007ffd4698f118 EFLAGS: 00010206 Jan 11 08:30:00 Homeserver kernel: RAX: 0000000000000000 RBX: 00007ffd4698f258 RCX: 000014d90dbb2c32 Jan 11 08:30:00 Homeserver kernel: RDX: 0000000000000f14 RSI: 0000000000000000 RDI: 000014d90c9910ec Jan 11 08:30:00 Homeserver kernel: RBP: 00007ffd4698f630 R08: 000014d90c9910ec R09: 0000000000012000 Jan 11 08:30:00 Homeserver kernel: R10: 0000000000000012 R11: 000014d90c992000 R12: 0000000000016000 Jan 11 08:30:00 Homeserver kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003 Jan 11 08:30:00 Homeserver kernel: Modules linked in: xt_CHECKSUM ipt_REJECT macvlan ip6table_mangle ip6table_nat iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper drm backlight agpgart syscopyarea sysfillrect nvidia_uvm(PO) sysimgblt fb_sys_fops nvidia(PO) bonding ixgbe mdio igb i2c_algo_bit edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel i2c_piix4 crypto_simd i2c_core mxm_wmi mpt3sas wmi_bmof cryptd ahci ccp raid_class scsi_transport_sas glue_helper wmi k10temp libahci rapl button acpi_cpufreq [last unloaded: mdio] Jan 11 08:30:00 Homeserver kernel: CR2: 00000000000053d8 Jan 11 08:30:00 Homeserver kernel: ---[ end trace 5ed146f2988e06a2 ]--- Jan 11 08:30:00 Homeserver kernel: RIP: 0010:slab_post_alloc_hook+0xe1/0x14a Jan 11 08:30:00 Homeserver kernel: Code: 0b 48 8b 55 08 f0 48 83 02 01 eb 04 65 48 ff 02 48 89 44 24 08 e8 da 4b f6 ff 49 8b 57 38 48 89 ef 48 8b 44 24 08 48 83 e2 fe <48> 89 2c c2 49 8b 07 8b 53 08 8b 4b 18 48 c1 e8 3a 48 8b 34 c5 a0 Jan 11 08:30:00 Homeserver kernel: RSP: 0000:ffffc90008cbfd90 EFLAGS: 00010202 Jan 11 08:30:00 Homeserver kernel: RAX: 0000000000000a79 RBX: ffff888100045500 RCX: 0000000000000005 Jan 11 08:30:00 Homeserver kernel: RDX: 0000000000000010 RSI: ffff888187c80000 RDI: ffff888164310f40 Jan 11 08:30:00 Homeserver kernel: RBP: ffff888164310f40 R08: ffffc90008cbfdd8 R09: 0000000080000000 Jan 11 08:30:00 Homeserver kernel: R10: ffffea00061f2000 R11: 000000000000ffff R12: 0000000000000000 Jan 11 08:30:00 Homeserver kernel: R13: 0000000000000cc0 R14: ffffc90008cbfdd8 R15: ffffea00061f2000 Jan 11 08:30:00 Homeserver kernel: FS: 000014d90dc12d48(0000) GS:ffff8887fed80000(0000) knlGS:0000000000000000 Jan 11 08:30:00 Homeserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 11 08:30:00 Homeserver kernel: CR2: 00000000000053d8 CR3: 0000000272046000 CR4: 00000000003506e0 Jan 11 08:44:18 Homeserver kernel: general protection fault, probably for non-canonical address 0xdead000000000122: 0000 [#2] SMP NOPTI Jan 11 08:44:18 Homeserver kernel: CPU: 12 PID: 0 Comm: swapper/12 Tainted: P S D W O 5.10.1-Unraid #1 Jan 11 08:44:18 Homeserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B78/X470 GAMING PRO CARBON (MS-7B78), BIOS 2.20 04/24/2018 Jan 11 08:44:18 Homeserver kernel: RIP: 0010:nf_nat_setup_info+0x5f3/0x652 [nf_nat] Jan 11 08:44:18 Homeserver kernel: Code: 8b 05 01 56 00 00 48 8d 8b 98 00 00 00 4a 8d 14 e0 48 8b 02 48 89 93 a0 00 00 00 48 89 83 98 00 00 00 48 85 c0 48 89 0a 74 04 <48> 89 48 08 4c 89 ef e8 bf 34 5b e1 eb 13 41 83 fc 01 75 0d 48 81 Jan 11 08:44:18 Homeserver kernel: RSP: 0018:ffffc900003a0838 EFLAGS: 00010286 Jan 11 08:44:18 Homeserver kernel: RAX: dead000000000122 RBX: ffff88826a62c3c0 RCX: ffff88826a62c458 Jan 11 08:44:18 Homeserver kernel: RDX: ffff888187ca9e60 RSI: 00000000bdba70db RDI: ffffffffa0146550 Jan 11 08:44:18 Homeserver kernel: RBP: ffffc900003a0900 R08: 000000003221cea3 R09: ffff888235dfdf80 Jan 11 08:44:18 Homeserver kernel: R10: 0000000000000098 R11: ffff88826ca1df00 R12: 00000000000053cc Jan 11 08:44:18 Homeserver kernel: R13: ffffffffa0146550 R14: ffffc900003a0914 R15: 0000000000000001 Jan 11 08:44:18 Homeserver kernel: FS: 0000000000000000(0000) GS:ffff8887fef00000(0000) knlGS:0000000000000000 Jan 11 08:44:18 Homeserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 11 08:44:18 Homeserver kernel: CR2: 000034f46d1b1000 CR3: 00000001e1848000 CR4: 00000000003506e0 Jan 11 08:44:18 Homeserver kernel: Call Trace: Jan 11 08:44:18 Homeserver kernel: <IRQ> Jan 11 08:44:18 Homeserver kernel: ? ipt_do_table+0x4a2/0x5c0 [ip_tables] Jan 11 08:44:18 Homeserver kernel: nf_nat_alloc_null_binding+0x71/0x88 [nf_nat] Jan 11 08:44:18 Homeserver kernel: nf_nat_inet_fn+0x91/0x182 [nf_nat] Jan 11 08:44:18 Homeserver kernel: nf_hook_slow+0x39/0x8e Jan 11 08:44:18 Homeserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 11 08:44:18 Homeserver kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe Jan 11 08:44:18 Homeserver kernel: ip_local_deliver+0x49/0x75 Jan 11 08:44:18 Homeserver kernel: __netif_receive_skb_one_core+0x74/0x95 Jan 11 08:44:18 Homeserver kernel: netif_receive_skb+0x79/0xa1 Jan 11 08:44:18 Homeserver kernel: br_handle_frame_finish+0x30d/0x351 Jan 11 08:44:18 Homeserver kernel: ? ipt_do_table+0x570/0x5c0 [ip_tables] Jan 11 08:44:18 Homeserver kernel: ? br_pass_frame_up+0xda/0xda Jan 11 08:44:18 Homeserver kernel: br_nf_hook_thresh+0xa3/0xc3 Jan 11 08:44:18 Homeserver kernel: ? br_pass_frame_up+0xda/0xda Jan 11 08:44:18 Homeserver kernel: br_nf_pre_routing_finish+0x23d/0x264 Jan 11 08:44:18 Homeserver kernel: ? br_pass_frame_up+0xda/0xda Jan 11 08:44:18 Homeserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 11 08:44:18 Homeserver kernel: ? nf_nat_ipv4_in+0x1e/0x4a [nf_nat] Jan 11 08:44:18 Homeserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 11 08:44:18 Homeserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 11 08:44:18 Homeserver kernel: NF_HOOK+0xd7/0xf7 Jan 11 08:44:18 Homeserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 11 08:44:18 Homeserver kernel: br_nf_pre_routing+0x229/0x239 Jan 11 08:44:18 Homeserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 11 08:44:18 Homeserver kernel: br_handle_frame+0x25e/0x2a6 Jan 11 08:44:18 Homeserver kernel: ? br_pass_frame_up+0xda/0xda Jan 11 08:44:18 Homeserver kernel: __netif_receive_skb_core+0x335/0x4e7 Jan 11 08:44:18 Homeserver kernel: __netif_receive_skb_list_core+0x78/0x104 Jan 11 08:44:18 Homeserver kernel: netif_receive_skb_list_internal+0x1bf/0x1f2 Jan 11 08:44:18 Homeserver kernel: ? dev_gro_receive+0x55d/0x578 Jan 11 08:44:18 Homeserver kernel: gro_normal_list+0x1d/0x39 Jan 11 08:44:18 Homeserver kernel: napi_complete_done+0x79/0x104 Jan 11 08:44:18 Homeserver kernel: ixgbe_poll+0xc95/0xd4f [ixgbe] Jan 11 08:44:18 Homeserver kernel: net_rx_action+0xf4/0x29d Jan 11 08:44:18 Homeserver kernel: __do_softirq+0xc4/0x1c2 Jan 11 08:44:18 Homeserver kernel: asm_call_irq_on_stack+0x12/0x20 Jan 11 08:44:18 Homeserver kernel: </IRQ> Jan 11 08:44:18 Homeserver kernel: do_softirq_own_stack+0x2c/0x39 Jan 11 08:44:18 Homeserver kernel: __irq_exit_rcu+0x45/0x80 Jan 11 08:44:18 Homeserver kernel: common_interrupt+0x119/0x12e Jan 11 08:44:18 Homeserver kernel: asm_common_interrupt+0x1e/0x40 Jan 11 08:44:18 Homeserver kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8 Jan 11 08:44:18 Homeserver kernel: Code: 00 48 83 c4 28 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 55 8b af 28 04 00 00 b8 01 00 00 00 45 31 c9 53 45 31 d2 39 c5 Jan 11 08:44:18 Homeserver kernel: RSP: 0018:ffffc9000016fea0 EFLAGS: 00000246 Jan 11 08:44:18 Homeserver kernel: RAX: ffff8887fef22300 RBX: 0000000000000002 RCX: 000000000000001f Jan 11 08:44:18 Homeserver kernel: RDX: 0000000000000000 RSI: 0000000022983893 RDI: 0000000000000000 Jan 11 08:44:18 Homeserver kernel: RBP: ffff8881059e0c00 R08: 0000b5548bebb872 R09: 0000b55487f71fc0 Jan 11 08:44:18 Homeserver kernel: R10: 00000000000012f5 R11: 071c71c71c71c71c R12: 0000b5548bebb872 Jan 11 08:44:18 Homeserver kernel: R13: ffffffff820cad00 R14: 0000000000000002 R15: 0000000000000000 Jan 11 08:44:18 Homeserver kernel: cpuidle_enter_state+0x101/0x1c4 Jan 11 08:44:18 Homeserver kernel: cpuidle_enter+0x25/0x31 Jan 11 08:44:18 Homeserver kernel: do_idle+0x1a1/0x20f Jan 11 08:44:18 Homeserver kernel: cpu_startup_entry+0x18/0x1a Jan 11 08:44:18 Homeserver kernel: secondary_startup_64_no_verify+0xb0/0xbb Jan 11 08:44:18 Homeserver kernel: Modules linked in: xt_CHECKSUM ipt_REJECT macvlan ip6table_mangle ip6table_nat iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper drm backlight agpgart syscopyarea sysfillrect nvidia_uvm(PO) sysimgblt fb_sys_fops nvidia(PO) bonding ixgbe mdio igb i2c_algo_bit edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel i2c_piix4 crypto_simd i2c_core mxm_wmi mpt3sas wmi_bmof cryptd ahci ccp raid_class scsi_transport_sas glue_helper wmi k10temp libahci rapl button acpi_cpufreq [last unloaded: mdio] Jan 11 08:44:18 Homeserver kernel: ---[ end trace 5ed146f2988e06a3 ]---
  16. Can anyone take a look and let me know what this kernel panic is caused by. Here is the trace from the syslog. I'm actually getting these somewhat regularlly on three different 6.9-RC@ unraid servers. Jan 12 07:15:08 Homeserver kernel: ------------[ cut here ]------------ Jan 12 07:15:08 Homeserver kernel: WARNING: CPU: 0 PID: 0 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x99/0x1e1 Jan 12 07:15:08 Homeserver kernel: Modules linked in: xt_CHECKSUM ipt_REJECT macvlan ip6table_mangle ip6table_nat iptable_mangle ip6table_filter ip6_tables vhost_net tun vhost vhost_iotlb tap veth xt_nat xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs nfsd lockd grace sunrpc md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper drm backlight agpgart syscopyarea sysfillrect nvidia_uvm(PO) sysimgblt fb_sys_fops nvidia(PO) bonding ixgbe mdio igb i2c_algo_bit edac_mce_amd kvm_amd kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel mpt3sas aesni_intel crypto_simd mxm_wmi raid_class wmi_bmof cryptd i2c_piix4 scsi_transport_sas wmi glue_helper i2c_core k10temp ccp ahci rapl libahci button acpi_cpufreq [last unloaded: mdio] Jan 12 07:15:08 Homeserver kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: P S O 5.10.1-Unraid #1 Jan 12 07:15:08 Homeserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B78/X470 GAMING PRO CARBON (MS-7B78), BIOS 2.20 04/24/2018 Jan 12 07:15:08 Homeserver kernel: RIP: 0010:__nf_conntrack_confirm+0x99/0x1e1 Jan 12 07:15:08 Homeserver kernel: Code: e4 e3 ff ff 8b 54 24 14 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 54 e1 ff ff 84 c0 75 b8 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 89 de ff ff e8 af e0 ff ff e9 1f 01 Jan 12 07:15:08 Homeserver kernel: RSP: 0018:ffffc90000003898 EFLAGS: 00010202 Jan 12 07:15:08 Homeserver kernel: RAX: 0000000000000188 RBX: 00000000000034f1 RCX: 000000003317c1ec Jan 12 07:15:08 Homeserver kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8200a078 Jan 12 07:15:08 Homeserver kernel: RBP: ffff8882471a2f00 R08: 000000005cf7db60 R09: ffff888182b03440 Jan 12 07:15:08 Homeserver kernel: R10: 0000000000000158 R11: ffff888104bed100 R12: 000000000000d20e Jan 12 07:15:08 Homeserver kernel: R13: ffffffff8210da40 R14: 00000000000034f1 R15: ffff8882471a2f0c Jan 12 07:15:08 Homeserver kernel: FS: 0000000000000000(0000) GS:ffff8887fec00000(0000) knlGS:0000000000000000 Jan 12 07:15:08 Homeserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 12 07:15:08 Homeserver kernel: CR2: 0000145dc4003428 CR3: 00000001aceda000 CR4: 00000000003506f0 Jan 12 07:15:08 Homeserver kernel: Call Trace: Jan 12 07:15:08 Homeserver kernel: <IRQ> Jan 12 07:15:08 Homeserver kernel: nf_conntrack_confirm+0x2f/0x36 Jan 12 07:15:08 Homeserver kernel: nf_hook_slow+0x39/0x8e Jan 12 07:15:08 Homeserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 12 07:15:08 Homeserver kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe Jan 12 07:15:08 Homeserver kernel: ip_local_deliver+0x49/0x75 Jan 12 07:15:08 Homeserver kernel: ip_sabotage_in+0x43/0x4d Jan 12 07:15:08 Homeserver kernel: nf_hook_slow+0x39/0x8e Jan 12 07:15:08 Homeserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 12 07:15:08 Homeserver kernel: ? l3mdev_l3_rcv.constprop.0+0x50/0x50 Jan 12 07:15:08 Homeserver kernel: ip_rcv+0x41/0x61 Jan 12 07:15:08 Homeserver kernel: __netif_receive_skb_one_core+0x74/0x95 Jan 12 07:15:08 Homeserver kernel: netif_receive_skb+0x79/0xa1 Jan 12 07:15:08 Homeserver kernel: br_handle_frame_finish+0x30d/0x351 Jan 12 07:15:08 Homeserver kernel: ? ipt_do_table+0x570/0x5c0 [ip_tables] Jan 12 07:15:08 Homeserver kernel: ? br_pass_frame_up+0xda/0xda Jan 12 07:15:08 Homeserver kernel: br_nf_hook_thresh+0xa3/0xc3 Jan 12 07:15:08 Homeserver kernel: ? br_pass_frame_up+0xda/0xda Jan 12 07:15:08 Homeserver kernel: br_nf_pre_routing_finish+0x23d/0x264 Jan 12 07:15:08 Homeserver kernel: ? br_pass_frame_up+0xda/0xda Jan 12 07:15:08 Homeserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 12 07:15:08 Homeserver kernel: ? nf_nat_ipv4_in+0x1e/0x4a [nf_nat] Jan 12 07:15:08 Homeserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 12 07:15:08 Homeserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 12 07:15:08 Homeserver kernel: NF_HOOK+0xd7/0xf7 Jan 12 07:15:08 Homeserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 12 07:15:08 Homeserver kernel: br_nf_pre_routing+0x229/0x239 Jan 12 07:15:08 Homeserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 12 07:15:08 Homeserver kernel: br_handle_frame+0x25e/0x2a6 Jan 12 07:15:08 Homeserver kernel: ? br_pass_frame_up+0xda/0xda Jan 12 07:15:08 Homeserver kernel: __netif_receive_skb_core+0x335/0x4e7 Jan 12 07:15:08 Homeserver kernel: ? find_busiest_group+0x3b/0x2bc Jan 12 07:15:08 Homeserver kernel: __netif_receive_skb_list_core+0x78/0x104 Jan 12 07:15:08 Homeserver kernel: netif_receive_skb_list_internal+0x1bf/0x1f2 Jan 12 07:15:08 Homeserver kernel: ? dev_gro_receive+0x55d/0x578 Jan 12 07:15:08 Homeserver kernel: gro_normal_list+0x1d/0x39 Jan 12 07:15:08 Homeserver kernel: napi_complete_done+0x79/0x104 Jan 12 07:15:08 Homeserver kernel: ixgbe_poll+0xc95/0xd4f [ixgbe] Jan 12 07:15:08 Homeserver kernel: ? update_cfs_rq_load_avg+0x14b/0x154 Jan 12 07:15:08 Homeserver kernel: net_rx_action+0xf4/0x29d Jan 12 07:15:08 Homeserver kernel: __do_softirq+0xc4/0x1c2 Jan 12 07:15:08 Homeserver kernel: asm_call_irq_on_stack+0x12/0x20 Jan 12 07:15:08 Homeserver kernel: </IRQ> Jan 12 07:15:08 Homeserver kernel: do_softirq_own_stack+0x2c/0x39 Jan 12 07:15:08 Homeserver kernel: __irq_exit_rcu+0x45/0x80 Jan 12 07:15:08 Homeserver kernel: common_interrupt+0x119/0x12e Jan 12 07:15:08 Homeserver kernel: asm_common_interrupt+0x1e/0x40 Jan 12 07:15:08 Homeserver kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8 Jan 12 07:15:08 Homeserver kernel: Code: 00 48 83 c4 28 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 55 8b af 28 04 00 00 b8 01 00 00 00 45 31 c9 53 45 31 d2 39 c5 Jan 12 07:15:08 Homeserver kernel: RSP: 0018:ffffffff82003e60 EFLAGS: 00000246 Jan 12 07:15:08 Homeserver kernel: RAX: ffff8887fec22300 RBX: 0000000000000002 RCX: 000000000000001f Jan 12 07:15:08 Homeserver kernel: RDX: 0000000000000000 RSI: 00000000229839cd RDI: 0000000000000000 Jan 12 07:15:08 Homeserver kernel: RBP: ffff888101ae5400 R08: 000016ef236af812 R09: 0000000000000000 Jan 12 07:15:08 Homeserver kernel: R10: 0000000000004653 R11: 071c71c71c71c71c R12: 000016ef236af812 Jan 12 07:15:08 Homeserver kernel: R13: ffffffff820cad00 R14: 0000000000000002 R15: 0000000000000000 Jan 12 07:15:08 Homeserver kernel: cpuidle_enter_state+0x101/0x1c4 Jan 12 07:15:08 Homeserver kernel: cpuidle_enter+0x25/0x31 Jan 12 07:15:08 Homeserver kernel: do_idle+0x1a1/0x20f Jan 12 07:15:08 Homeserver kernel: cpu_startup_entry+0x18/0x1a Jan 12 07:15:08 Homeserver kernel: start_kernel+0x4c0/0x4e3 Jan 12 07:15:08 Homeserver kernel: secondary_startup_64_no_verify+0xb0/0xbb Jan 12 07:15:08 Homeserver kernel: ---[ end trace 20ff384d3480aa64 ]---
  17. Having one of my servers crash daily. Any ideas on what is generating the docker networking issues below? I see them constantly in the logs then the server will freeze or throw a kernel error and crash. I will see docker0: port 2-7 in the errors. The server is a Intel 9700K with a Intel 10gb NIC. The NIC has been replaced and the SFP+ DAC cable and still get the error. I have swapped most of the hardware out to check for hardware fault and no luck (New proc/MB/PSU/RAID card). I have also moved data off each drive and rebuilt the partition on each drive and moved data back to account for drive/filesystem issues. I have also rebuilt the docker.img and the docker appdata several times and moved them around to different cache and unassigned drive SSD's. Any help would be appreciated. Seeing this in my logs: Jan 2 09:31:13 NASserver kernel: veth2251ce3: renamed from eth0 Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth639656d) entered disabled state Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth639656d) entered disabled state Jan 2 09:31:13 NASserver kernel: device veth639656d left promiscuous mode Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth639656d) entered disabled state Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth6a994b4) entered blocking state Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth6a994b4) entered disabled state Jan 2 09:31:13 NASserver kernel: device veth6a994b4 entered promiscuous mode Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth6a994b4) entered blocking state Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth6a994b4) entered forwarding state Jan 2 09:31:13 NASserver kernel: eth0: renamed from veth7f12a06 Jan 2 09:31:13 NASserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth6a994b4: link becomes ready Jan 2 09:34:39 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 2 10:33:46 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 2 10:41:36 NASserver kernel: veth7f12a06: renamed from eth0 Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth6a994b4) entered disabled state Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth6a994b4) entered disabled state Jan 2 10:41:36 NASserver kernel: device veth6a994b4 left promiscuous mode Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth6a994b4) entered disabled state Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth7a586fa) entered blocking state Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth7a586fa) entered disabled state Jan 2 10:41:36 NASserver kernel: device veth7a586fa entered promiscuous mode Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth7a586fa) entered blocking state Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth7a586fa) entered forwarding state Jan 2 10:41:36 NASserver kernel: eth0: renamed from veth1986f4c Jan 2 10:41:36 NASserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7a586fa: link becomes ready Jan 2 10:56:52 NASserver kernel: veth1986f4c: renamed from eth0 Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth7a586fa) entered disabled state Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth7a586fa) entered disabled state Jan 2 10:56:52 NASserver kernel: device veth7a586fa left promiscuous mode Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth7a586fa) entered disabled state Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth0af469d) entered blocking state Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth0af469d) entered disabled state Jan 2 10:56:52 NASserver kernel: device veth0af469d entered promiscuous mode Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth0af469d) entered blocking state Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth0af469d) entered forwarding state Jan 2 10:56:53 NASserver kernel: eth0: renamed from veth2a96f02 Jan 2 10:56:53 NASserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0af469d: link becomes ready Jan 2 11:33:53 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 2 12:07:12 NASserver kernel: veth2a96f02: renamed from eth0 Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth0af469d) entered disabled state Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth0af469d) entered disabled state Jan 2 12:07:12 NASserver kernel: device veth0af469d left promiscuous mode Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth0af469d) entered disabled state Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth2629056) entered blocking state Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth2629056) entered disabled state Jan 2 12:07:12 NASserver kernel: device veth2629056 entered promiscuous mode Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth2629056) entered blocking state Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth2629056) entered forwarding state Jan 2 12:07:12 NASserver kernel: eth0: renamed from vethd0b0abb Jan 2 12:07:12 NASserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth2629056: link becomes ready Jan 2 12:33:59 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 2 13:34:06 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update When the kernel crashes I get a log below: Jan 1 16:04:15 NASserver kernel: ------------[ cut here ]------------ Jan 1 16:04:15 NASserver kernel: WARNING: CPU: 5 PID: 0 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x99/0x1e1 Jan 1 16:04:15 NASserver kernel: Modules linked in: macvlan tun xt_nat veth xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs md_mod i915 iosf_mbi i2c_algo_bit drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha bonding ixgbe mdio x86_pkg_temp_thermal wmi_bmof intel_powerclamp mxm_wmi intel_wmi_thunderbolt coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd btusb glue_helper btrtl rapl btbcm btintel intel_cstate bluetooth intel_uncore nvme ahci i2c_i801 mpt3sas i2c_smbus i2c_core nvme_core ecdh_generic libahci raid_class ecc scsi_transport_sas video thermal backlight fan wmi button acpi_pad [last unloaded: mdio] Jan 1 16:04:15 NASserver kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: G S 5.10.1-Unraid #1 Jan 1 16:04:15 NASserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B17/MPG Z390 GAMING EDGE AC (MS-7B17), BIOS A.B0 10/12/2020 Jan 1 16:04:15 NASserver kernel: RIP: 0010:__nf_conntrack_confirm+0x99/0x1e1 Jan 1 16:04:15 NASserver kernel: Code: e4 e3 ff ff 8b 54 24 14 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 54 e1 ff ff 84 c0 75 b8 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 89 de ff ff e8 af e0 ff ff e9 1f 01 Jan 1 16:04:15 NASserver kernel: RSP: 0018:ffffc900001ec898 EFLAGS: 00010202 Jan 1 16:04:15 NASserver kernel: RAX: 0000000000000188 RBX: 0000000000004ff2 RCX: 0000000038544019 Jan 1 16:04:15 NASserver kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8200a808 Jan 1 16:04:15 NASserver kernel: RBP: ffff88815d07c640 R08: 00000000de3985d0 R09: ffff88815ccb2bc0 Jan 1 16:04:15 NASserver kernel: R10: 0000000000000158 R11: ffff8881061fda00 R12: 0000000000004824 Jan 1 16:04:15 NASserver kernel: R13: ffffffff8210da40 R14: 0000000000004ff2 R15: ffff88815d07c64c Jan 1 16:04:15 NASserver kernel: FS: 0000000000000000(0000) GS:ffff88884dd40000(0000) knlGS:0000000000000000 Jan 1 16:04:15 NASserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 1 16:04:15 NASserver kernel: CR2: 0000153736b90f20 CR3: 0000000169908001 CR4: 00000000003706e0 Jan 1 16:04:15 NASserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 1 16:04:15 NASserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Jan 1 16:04:15 NASserver kernel: Call Trace: Jan 1 16:04:15 NASserver kernel: <IRQ> Jan 1 16:04:15 NASserver kernel: nf_conntrack_confirm+0x2f/0x36 Jan 1 16:04:15 NASserver kernel: nf_hook_slow+0x39/0x8e Jan 1 16:04:15 NASserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 1 16:04:15 NASserver kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe Jan 1 16:04:15 NASserver kernel: ip_local_deliver+0x49/0x75 Jan 1 16:04:15 NASserver kernel: ip_sabotage_in+0x43/0x4d Jan 1 16:04:15 NASserver kernel: nf_hook_slow+0x39/0x8e Jan 1 16:04:15 NASserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 1 16:04:15 NASserver kernel: ? l3mdev_l3_rcv.constprop.0+0x50/0x50 Jan 1 16:04:15 NASserver kernel: ip_rcv+0x41/0x61 Jan 1 16:04:15 NASserver kernel: __netif_receive_skb_one_core+0x74/0x95 Jan 1 16:04:15 NASserver kernel: netif_receive_skb+0x79/0xa1 Jan 1 16:04:15 NASserver kernel: br_handle_frame_finish+0x30d/0x351 Jan 1 16:04:15 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 16:04:15 NASserver kernel: br_nf_hook_thresh+0xa3/0xc3 Jan 1 16:04:15 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 16:04:15 NASserver kernel: br_nf_pre_routing_finish+0x23d/0x264 Jan 1 16:04:15 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 16:04:15 NASserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 1 16:04:15 NASserver kernel: ? nf_nat_ipv4_in+0x1e/0x4a [nf_nat] Jan 1 16:04:15 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 16:04:15 NASserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 1 16:04:15 NASserver kernel: NF_HOOK+0xd7/0xf7 Jan 1 16:04:15 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 16:04:15 NASserver kernel: br_nf_pre_routing+0x229/0x239 Jan 1 16:04:15 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 16:04:15 NASserver kernel: br_handle_frame+0x25e/0x2a6 Jan 1 16:04:15 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 16:04:15 NASserver kernel: __netif_receive_skb_core+0x335/0x4e7 Jan 1 16:04:15 NASserver kernel: __netif_receive_skb_list_core+0x78/0x104 Jan 1 16:04:15 NASserver kernel: netif_receive_skb_list_internal+0x1bf/0x1f2 Jan 1 16:04:15 NASserver kernel: ? dev_gro_receive+0x55d/0x578 Jan 1 16:04:15 NASserver kernel: gro_normal_list+0x1d/0x39 Jan 1 16:04:15 NASserver kernel: napi_complete_done+0x79/0x104 Jan 1 16:04:15 NASserver kernel: ixgbe_poll+0xc95/0xd4f [ixgbe] Jan 1 16:04:15 NASserver kernel: net_rx_action+0xf4/0x29d Jan 1 16:04:15 NASserver kernel: __do_softirq+0xc4/0x1c2 Jan 1 16:04:15 NASserver kernel: asm_call_irq_on_stack+0xf/0x20 Jan 1 16:04:15 NASserver kernel: </IRQ> Jan 1 16:04:15 NASserver kernel: do_softirq_own_stack+0x2c/0x39 Jan 1 16:04:15 NASserver kernel: __irq_exit_rcu+0x45/0x80 Jan 1 16:04:15 NASserver kernel: common_interrupt+0x119/0x12e Jan 1 16:04:15 NASserver kernel: asm_common_interrupt+0x1e/0x40 Jan 1 16:04:15 NASserver kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8 Jan 1 16:04:15 NASserver kernel: Code: 00 48 83 c4 28 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 55 8b af 28 04 00 00 b8 01 00 00 00 45 31 c9 53 45 31 d2 39 c5 Jan 1 16:04:15 NASserver kernel: RSP: 0018:ffffc900000f7ea0 EFLAGS: 00000246 Jan 1 16:04:15 NASserver kernel: RAX: ffff88884dd62300 RBX: 0000000000000001 RCX: 000000000000001f Jan 1 16:04:15 NASserver kernel: RDX: 0000000000000000 RSI: 00000000238e3989 RDI: 0000000000000000 Jan 1 16:04:15 NASserver kernel: RBP: ffff88884dd6ab00 R08: 00000456cfa37b44 R09: 000000000000002c Jan 1 16:04:15 NASserver kernel: R10: 000000007fffffff R11: 071c71c71c71c71c R12: 00000456cfa37b44 Jan 1 16:04:15 NASserver kernel: R13: ffffffff820c7e80 R14: 0000000000000001 R15: 0000000000000000 Jan 1 16:04:15 NASserver kernel: cpuidle_enter_state+0x101/0x1c4 Jan 1 16:04:15 NASserver kernel: cpuidle_enter+0x25/0x31 Jan 1 16:04:15 NASserver kernel: do_idle+0x1a1/0x20f Jan 1 16:04:15 NASserver kernel: cpu_startup_entry+0x18/0x1a Jan 1 16:04:15 NASserver kernel: secondary_startup_64_no_verify+0xb0/0xbb Jan 1 16:04:15 NASserver kernel: ---[ end trace 2fdb053ccd479b99 ]--- Jan 1 03:53:10 NASserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU Jan 1 03:53:10 NASserver kernel: rcu: #0113-....: (59999 ticks this GP) idle=ac6/1/0x4000000000000000 softirq=116935564/116935564 fqs=13901 Jan 1 03:53:10 NASserver kernel: #011(t=60000 jiffies g=110201057 q=3308366) Jan 1 03:53:10 NASserver kernel: NMI backtrace for cpu 3 Jan 1 03:53:10 NASserver kernel: CPU: 3 PID: 32728 Comm: python3 Tainted: G S W 5.10.1-Unraid #1 Jan 1 03:53:10 NASserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B17/MPG Z390 GAMING EDGE AC (MS-7B17), BIOS A.B0 10/12/2020 Jan 1 03:53:10 NASserver kernel: Call Trace: Jan 1 03:53:10 NASserver kernel: <IRQ> Jan 1 03:53:10 NASserver kernel: dump_stack+0x6b/0x83 Jan 1 03:53:10 NASserver kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Jan 1 03:53:10 NASserver kernel: nmi_cpu_backtrace+0x7d/0x8f Jan 1 03:53:10 NASserver kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3 Jan 1 03:53:10 NASserver kernel: rcu_dump_cpu_stacks+0x9f/0xc6 Jan 1 03:53:10 NASserver kernel: rcu_sched_clock_irq+0x1ec/0x543 Jan 1 03:53:10 NASserver kernel: ? trigger_load_balance+0x5a/0x1ca Jan 1 03:53:10 NASserver kernel: update_process_times+0x50/0x6e Jan 1 03:53:10 NASserver kernel: tick_sched_timer+0x36/0x64 Jan 1 03:53:10 NASserver kernel: __hrtimer_run_queues+0xb7/0x10b Jan 1 03:53:10 NASserver kernel: ? tick_sched_do_timer+0x39/0x39 Jan 1 03:53:10 NASserver kernel: hrtimer_interrupt+0x8d/0x160 Jan 1 03:53:10 NASserver kernel: __sysvec_apic_timer_interrupt+0x5d/0x68 Jan 1 03:53:10 NASserver kernel: sysvec_apic_timer_interrupt+0x82/0x95 Jan 1 03:53:10 NASserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20 Jan 1 03:53:10 NASserver kernel: RIP: 0010:nf_nat_setup_info+0x11c/0x652 [nf_nat] Jan 1 03:53:10 NASserver kernel: Code: 00 00 00 4c 89 ee 4c 89 ff e8 79 f9 ff ff 48 8b 15 c9 5a 00 00 89 c0 48 8d 04 c2 48 8b 10 48 85 d2 74 80 48 81 ea 98 00 00 00 <48> 85 d2 0f 84 70 ff ff ff 8a 44 24 46 38 42 46 74 09 48 8b 92 98 Jan 1 03:53:10 NASserver kernel: RSP: 0018:ffffc90000194700 EFLAGS: 00000282 Jan 1 03:53:10 NASserver kernel: RAX: ffff88820b7e2806 RBX: ffff88822d69db80 RCX: 0000000000000000 Jan 1 03:53:10 NASserver kernel: RDX: ffff8883c0c94640 RSI: 00000000658cf1b5 RDI: ffffc90000194720 Jan 1 03:53:10 NASserver kernel: RBP: ffffc900001947c8 R08: 00000000fd2066c5 R09: ffff8882081d5440 Jan 1 03:53:10 NASserver kernel: R10: 0000000000000348 R11: ffffffff815fd19e R12: 0000000000000000 Jan 1 03:53:10 NASserver kernel: R13: ffffc90000194720 R14: ffffc900001947dc R15: ffffffff8210da40 Jan 1 03:53:10 NASserver kernel: ? __ip_finish_output+0x142/0x142 Jan 1 03:53:10 NASserver kernel: ? _raw_read_unlock_irqrestore+0xd/0xe Jan 1 03:53:10 NASserver kernel: ? ep_poll_callback+0x141/0x1a0 Jan 1 03:53:10 NASserver kernel: ? __wake_up_common+0xa7/0x12f Jan 1 03:53:10 NASserver kernel: ? __ksize+0x15/0x64 Jan 1 03:53:10 NASserver kernel: ? krealloc+0x26/0x7a Jan 1 03:53:10 NASserver kernel: nf_nat_masquerade_ipv4+0x10a/0x130 [nf_nat] Jan 1 03:53:10 NASserver kernel: masquerade_tg+0x44/0x5e [xt_MASQUERADE] Jan 1 03:53:10 NASserver kernel: ipt_do_table+0x51a/0x5c0 [ip_tables] Jan 1 03:53:10 NASserver kernel: ? fib_validate_source+0xb0/0xda Jan 1 03:53:10 NASserver kernel: nf_nat_inet_fn+0xe9/0x182 [nf_nat] Jan 1 03:53:10 NASserver kernel: nf_nat_ipv4_out+0xf/0x88 [nf_nat] Jan 1 03:53:10 NASserver kernel: nf_hook_slow+0x39/0x8e Jan 1 03:53:10 NASserver kernel: nf_hook+0xab/0xd3 Jan 1 03:53:10 NASserver kernel: ? __ip_finish_output+0x142/0x142 Jan 1 03:53:10 NASserver kernel: ip_output+0x7d/0x8a Jan 1 03:53:10 NASserver kernel: ? __ip_finish_output+0x142/0x142 Jan 1 03:53:10 NASserver kernel: ip_forward+0x3f1/0x420 Jan 1 03:53:10 NASserver kernel: ? ip_check_defrag+0x18f/0x18f Jan 1 03:53:10 NASserver kernel: ip_sabotage_in+0x43/0x4d Jan 1 03:53:10 NASserver kernel: nf_hook_slow+0x39/0x8e Jan 1 03:53:10 NASserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 1 03:53:10 NASserver kernel: ? l3mdev_l3_rcv.constprop.0+0x50/0x50 Jan 1 03:53:10 NASserver kernel: ip_rcv+0x41/0x61 Jan 1 03:53:10 NASserver kernel: __netif_receive_skb_one_core+0x74/0x95 Jan 1 03:53:10 NASserver kernel: netif_receive_skb+0x79/0xa1 Jan 1 03:53:10 NASserver kernel: br_handle_frame_finish+0x30d/0x351 Jan 1 03:53:10 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 03:53:10 NASserver kernel: br_nf_hook_thresh+0xa3/0xc3 Jan 1 03:53:10 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 03:53:10 NASserver kernel: br_nf_pre_routing_finish+0x23d/0x264 Jan 1 03:53:10 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 03:53:10 NASserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 1 03:53:10 NASserver kernel: ? nf_nat_ipv4_in+0x1e/0x4a [nf_nat] Jan 1 03:53:10 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 03:53:10 NASserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 1 03:53:10 NASserver kernel: NF_HOOK+0xd7/0xf7 Jan 1 03:53:10 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 03:53:10 NASserver kernel: br_nf_pre_routing+0x229/0x239 Jan 1 03:53:10 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 03:53:10 NASserver kernel: br_handle_frame+0x25e/0x2a6 Jan 1 03:53:10 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 03:53:10 NASserver kernel: __netif_receive_skb_core+0x335/0x4e7 Jan 1 03:53:10 NASserver kernel: ? memcg_slab_free_hook+0x8a/0xfc Jan 1 03:53:10 NASserver kernel: __netif_receive_skb_one_core+0x3d/0x95 Jan 1 03:53:10 NASserver kernel: process_backlog+0xa3/0x13b Jan 1 03:53:10 NASserver kernel: net_rx_action+0xf4/0x29d Jan 1 03:53:10 NASserver kernel: __do_softirq+0xc4/0x1c2 Jan 1 03:53:10 NASserver kernel: asm_call_irq_on_stack+0xf/0x20 Jan 1 03:53:10 NASserver kernel: </IRQ> Jan 1 03:53:10 NASserver kernel: do_softirq_own_stack+0x2c/0x39 Jan 1 03:53:10 NASserver kernel: do_softirq+0x3a/0x44 Jan 1 03:53:10 NASserver kernel: __local_bh_enable_ip+0x3b/0x43 Jan 1 03:53:10 NASserver kernel: ip_finish_output2+0x2ec/0x31f Jan 1 03:53:10 NASserver kernel: ? ipv4_mtu+0x3d/0x64 Jan 1 03:53:10 NASserver kernel: __ip_queue_xmit+0x2a3/0x2df Jan 1 03:53:10 NASserver kernel: __tcp_transmit_skb+0x845/0x8ba Jan 1 03:53:10 NASserver kernel: tcp_connect+0x76d/0x7f4 Jan 1 03:53:10 NASserver kernel: tcp_v4_connect+0x3fc/0x455 Jan 1 03:53:10 NASserver kernel: __inet_stream_connect+0xd3/0x2b6 Jan 1 03:53:10 NASserver kernel: inet_stream_connect+0x34/0x49 Jan 1 03:53:10 NASserver kernel: __sys_connect+0x62/0x9d Jan 1 03:53:10 NASserver kernel: __x64_sys_connect+0x11/0x14 Jan 1 03:53:10 NASserver kernel: do_syscall_64+0x5d/0x6a Jan 1 03:53:10 NASserver kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Jan 1 03:53:10 NASserver kernel: RIP: 0033:0x149b8bf963db Jan 1 03:53:10 NASserver kernel: Code: 83 ec 18 89 54 24 0c 48 89 34 24 89 7c 24 08 e8 cb 44 f8 ff 8b 54 24 0c 48 8b 34 24 41 89 c0 8b 7c 24 08 b8 2a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 08 e8 21 45 f8 ff 8b 44 Jan 1 03:53:10 NASserver kernel: RSP: 002b:0000149b84d9c9f0 EFLAGS: 00000293 ORIG_RAX: 000000000000002a Jan 1 03:53:10 NASserver kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000149b8bf963db Jan 1 03:53:10 NASserver kernel: RDX: 0000000000000010 RSI: 0000149b84d9ca60 RDI: 000000000000001d Jan 1 03:53:10 NASserver kernel: RBP: 0000149b440e9880 R08: 0000000000000000 R09: 0000000000000012 Jan 1 03:53:10 NASserver kernel: R10: 0000149b3f91622d R11: 0000000000000293 R12: 0000149b84d9ca60 Jan 1 03:53:10 NASserver kernel: R13: 0000149b6c000db0 R14: 0000000000000010 R15: 0000149b8bc45690 Jan 1 04:04:17 NASserver kernel: clocksource: timekeeping watchdog on CPU3: Marking clocksource 'tsc' as unstable because the skew is too large: Jan 1 04:04:17 NASserver kernel: clocksource: 'hpet' wd_now: 58d58c8 wd_last: f605952c mask: ffffffff Jan 1 04:04:17 NASserver kernel: clocksource: 'tsc' cs_now: 18431d7cb85b8 cs_last: 181d0be3ee3a0 mask: ffffffffffffffff Jan 1 04:04:17 NASserver kernel: tsc: Marking TSC unstable due to clocksource watchdog nasserver-diagnostics-20210103-0229.zip
  18. So trying out the RC branch for the first time and installed RC5. Went to setup wiregaurd with the Dynamix plugin. Ran into a show stopper up front. I use a DDNS with a .network FQDN. When I entered my .network domain in the local endpoint input box it generates an error that my .network domain is not a true FQDN. I assume someone just hard coded the vanilla .com / .net / .org in the error checking for that field. Can we get that fixed?
  19. So trying out the RC branch for the first time. Went to setup wiregaurd with the Dynamix plugin. Ran into a show stopper up front. I use a DDNS with a .network FQDN. When I entered my .network domain in the local endpoint input box it generates an error that my .network domain is not a true FQDN. I assume someone just hard coded the vanilla .com / .net / .org in the error checking for that field. Can we get that fixed?
  20. I found that earlier. However, it does not have an example of the actual string that needs to be used in the field. I understand I need to mount it as a slave. I'm looking for the actual command to use.
  21. Can someone give me an example of what to put in the host 5 field in place of /mnt/disks? The mount path I'm using is /mnt/disks/SSD . I'm wrestling with the same error.
  22. Thanks for the reply. I have been working on this and running into a few errors. I'm trying to empty the md6 drive (the one with the BTRFS errors) to be able to reformat. The metadata problem is so bad it fails on the copy so I can't empty the drive. I have two copies of the data on other servers. I just want to reformat the drive and sync the data back. How do I reformat and not have a rebuild on a drive I can't empty?
  23. nasserver-diagnostics-20190115-1127.zip As requested
  24. Keep getting recurring errors in my system log: Jan 15 10:06:19 NASserver kernel: BTRFS critical (device md6): corrupt leaf, slot offset bad: block=1187753492480, root=1, slot=159 Jan 15 10:06:19 NASserver kernel: BTRFS critical (device md6): corrupt leaf, slot offset bad: block=1187753492480, root=1, slot=159 And then this one repeats every 10 seconds: Jan 15 10:53:36 NASserver kernel: pcieport 0000:00:03.2: AER: Corrected error received: id=0000 Jan 15 10:53:36 NASserver kernel: pcieport 0000:00:03.2: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=001a(Transmitter ID) Jan 15 10:53:36 NASserver kernel: pcieport 0000:00:03.2: device [1022:1453] error status/mask=00001000/00006000 Jan 15 10:53:36 NASserver kernel: pcieport 0000:00:03.2: [12] Replay Timer Timeout Not sure if I have had a flaky disk (md6) and that has caused the stability issues. Need help in interpreting what is happening with the system. Before I noticed the errors in the log this morning I was attempting to upgrade the system to 6.6.6 last night. When I upgraded to 6.6.6 last night, the system stopped disk 6/md6 in the middle of the night and make it read-only and it really screwed up the system. Backed off the update this morning to 6.5.3 and it is all stable again. This is not the first issue with this system and the 6.6.x update. Previous upgrades to 6.6.x can't stay up without the shares disappearing after 24 hours of uptime. When on 6.5.x months of stable uptime.