gdeyoung Posted January 3, 2021 Share Posted January 3, 2021 (edited) Having one of my servers crash daily. Any ideas on what is generating the docker networking issues below? I see them constantly in the logs then the server will freeze or throw a kernel error and crash. I will see docker0: port 2-7 in the errors. The server is a Intel 9700K with a Intel 10gb NIC. The NIC has been replaced and the SFP+ DAC cable and still get the error. I have swapped most of the hardware out to check for hardware fault and no luck (New proc/MB/PSU/RAID card). I have also moved data off each drive and rebuilt the partition on each drive and moved data back to account for drive/filesystem issues. I have also rebuilt the docker.img and the docker appdata several times and moved them around to different cache and unassigned drive SSD's. Any help would be appreciated. Seeing this in my logs: Jan 2 09:31:13 NASserver kernel: veth2251ce3: renamed from eth0 Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth639656d) entered disabled state Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth639656d) entered disabled state Jan 2 09:31:13 NASserver kernel: device veth639656d left promiscuous mode Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth639656d) entered disabled state Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth6a994b4) entered blocking state Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth6a994b4) entered disabled state Jan 2 09:31:13 NASserver kernel: device veth6a994b4 entered promiscuous mode Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth6a994b4) entered blocking state Jan 2 09:31:13 NASserver kernel: docker0: port 7(veth6a994b4) entered forwarding state Jan 2 09:31:13 NASserver kernel: eth0: renamed from veth7f12a06 Jan 2 09:31:13 NASserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth6a994b4: link becomes ready Jan 2 09:34:39 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 2 10:33:46 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 2 10:41:36 NASserver kernel: veth7f12a06: renamed from eth0 Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth6a994b4) entered disabled state Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth6a994b4) entered disabled state Jan 2 10:41:36 NASserver kernel: device veth6a994b4 left promiscuous mode Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth6a994b4) entered disabled state Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth7a586fa) entered blocking state Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth7a586fa) entered disabled state Jan 2 10:41:36 NASserver kernel: device veth7a586fa entered promiscuous mode Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth7a586fa) entered blocking state Jan 2 10:41:36 NASserver kernel: docker0: port 7(veth7a586fa) entered forwarding state Jan 2 10:41:36 NASserver kernel: eth0: renamed from veth1986f4c Jan 2 10:41:36 NASserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth7a586fa: link becomes ready Jan 2 10:56:52 NASserver kernel: veth1986f4c: renamed from eth0 Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth7a586fa) entered disabled state Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth7a586fa) entered disabled state Jan 2 10:56:52 NASserver kernel: device veth7a586fa left promiscuous mode Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth7a586fa) entered disabled state Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth0af469d) entered blocking state Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth0af469d) entered disabled state Jan 2 10:56:52 NASserver kernel: device veth0af469d entered promiscuous mode Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth0af469d) entered blocking state Jan 2 10:56:52 NASserver kernel: docker0: port 7(veth0af469d) entered forwarding state Jan 2 10:56:53 NASserver kernel: eth0: renamed from veth2a96f02 Jan 2 10:56:53 NASserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth0af469d: link becomes ready Jan 2 11:33:53 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 2 12:07:12 NASserver kernel: veth2a96f02: renamed from eth0 Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth0af469d) entered disabled state Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth0af469d) entered disabled state Jan 2 12:07:12 NASserver kernel: device veth0af469d left promiscuous mode Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth0af469d) entered disabled state Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth2629056) entered blocking state Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth2629056) entered disabled state Jan 2 12:07:12 NASserver kernel: device veth2629056 entered promiscuous mode Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth2629056) entered blocking state Jan 2 12:07:12 NASserver kernel: docker0: port 7(veth2629056) entered forwarding state Jan 2 12:07:12 NASserver kernel: eth0: renamed from vethd0b0abb Jan 2 12:07:12 NASserver kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth2629056: link becomes ready Jan 2 12:33:59 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update Jan 2 13:34:06 NASserver flash_backup: adding task: php /usr/local/emhttp/plugins/dynamix.unraid.net/include/UpdateFlashBackup.php update When the kernel crashes I get a log below: Jan 1 16:04:15 NASserver kernel: ------------[ cut here ]------------ Jan 1 16:04:15 NASserver kernel: WARNING: CPU: 5 PID: 0 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x99/0x1e1 Jan 1 16:04:15 NASserver kernel: Modules linked in: macvlan tun xt_nat veth xt_MASQUERADE iptable_filter iptable_nat nf_nat ip_tables xfs md_mod i915 iosf_mbi i2c_algo_bit drm_kms_helper drm intel_gtt agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha bonding ixgbe mdio x86_pkg_temp_thermal wmi_bmof intel_powerclamp mxm_wmi intel_wmi_thunderbolt coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd btusb glue_helper btrtl rapl btbcm btintel intel_cstate bluetooth intel_uncore nvme ahci i2c_i801 mpt3sas i2c_smbus i2c_core nvme_core ecdh_generic libahci raid_class ecc scsi_transport_sas video thermal backlight fan wmi button acpi_pad [last unloaded: mdio] Jan 1 16:04:15 NASserver kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: G S 5.10.1-Unraid #1 Jan 1 16:04:15 NASserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B17/MPG Z390 GAMING EDGE AC (MS-7B17), BIOS A.B0 10/12/2020 Jan 1 16:04:15 NASserver kernel: RIP: 0010:__nf_conntrack_confirm+0x99/0x1e1 Jan 1 16:04:15 NASserver kernel: Code: e4 e3 ff ff 8b 54 24 14 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 54 e1 ff ff 84 c0 75 b8 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 89 de ff ff e8 af e0 ff ff e9 1f 01 Jan 1 16:04:15 NASserver kernel: RSP: 0018:ffffc900001ec898 EFLAGS: 00010202 Jan 1 16:04:15 NASserver kernel: RAX: 0000000000000188 RBX: 0000000000004ff2 RCX: 0000000038544019 Jan 1 16:04:15 NASserver kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff8200a808 Jan 1 16:04:15 NASserver kernel: RBP: ffff88815d07c640 R08: 00000000de3985d0 R09: ffff88815ccb2bc0 Jan 1 16:04:15 NASserver kernel: R10: 0000000000000158 R11: ffff8881061fda00 R12: 0000000000004824 Jan 1 16:04:15 NASserver kernel: R13: ffffffff8210da40 R14: 0000000000004ff2 R15: ffff88815d07c64c Jan 1 16:04:15 NASserver kernel: FS: 0000000000000000(0000) GS:ffff88884dd40000(0000) knlGS:0000000000000000 Jan 1 16:04:15 NASserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 1 16:04:15 NASserver kernel: CR2: 0000153736b90f20 CR3: 0000000169908001 CR4: 00000000003706e0 Jan 1 16:04:15 NASserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 1 16:04:15 NASserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Jan 1 16:04:15 NASserver kernel: Call Trace: Jan 1 16:04:15 NASserver kernel: <IRQ> Jan 1 16:04:15 NASserver kernel: nf_conntrack_confirm+0x2f/0x36 Jan 1 16:04:15 NASserver kernel: nf_hook_slow+0x39/0x8e Jan 1 16:04:15 NASserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 1 16:04:15 NASserver kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe Jan 1 16:04:15 NASserver kernel: ip_local_deliver+0x49/0x75 Jan 1 16:04:15 NASserver kernel: ip_sabotage_in+0x43/0x4d Jan 1 16:04:15 NASserver kernel: nf_hook_slow+0x39/0x8e Jan 1 16:04:15 NASserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 1 16:04:15 NASserver kernel: ? l3mdev_l3_rcv.constprop.0+0x50/0x50 Jan 1 16:04:15 NASserver kernel: ip_rcv+0x41/0x61 Jan 1 16:04:15 NASserver kernel: __netif_receive_skb_one_core+0x74/0x95 Jan 1 16:04:15 NASserver kernel: netif_receive_skb+0x79/0xa1 Jan 1 16:04:15 NASserver kernel: br_handle_frame_finish+0x30d/0x351 Jan 1 16:04:15 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 16:04:15 NASserver kernel: br_nf_hook_thresh+0xa3/0xc3 Jan 1 16:04:15 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 16:04:15 NASserver kernel: br_nf_pre_routing_finish+0x23d/0x264 Jan 1 16:04:15 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 16:04:15 NASserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 1 16:04:15 NASserver kernel: ? nf_nat_ipv4_in+0x1e/0x4a [nf_nat] Jan 1 16:04:15 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 16:04:15 NASserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 1 16:04:15 NASserver kernel: NF_HOOK+0xd7/0xf7 Jan 1 16:04:15 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 16:04:15 NASserver kernel: br_nf_pre_routing+0x229/0x239 Jan 1 16:04:15 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 16:04:15 NASserver kernel: br_handle_frame+0x25e/0x2a6 Jan 1 16:04:15 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 16:04:15 NASserver kernel: __netif_receive_skb_core+0x335/0x4e7 Jan 1 16:04:15 NASserver kernel: __netif_receive_skb_list_core+0x78/0x104 Jan 1 16:04:15 NASserver kernel: netif_receive_skb_list_internal+0x1bf/0x1f2 Jan 1 16:04:15 NASserver kernel: ? dev_gro_receive+0x55d/0x578 Jan 1 16:04:15 NASserver kernel: gro_normal_list+0x1d/0x39 Jan 1 16:04:15 NASserver kernel: napi_complete_done+0x79/0x104 Jan 1 16:04:15 NASserver kernel: ixgbe_poll+0xc95/0xd4f [ixgbe] Jan 1 16:04:15 NASserver kernel: net_rx_action+0xf4/0x29d Jan 1 16:04:15 NASserver kernel: __do_softirq+0xc4/0x1c2 Jan 1 16:04:15 NASserver kernel: asm_call_irq_on_stack+0xf/0x20 Jan 1 16:04:15 NASserver kernel: </IRQ> Jan 1 16:04:15 NASserver kernel: do_softirq_own_stack+0x2c/0x39 Jan 1 16:04:15 NASserver kernel: __irq_exit_rcu+0x45/0x80 Jan 1 16:04:15 NASserver kernel: common_interrupt+0x119/0x12e Jan 1 16:04:15 NASserver kernel: asm_common_interrupt+0x1e/0x40 Jan 1 16:04:15 NASserver kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8 Jan 1 16:04:15 NASserver kernel: Code: 00 48 83 c4 28 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 55 8b af 28 04 00 00 b8 01 00 00 00 45 31 c9 53 45 31 d2 39 c5 Jan 1 16:04:15 NASserver kernel: RSP: 0018:ffffc900000f7ea0 EFLAGS: 00000246 Jan 1 16:04:15 NASserver kernel: RAX: ffff88884dd62300 RBX: 0000000000000001 RCX: 000000000000001f Jan 1 16:04:15 NASserver kernel: RDX: 0000000000000000 RSI: 00000000238e3989 RDI: 0000000000000000 Jan 1 16:04:15 NASserver kernel: RBP: ffff88884dd6ab00 R08: 00000456cfa37b44 R09: 000000000000002c Jan 1 16:04:15 NASserver kernel: R10: 000000007fffffff R11: 071c71c71c71c71c R12: 00000456cfa37b44 Jan 1 16:04:15 NASserver kernel: R13: ffffffff820c7e80 R14: 0000000000000001 R15: 0000000000000000 Jan 1 16:04:15 NASserver kernel: cpuidle_enter_state+0x101/0x1c4 Jan 1 16:04:15 NASserver kernel: cpuidle_enter+0x25/0x31 Jan 1 16:04:15 NASserver kernel: do_idle+0x1a1/0x20f Jan 1 16:04:15 NASserver kernel: cpu_startup_entry+0x18/0x1a Jan 1 16:04:15 NASserver kernel: secondary_startup_64_no_verify+0xb0/0xbb Jan 1 16:04:15 NASserver kernel: ---[ end trace 2fdb053ccd479b99 ]--- Jan 1 03:53:10 NASserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU Jan 1 03:53:10 NASserver kernel: rcu: #0113-....: (59999 ticks this GP) idle=ac6/1/0x4000000000000000 softirq=116935564/116935564 fqs=13901 Jan 1 03:53:10 NASserver kernel: #011(t=60000 jiffies g=110201057 q=3308366) Jan 1 03:53:10 NASserver kernel: NMI backtrace for cpu 3 Jan 1 03:53:10 NASserver kernel: CPU: 3 PID: 32728 Comm: python3 Tainted: G S W 5.10.1-Unraid #1 Jan 1 03:53:10 NASserver kernel: Hardware name: Micro-Star International Co., Ltd. MS-7B17/MPG Z390 GAMING EDGE AC (MS-7B17), BIOS A.B0 10/12/2020 Jan 1 03:53:10 NASserver kernel: Call Trace: Jan 1 03:53:10 NASserver kernel: <IRQ> Jan 1 03:53:10 NASserver kernel: dump_stack+0x6b/0x83 Jan 1 03:53:10 NASserver kernel: ? lapic_can_unplug_cpu+0x8e/0x8e Jan 1 03:53:10 NASserver kernel: nmi_cpu_backtrace+0x7d/0x8f Jan 1 03:53:10 NASserver kernel: nmi_trigger_cpumask_backtrace+0x56/0xd3 Jan 1 03:53:10 NASserver kernel: rcu_dump_cpu_stacks+0x9f/0xc6 Jan 1 03:53:10 NASserver kernel: rcu_sched_clock_irq+0x1ec/0x543 Jan 1 03:53:10 NASserver kernel: ? trigger_load_balance+0x5a/0x1ca Jan 1 03:53:10 NASserver kernel: update_process_times+0x50/0x6e Jan 1 03:53:10 NASserver kernel: tick_sched_timer+0x36/0x64 Jan 1 03:53:10 NASserver kernel: __hrtimer_run_queues+0xb7/0x10b Jan 1 03:53:10 NASserver kernel: ? tick_sched_do_timer+0x39/0x39 Jan 1 03:53:10 NASserver kernel: hrtimer_interrupt+0x8d/0x160 Jan 1 03:53:10 NASserver kernel: __sysvec_apic_timer_interrupt+0x5d/0x68 Jan 1 03:53:10 NASserver kernel: sysvec_apic_timer_interrupt+0x82/0x95 Jan 1 03:53:10 NASserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20 Jan 1 03:53:10 NASserver kernel: RIP: 0010:nf_nat_setup_info+0x11c/0x652 [nf_nat] Jan 1 03:53:10 NASserver kernel: Code: 00 00 00 4c 89 ee 4c 89 ff e8 79 f9 ff ff 48 8b 15 c9 5a 00 00 89 c0 48 8d 04 c2 48 8b 10 48 85 d2 74 80 48 81 ea 98 00 00 00 <48> 85 d2 0f 84 70 ff ff ff 8a 44 24 46 38 42 46 74 09 48 8b 92 98 Jan 1 03:53:10 NASserver kernel: RSP: 0018:ffffc90000194700 EFLAGS: 00000282 Jan 1 03:53:10 NASserver kernel: RAX: ffff88820b7e2806 RBX: ffff88822d69db80 RCX: 0000000000000000 Jan 1 03:53:10 NASserver kernel: RDX: ffff8883c0c94640 RSI: 00000000658cf1b5 RDI: ffffc90000194720 Jan 1 03:53:10 NASserver kernel: RBP: ffffc900001947c8 R08: 00000000fd2066c5 R09: ffff8882081d5440 Jan 1 03:53:10 NASserver kernel: R10: 0000000000000348 R11: ffffffff815fd19e R12: 0000000000000000 Jan 1 03:53:10 NASserver kernel: R13: ffffc90000194720 R14: ffffc900001947dc R15: ffffffff8210da40 Jan 1 03:53:10 NASserver kernel: ? __ip_finish_output+0x142/0x142 Jan 1 03:53:10 NASserver kernel: ? _raw_read_unlock_irqrestore+0xd/0xe Jan 1 03:53:10 NASserver kernel: ? ep_poll_callback+0x141/0x1a0 Jan 1 03:53:10 NASserver kernel: ? __wake_up_common+0xa7/0x12f Jan 1 03:53:10 NASserver kernel: ? __ksize+0x15/0x64 Jan 1 03:53:10 NASserver kernel: ? krealloc+0x26/0x7a Jan 1 03:53:10 NASserver kernel: nf_nat_masquerade_ipv4+0x10a/0x130 [nf_nat] Jan 1 03:53:10 NASserver kernel: masquerade_tg+0x44/0x5e [xt_MASQUERADE] Jan 1 03:53:10 NASserver kernel: ipt_do_table+0x51a/0x5c0 [ip_tables] Jan 1 03:53:10 NASserver kernel: ? fib_validate_source+0xb0/0xda Jan 1 03:53:10 NASserver kernel: nf_nat_inet_fn+0xe9/0x182 [nf_nat] Jan 1 03:53:10 NASserver kernel: nf_nat_ipv4_out+0xf/0x88 [nf_nat] Jan 1 03:53:10 NASserver kernel: nf_hook_slow+0x39/0x8e Jan 1 03:53:10 NASserver kernel: nf_hook+0xab/0xd3 Jan 1 03:53:10 NASserver kernel: ? __ip_finish_output+0x142/0x142 Jan 1 03:53:10 NASserver kernel: ip_output+0x7d/0x8a Jan 1 03:53:10 NASserver kernel: ? __ip_finish_output+0x142/0x142 Jan 1 03:53:10 NASserver kernel: ip_forward+0x3f1/0x420 Jan 1 03:53:10 NASserver kernel: ? ip_check_defrag+0x18f/0x18f Jan 1 03:53:10 NASserver kernel: ip_sabotage_in+0x43/0x4d Jan 1 03:53:10 NASserver kernel: nf_hook_slow+0x39/0x8e Jan 1 03:53:10 NASserver kernel: nf_hook.constprop.0+0xb1/0xd8 Jan 1 03:53:10 NASserver kernel: ? l3mdev_l3_rcv.constprop.0+0x50/0x50 Jan 1 03:53:10 NASserver kernel: ip_rcv+0x41/0x61 Jan 1 03:53:10 NASserver kernel: __netif_receive_skb_one_core+0x74/0x95 Jan 1 03:53:10 NASserver kernel: netif_receive_skb+0x79/0xa1 Jan 1 03:53:10 NASserver kernel: br_handle_frame_finish+0x30d/0x351 Jan 1 03:53:10 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 03:53:10 NASserver kernel: br_nf_hook_thresh+0xa3/0xc3 Jan 1 03:53:10 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 03:53:10 NASserver kernel: br_nf_pre_routing_finish+0x23d/0x264 Jan 1 03:53:10 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 03:53:10 NASserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 1 03:53:10 NASserver kernel: ? nf_nat_ipv4_in+0x1e/0x4a [nf_nat] Jan 1 03:53:10 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 03:53:10 NASserver kernel: ? br_handle_frame_finish+0x351/0x351 Jan 1 03:53:10 NASserver kernel: NF_HOOK+0xd7/0xf7 Jan 1 03:53:10 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 03:53:10 NASserver kernel: br_nf_pre_routing+0x229/0x239 Jan 1 03:53:10 NASserver kernel: ? br_nf_forward_finish+0xd0/0xd0 Jan 1 03:53:10 NASserver kernel: br_handle_frame+0x25e/0x2a6 Jan 1 03:53:10 NASserver kernel: ? br_pass_frame_up+0xda/0xda Jan 1 03:53:10 NASserver kernel: __netif_receive_skb_core+0x335/0x4e7 Jan 1 03:53:10 NASserver kernel: ? memcg_slab_free_hook+0x8a/0xfc Jan 1 03:53:10 NASserver kernel: __netif_receive_skb_one_core+0x3d/0x95 Jan 1 03:53:10 NASserver kernel: process_backlog+0xa3/0x13b Jan 1 03:53:10 NASserver kernel: net_rx_action+0xf4/0x29d Jan 1 03:53:10 NASserver kernel: __do_softirq+0xc4/0x1c2 Jan 1 03:53:10 NASserver kernel: asm_call_irq_on_stack+0xf/0x20 Jan 1 03:53:10 NASserver kernel: </IRQ> Jan 1 03:53:10 NASserver kernel: do_softirq_own_stack+0x2c/0x39 Jan 1 03:53:10 NASserver kernel: do_softirq+0x3a/0x44 Jan 1 03:53:10 NASserver kernel: __local_bh_enable_ip+0x3b/0x43 Jan 1 03:53:10 NASserver kernel: ip_finish_output2+0x2ec/0x31f Jan 1 03:53:10 NASserver kernel: ? ipv4_mtu+0x3d/0x64 Jan 1 03:53:10 NASserver kernel: __ip_queue_xmit+0x2a3/0x2df Jan 1 03:53:10 NASserver kernel: __tcp_transmit_skb+0x845/0x8ba Jan 1 03:53:10 NASserver kernel: tcp_connect+0x76d/0x7f4 Jan 1 03:53:10 NASserver kernel: tcp_v4_connect+0x3fc/0x455 Jan 1 03:53:10 NASserver kernel: __inet_stream_connect+0xd3/0x2b6 Jan 1 03:53:10 NASserver kernel: inet_stream_connect+0x34/0x49 Jan 1 03:53:10 NASserver kernel: __sys_connect+0x62/0x9d Jan 1 03:53:10 NASserver kernel: __x64_sys_connect+0x11/0x14 Jan 1 03:53:10 NASserver kernel: do_syscall_64+0x5d/0x6a Jan 1 03:53:10 NASserver kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9 Jan 1 03:53:10 NASserver kernel: RIP: 0033:0x149b8bf963db Jan 1 03:53:10 NASserver kernel: Code: 83 ec 18 89 54 24 0c 48 89 34 24 89 7c 24 08 e8 cb 44 f8 ff 8b 54 24 0c 48 8b 34 24 41 89 c0 8b 7c 24 08 b8 2a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 08 e8 21 45 f8 ff 8b 44 Jan 1 03:53:10 NASserver kernel: RSP: 002b:0000149b84d9c9f0 EFLAGS: 00000293 ORIG_RAX: 000000000000002a Jan 1 03:53:10 NASserver kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000149b8bf963db Jan 1 03:53:10 NASserver kernel: RDX: 0000000000000010 RSI: 0000149b84d9ca60 RDI: 000000000000001d Jan 1 03:53:10 NASserver kernel: RBP: 0000149b440e9880 R08: 0000000000000000 R09: 0000000000000012 Jan 1 03:53:10 NASserver kernel: R10: 0000149b3f91622d R11: 0000000000000293 R12: 0000149b84d9ca60 Jan 1 03:53:10 NASserver kernel: R13: 0000149b6c000db0 R14: 0000000000000010 R15: 0000149b8bc45690 Jan 1 04:04:17 NASserver kernel: clocksource: timekeeping watchdog on CPU3: Marking clocksource 'tsc' as unstable because the skew is too large: Jan 1 04:04:17 NASserver kernel: clocksource: 'hpet' wd_now: 58d58c8 wd_last: f605952c mask: ffffffff Jan 1 04:04:17 NASserver kernel: clocksource: 'tsc' cs_now: 18431d7cb85b8 cs_last: 181d0be3ee3a0 mask: ffffffffffffffff Jan 1 04:04:17 NASserver kernel: tsc: Marking TSC unstable due to clocksource watchdog nasserver-diagnostics-20210103-0229.zip Edited January 3, 2021 by gdeyoung add info Quote Link to comment
gdeyoung Posted January 6, 2021 Author Share Posted January 6, 2021 Bump - Anyone? Quote Link to comment
DataLoreQ Posted January 19, 2021 Share Posted January 19, 2021 (edited) Looks like I'm getting the same or a similar error on occasion. Mine seems to occur every 1-5 days. Diagnostics would not gather when I was able to log into the machine via IPMI or direct KVM and did a poweroff, and had to end up doing a hard shutdown. I've included the portion of the logs for when it started last night. Log File Snippet.txt Edited January 19, 2021 by DataLoreQ Quote Link to comment
JorgeB Posted January 19, 2021 Share Posted January 19, 2021 Problem appears to be network related, try to simplify network config as much as possible. Quote Link to comment
DataLoreQ Posted January 19, 2021 Share Posted January 19, 2021 I have the network cabling unplugged from my 1gbe connections, and it's all going through the 10gbe connection. I had tried to set it up where the 1gbe connections were the backup to the 10gbe, but data was still being transferred on the 1gbe connections rather than the 10gbe connection. Looks like I may need to remove my 10gbe card and go back to slower speeds... Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.