Help! Unraid is freezing randomly - Possible Solar Flare driver issue?

[Jase]

Hello, I'm hoping that someone could help me through this excerpt from a Kernel crash on my Unraid server. These system wide outages have been happening frequently over the past week and are getting worse. Today, the server was only up for about an half an hour before it froze. So I did this tail -f /var/log/syslog to catch the processes and found quite a run of alarming looking text which I have included in the below attachments. (Diagnostics and my Syslog capture)

 

I also noticed that under devices/IOMMU there are duplicates (126 times) of my Solarflare controller.

IOMMU group 229:                 [1924:1803] 02:1f.6 Ethernet controller: Solarflare Communications SFC9020 10G Ethernet Controller (Virtual Function)

 

I'm not sure what is going on but some and any guidance would be much appreciated.

 

Unraid version: 6.11.5

Solarflare: SFN5322F Dual port.

 

 

plexunraid-diagnostics-20221208-2025.zip Unraid crash logs with Solarflare .txt

Edited December 9, 2022 by Jase

[Jase]

And this just popped in...

 

Dec  8 21:23:23 PlexUnraid kernel: ------------[ cut here ]------------
Dec  8 21:23:23 PlexUnraid kernel: WARNING: CPU: 19 PID: 0 at net/netfilter/nf_conntrack_core.c:1208 __nf_conntrack_confirm+0xa5/0x2cb [nf_conntrack]
Dec  8 21:23:23 PlexUnraid kernel: Modules linked in: macvlan md_mod nvidia_uvm(PO) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs nct7904 watchdog efivarfs wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables 8021q garp mrp bridge stp llc sfc_siena mdio igb i2c_algo_bit nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) sr_mod cdrom x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mxm_wmi crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd drm_kms_helper rapl intel_cstate drm mpt3sas backlight i2c_i801 syscopyarea nvme ahci firewire_ohci sysfillrect raid_class i2c_smbus input_leds sysimgblt intel_uncore scsi_transport_sas fb_sys_fops firewire_core nvme_core led_class libahci
Dec  8 21:23:23 PlexUnraid kernel: i2c_core wmi button unix [last unloaded: md_mod]
Dec  8 21:23:23 PlexUnraid kernel: CPU: 19 PID: 0 Comm: swapper/19 Tainted: P           O      5.19.17-Unraid #2
Dec  8 21:23:23 PlexUnraid kernel: Hardware name: Supermicro X9DAi/X9DAi, BIOS 3.0 08/05/2013
Dec  8 21:23:23 PlexUnraid kernel: RIP: 0010:__nf_conntrack_confirm+0xa5/0x2cb [nf_conntrack]
Dec  8 21:23:23 PlexUnraid kernel: Code: c6 48 89 44 24 10 e8 dd e2 ff ff 8b 7c 24 04 89 da 89 c6 89 04 24 e8 56 e6 ff ff 84 c0 75 a2 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 16 de ff ff e8 2c e3 ff ff e9 7e 01
Dec  8 21:23:23 PlexUnraid kernel: RSP: 0018:ffffc900036e8838 EFLAGS: 00010202
Dec  8 21:23:23 PlexUnraid kernel: RAX: 0000000000000188 RBX: 0000000000000000 RCX: 14df2824b3d9c855
Dec  8 21:23:23 PlexUnraid kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffa027ced8
Dec  8 21:23:23 PlexUnraid kernel: RBP: ffff8881adc52c00 R08: c66792359254778b R09: f04e321243f93132
Dec  8 21:23:23 PlexUnraid kernel: R10: 98997827190921fc R11: 098391baf9ae4cd7 R12: ffffffff82909480
Dec  8 21:23:23 PlexUnraid kernel: R13: 0000000000031836 R14: ffff8881cec8dc00 R15: 0000000000000000
Dec  8 21:23:23 PlexUnraid kernel: FS:  0000000000000000(0000) GS:ffff88846fcc0000(0000) knlGS:0000000000000000
Dec  8 21:23:23 PlexUnraid kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  8 21:23:23 PlexUnraid kernel: CR2: 000014a8c6b08000 CR3: 000000049eefe002 CR4: 00000000000606e0
Dec  8 21:23:23 PlexUnraid kernel: Call Trace:
Dec  8 21:23:23 PlexUnraid kernel: <IRQ>
Dec  8 21:23:23 PlexUnraid kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
Dec  8 21:23:23 PlexUnraid kernel: nf_hook_slow+0x3d/0x96
Dec  8 21:23:23 PlexUnraid kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Dec  8 21:23:23 PlexUnraid kernel: NF_HOOK.constprop.0+0x79/0xd9
Dec  8 21:23:23 PlexUnraid kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Dec  8 21:23:23 PlexUnraid kernel: ip_sabotage_in+0x4a/0x58 [br_netfilter]
Dec  8 21:23:23 PlexUnraid kernel: nf_hook_slow+0x3d/0x96
Dec  8 21:23:23 PlexUnraid kernel: ? ip_rcv_finish_core.constprop.0+0x3b7/0x3b7
Dec  8 21:23:23 PlexUnraid kernel: NF_HOOK.constprop.0+0x79/0xd9
Dec  8 21:23:23 PlexUnraid kernel: ? ip_rcv_finish_core.constprop.0+0x3b7/0x3b7
Dec  8 21:23:23 PlexUnraid kernel: __netif_receive_skb_one_core+0x77/0x9c
Dec  8 21:23:23 PlexUnraid kernel: netif_receive_skb+0xbf/0x127
Dec  8 21:23:23 PlexUnraid kernel: br_handle_frame_finish+0x476/0x4b0 [bridge]
Dec  8 21:23:23 PlexUnraid kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Dec  8 21:23:23 PlexUnraid kernel: br_nf_hook_thresh+0xe5/0x109 [br_netfilter]
Dec  8 21:23:23 PlexUnraid kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Dec  8 21:23:23 PlexUnraid kernel: br_nf_pre_routing_finish+0x2c1/0x2ec [br_netfilter]
Dec  8 21:23:23 PlexUnraid kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Dec  8 21:23:23 PlexUnraid kernel: ? NF_HOOK.isra.0+0xe4/0x140 [br_netfilter]
Dec  8 21:23:23 PlexUnraid kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Dec  8 21:23:23 PlexUnraid kernel: br_nf_pre_routing+0x226/0x23a [br_netfilter]
Dec  8 21:23:23 PlexUnraid kernel: ? br_nf_hook_thresh+0x109/0x109 [br_netfilter]
Dec  8 21:23:23 PlexUnraid kernel: br_handle_frame+0x27f/0x2e7 [bridge]
Dec  8 21:23:23 PlexUnraid kernel: ? br_pass_frame_up+0xdd/0xdd [bridge]
Dec  8 21:23:23 PlexUnraid kernel: __netif_receive_skb_core.constprop.0+0x4f9/0x6e3
Dec  8 21:23:23 PlexUnraid kernel: __netif_receive_skb_list_core+0x8a/0x11e
Dec  8 21:23:23 PlexUnraid kernel: netif_receive_skb_list_internal+0x1d7/0x210
Dec  8 21:23:23 PlexUnraid kernel: gro_normal_list+0x1d/0x3f
Dec  8 21:23:23 PlexUnraid kernel: napi_complete_done+0x7b/0x11a
Dec  8 21:23:23 PlexUnraid kernel: igb_poll+0xdb3/0xfb9 [igb]
Dec  8 21:23:23 PlexUnraid kernel: __napi_poll.constprop.0+0x2b/0x124
Dec  8 21:23:23 PlexUnraid kernel: net_rx_action+0x159/0x24f
Dec  8 21:23:23 PlexUnraid kernel: __do_softirq+0x129/0x288
Dec  8 21:23:23 PlexUnraid kernel: __irq_exit_rcu+0x79/0xb8
Dec  8 21:23:23 PlexUnraid kernel: common_interrupt+0x9b/0xc1
Dec  8 21:23:23 PlexUnraid kernel: </IRQ>
Dec  8 21:23:23 PlexUnraid kernel: <TASK>
Dec  8 21:23:23 PlexUnraid kernel: asm_common_interrupt+0x22/0x40
Dec  8 21:23:23 PlexUnraid kernel: RIP: 0010:raw_spin_rq_unlock_irq+0xb/0x10
Dec  8 21:23:23 PlexUnraid kernel: Code: 89 e2 48 89 ee 48 8b 7b f0 e8 46 19 b7 00 48 83 3b 00 75 e4 5b 31 c0 5d 41 5c c3 cc cc cc cc e8 c1 aa 7a 00 fb 0f 1f 44 00 00 <c3> cc cc cc cc 83 bf 98 09 00 00 02 76 07 c7 46 08 04 00 00 00 e9
Dec  8 21:23:23 PlexUnraid kernel: RSP: 0018:ffffc90002337e80 EFLAGS: 00000282
Dec  8 21:23:23 PlexUnraid kernel: RAX: 0000000080000001 RBX: ffff888276608000 RCX: 0000000000000000
Dec  8 21:23:23 PlexUnraid kernel: RDX: 0000000000000002 RSI: ffffffff820d7be1 RDI: 00000000ffffffff
Dec  8 21:23:23 PlexUnraid kernel: RBP: ffffc90002337eb8 R08: 0000000000000000 R09: 0000000000000002
Dec  8 21:23:23 PlexUnraid kernel: R10: 0000000000000003 R11: 0000000000000000 R12: ffff88846fcec100
Dec  8 21:23:23 PlexUnraid kernel: R13: 0000000000000000 R14: ffff8881009f4ec0 R15: ffff8881009f5480
Dec  8 21:23:23 PlexUnraid kernel: finish_task_switch.isra.0+0x140/0x21c
Dec  8 21:23:23 PlexUnraid kernel: __schedule+0x59e/0x5f6
Dec  8 21:23:23 PlexUnraid kernel: schedule_idle+0x27/0x34
Dec  8 21:23:23 PlexUnraid kernel: cpu_startup_entry+0x1d/0x1f
Dec  8 21:23:23 PlexUnraid kernel: start_secondary+0xeb/0xeb
Dec  8 21:23:23 PlexUnraid kernel: secondary_startup_64_no_verify+0xce/0xdb
Dec  8 21:23:23 PlexUnraid kernel: </TASK>
Dec  8 21:23:23 PlexUnraid kernel: ---[ end trace 0000000000000000 ]---

[JorgeB]

Try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

[Jase]

Thank you JorgeB. I will try your suggestion. 

[TheLinuxGuy]

I am curious if you ever went back to macvlan or if you saw any issues? 

I have the same NIC; on latest unraid I get the kernel crashes macvlan.