I am getting random server lock ups.

 

No web, SSH, or direct interface is available.

 

I have to do a hard reset.

 

I am thinking it is an issue with memory maybe, but wanted additional eyes if possible.

tower-diagnostics-20210614-1839.zip

Enable syslog mirror to flash then post that log after a crash.

Right before lockup:

 

Jun 14 19:05:56 Tower kernel: ------------[ cut here ]------------
Jun 14 19:05:56 Tower kernel: WARNING: CPU: 1 PID: 11161 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Jun 14 19:05:56 Tower kernel: Modules linked in: iptable_mangle xt_mark macvlan tun nvidia_uvm(PO) veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 br_netfilter xfs md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper drm backlight agpgart syscopyarea sysfillrect sysimgblt fb_sys_fops nvidia(PO) ipmi_devintf nct6775 hwmon_vid lm92 ip6table_filter ip6_tables iptable_filter ip_tables x_tables igb i2c_algo_bit sb_edac x86_pkg_temp_thermal intel_powerclamp ipmi_ssif coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper rapl intel_cstate mpt3sas input_leds i2c_i801 intel_uncore i2c_smbus i2c_core ahci led_class raid_class scsi_transport_sas libahci wmi acpi_ipmi ipmi_si button [last unloaded: i2c_algo_bit]
Jun 14 19:05:56 Tower kernel: CPU: 1 PID: 11161 Comm: kworker/1:1 Tainted: P           O      5.10.28-Unraid #1
Jun 14 19:05:56 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0a 06/23/2016
Jun 14 19:05:56 Tower kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Jun 14 19:05:56 Tower kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Jun 14 19:05:56 Tower kernel: Code: e8 dc f8 ff ff 44 89 fa 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 36 f6 ff ff 84 c0 75 bb 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 6d f3 ff ff e8 35 f5 ff ff e9 22 01
Jun 14 19:05:56 Tower kernel: RSP: 0018:ffffc9000001ddd8 EFLAGS: 00010202
Jun 14 19:05:56 Tower kernel: RAX: 0000000000000188 RBX: 0000000000000151 RCX: 000000005e78c745
Jun 14 19:05:56 Tower kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa0271744
Jun 14 19:05:56 Tower kernel: RBP: ffff888224f1a780 R08: 000000002b6915c7 R09: ffff888247e13a40
Jun 14 19:05:56 Tower kernel: R10: 0000000000000000 R11: ffff8881360d6600 R12: 0000000000001543
Jun 14 19:05:56 Tower kernel: R13: ffffffff8210b440 R14: 0000000000000151 R15: 0000000000000000
Jun 14 19:05:56 Tower kernel: FS:  0000000000000000(0000) GS:ffff88903f840000(0000) knlGS:0000000000000000
Jun 14 19:05:56 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 14 19:05:56 Tower kernel: CR2: 000055c1a6f32a8c CR3: 000000000200a006 CR4: 00000000003706e0
Jun 14 19:05:56 Tower kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 14 19:05:56 Tower kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jun 14 19:05:56 Tower kernel: Call Trace:
Jun 14 19:05:56 Tower kernel: <IRQ>
Jun 14 19:05:56 Tower kernel: nf_conntrack_confirm+0x2f/0x36 [nf_conntrack]
Jun 14 19:05:56 Tower kernel: nf_hook_slow+0x39/0x8e
Jun 14 19:05:56 Tower kernel: nf_hook.constprop.0+0xb1/0xd8
Jun 14 19:05:56 Tower kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe
Jun 14 19:05:56 Tower kernel: ip_local_deliver+0x49/0x75
Jun 14 19:05:56 Tower kernel: __netif_receive_skb_one_core+0x74/0x95
Jun 14 19:05:56 Tower kernel: process_backlog+0xa3/0x13b
Jun 14 19:05:56 Tower kernel: net_rx_action+0xf4/0x29d
Jun 14 19:05:56 Tower kernel: __do_softirq+0xc4/0x1c2
Jun 14 19:05:56 Tower kernel: asm_call_irq_on_stack+0x12/0x20
Jun 14 19:05:56 Tower kernel: </IRQ>
Jun 14 19:05:56 Tower kernel: do_softirq_own_stack+0x2c/0x39
Jun 14 19:05:56 Tower kernel: do_softirq+0x3a/0x44
Jun 14 19:05:56 Tower kernel: netif_rx_ni+0x1c/0x22
Jun 14 19:05:56 Tower kernel: macvlan_broadcast+0x10e/0x13c [macvlan]
Jun 14 19:05:56 Tower kernel: macvlan_process_broadcast+0xf8/0x143 [macvlan]
Jun 14 19:05:56 Tower kernel: process_one_work+0x13c/0x1d5
Jun 14 19:05:56 Tower kernel: worker_thread+0x18b/0x22f
Jun 14 19:05:56 Tower kernel: ? process_scheduled_works+0x27/0x27
Jun 14 19:05:56 Tower kernel: kthread+0xe5/0xea
Jun 14 19:05:56 Tower kernel: ? __kthread_bind_mask+0x57/0x57
Jun 14 19:05:56 Tower kernel: ret_from_fork+0x22/0x30
Jun 14 19:05:56 Tower kernel: ---[ end trace dd645e3d070463ba ]---
Jun 14 19:35:13 Tower autofan: Highest disk temp is 43C, adjusting fan speed from: 119 (46% @ 1638rpm) to: 136 (53% @ 1859rpm)
Jun 14 19:42:11 Tower webGUI: Successful login user root from 192.168.5.103
Jun 14 19:45:19 Tower kernel: TCP: request_sock_TCP: Possible SYN flooding on port 8181. Sending cookies.  Check SNMP counters.
Jun 14 20:38:53 Tower autofan: Highest disk temp is 42C, adjusting fan speed from: 136 (53% @ 1869rpm) to: 119 (46% @ 1646rpm)
Jun 14 21:42:35 Tower autofan: Highest disk temp is 41C, adjusting fan speed from: 119 (46% @ 1650rpm) to: 102 (40% @ 1384rpm)
Jun 14 23:07:29 Tower autofan: Highest disk temp is 42C, adjusting fan speed from: 102 (40% @ 1377rpm) to: 119 (46% @ 1634rpm)
Jun 14 23:41:02 Tower autofan: Highest disk temp is 41C, adjusting fan speed from: 119 (46% @ 1624rpm) to: 102 (40% @ 1371rpm)
Jun 14 23:47:10 Tower autofan: Highest disk temp is 42C, adjusting fan speed from: 102 (40% @ 1371rpm) to: 119 (46% @ 1630rpm)
Jun 15 00:17:33 Tower autofan: Highest disk temp is 43C, adjusting fan speed from: 119 (46% @ 1636rpm) to: 136 (53% @ 1867rpm)
Jun 15 02:24:54 Tower autofan: Highest disk temp is 42C, adjusting fan speed from: 136 (53% @ 1862rpm) to: 119 (46% @ 1650rpm)
Jun 15 03:00:01 Tower Plugin Auto Update: Checking for available plugin updates
Jun 15 03:00:01 Tower Docker Auto Update: Community Applications Docker Autoupdate running
Jun 15 03:00:01 Tower Docker Auto Update: Checking for available updates
Jun 15 03:00:12 Tower crond[2860]: exit status 255 from user root /usr/local/emhttp/plugins/ca.update.applications/scripts/updateApplications.php >/dev/null 2>&1
Jun 15 03:00:13 Tower Docker Auto Update: Found update for unifi-controller.  Not set to autoupdate
Jun 15 03:00:13 Tower Docker Auto Update: Stopping binhex-plexpass
Jun 15 03:00:34 Tower Docker Auto Update: Stopping MusicBrainz-Picard
Jun 15 03:00:46 Tower kernel: veth8190a5a: renamed from eth0
Jun 15 03:00:48 Tower Docker Auto Update: Installing Updates for binhex-plexpass MusicBrainz-Picard
Jun 15 03:01:35 Tower root: /var/lib/docker: 78.2 GiB (83950587904 bytes) trimmed on /dev/loop2
Jun 15 03:01:35 Tower root: /mnt/app_data: 254.1 GiB (272818348032 bytes) trimmed on /dev/sdq1
Jun 15 03:03:35 Tower Docker Auto Update: Restarting binhex-plexpass
Jun 15 03:03:38 Tower kernel: pcieport 0000:00:1c.7: Enabling MPC IRBNCE
Jun 15 03:03:38 Tower kernel: pcieport 0000:00:1c.7: Intel PCH root port ACS workaround enabled
Jun 15 03:04:01 Tower Docker Auto Update: Restarting MusicBrainz-Picard
Jun 15 03:04:04 Tower kernel: eth0: renamed from veth8b6785d
Jun 15 03:04:05 Tower Docker Auto Update: Community Applications Docker Autoupdate finished
Jun 15 04:04:58 Tower autofan: Highest disk temp is 41C, adjusting fan speed from: 119 (46% @ 1636rpm) to: 102 (40% @ 1380rpm)
Jun 15 05:14:43 Tower autofan: Highest disk temp is 42C, adjusting fan speed from: 102 (40% @ 1376rpm) to: 119 (46% @ 1636rpm)
Jun 15 05:32:58 Tower autofan: Highest disk temp is 43C, adjusting fan speed from: 119 (46% @ 1658rpm) to: 136 (53% @ 1877rpm)

syslog-192.168.5.20.log

Macvlan call traces are usually the result of having dockers with a custom IP address, more info below.

https://forums.unraid.net/topic/70529-650-call-traces-when-assigning-ip-address-to-docker-containers/

See also here:

https://forums.unraid.net/bug-reports/stable-releases/690691-kernel-panic-due-to-netfilter-nf_nat_setup_info-docker-static-ip-macvlan-r1356/

Thank you for the help. I found this last night and have already delete the staticly assigned ip from my dockers. It is sad I can no longer do this after doing it for years.

Just now, strahd_zarovich said:

It is sad I can no longer do this after doing it for years.

You might be able to use it again once v6.10 is released.