Every once in a while I'll get a hard lock where dockers/fileshares/UI won't respond.  Only way to get access again is to hard reboot.

 

Only items I can see in the logs were a XFS corruption that I was able to successfully repair and haven't seen since and the following lines in the log:
 

Feb  7 08:37:44 UNRAID kernel: ------------[ cut here ]------------
Feb  7 08:37:44 UNRAID kernel: WARNING: CPU: 5 PID: 20303 at net/netfilter/nf_conntrack_core.c:1208 __nf_conntrack_confirm+0xa5/0x2cb [nf_conntrack]
Feb  7 08:37:44 UNRAID kernel: Modules linked in: xt_mark nvidia_uvm(PO) veth xt_nat xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp macvlan ip6table_mangle ip6table_nat iptable_mangle vhost_net tun vhost vhost_iotlb tap xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs md_mod iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bridge stp llc igb nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) mgag200 drm_shmem_helper x86_pkg_temp_thermal intel_powerclamp mxm_wmi coretemp kvm_intel ipmi_ssif kvm drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm aesni_intel mpt3sas crypto_simd cryptd rapl intel_cstate isci i2c_i801 backlight ahci syscopyarea intel_uncore i2c_algo_bit i2c_smbus sysfillrect input_leds libsas joydev
Feb  7 08:37:44 UNRAID kernel: led_class libahci acpi_ipmi sysimgblt raid_class i2c_core fb_sys_fops scsi_transport_sas ipmi_si wmi button unix [last unloaded: igb]
Feb  7 08:37:44 UNRAID kernel: CPU: 5 PID: 20303 Comm: kworker/5:1 Tainted: P           O      5.19.17-Unraid #2
Feb  7 08:37:44 UNRAID kernel: Hardware name: Penguin Computing Icebreaker 4824/X9DR3-F, BIOS 3.3 07/12/2018
Feb  7 08:37:44 UNRAID kernel: Workqueue: events macvlan_process_broadcast [macvlan]
Feb  7 08:37:44 UNRAID kernel: RIP: 0010:__nf_conntrack_confirm+0xa5/0x2cb [nf_conntrack]
Feb  7 08:37:44 UNRAID kernel: Code: c6 48 89 44 24 10 e8 dd e2 ff ff 8b 7c 24 04 89 da 89 c6 89 04 24 e8 56 e6 ff ff 84 c0 75 a2 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 8b 34 24 8b 7c 24 04 e8 16 de ff ff e8 2c e3 ff ff e9 7e 01
Feb  7 08:37:44 UNRAID kernel: RSP: 0018:ffffc90006590da0 EFLAGS: 00010202
Feb  7 08:37:44 UNRAID kernel: RAX: 0000000000000188 RBX: 0000000000000000 RCX: 9706b266e25f0069
Feb  7 08:37:44 UNRAID kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffa02f8f50
Feb  7 08:37:44 UNRAID kernel: RBP: ffff8881df96a800 R08: d85ec73e09aa0329 R09: 64d95293d5238370
Feb  7 08:37:44 UNRAID kernel: R10: c1b8326ca949ea91 R11: 85f488a948bf155a R12: ffffffff82909480
Feb  7 08:37:44 UNRAID kernel: R13: 0000000000018449 R14: ffff8883811b4800 R15: 0000000000000000
Feb  7 08:37:44 UNRAID kernel: FS:  0000000000000000(0000) GS:ffff888c4fb40000(0000) knlGS:0000000000000000
Feb  7 08:37:44 UNRAID kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb  7 08:37:44 UNRAID kernel: CR2: 00001518be7ad4d0 CR3: 000000000220a002 CR4: 00000000001706e0
Feb  7 08:37:44 UNRAID kernel: Call Trace:
Feb  7 08:37:44 UNRAID kernel: <IRQ>
Feb  7 08:37:44 UNRAID kernel: nf_conntrack_confirm+0x25/0x54 [nf_conntrack]
Feb  7 08:37:44 UNRAID kernel: nf_hook_slow+0x3a/0x96
Feb  7 08:37:44 UNRAID kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Feb  7 08:37:44 UNRAID kernel: NF_HOOK.constprop.0+0x79/0xd9
Feb  7 08:37:44 UNRAID kernel: ? ip_protocol_deliver_rcu+0x164/0x164
Feb  7 08:37:44 UNRAID kernel: __netif_receive_skb_one_core+0x77/0x9c
Feb  7 08:37:44 UNRAID kernel: process_backlog+0x8c/0x116
Feb  7 08:37:44 UNRAID kernel: __napi_poll.constprop.0+0x28/0x124
Feb  7 08:37:44 UNRAID kernel: net_rx_action+0x159/0x24f
Feb  7 08:37:44 UNRAID kernel: __do_softirq+0x126/0x288
Feb  7 08:37:44 UNRAID kernel: do_softirq+0x7f/0xab
Feb  7 08:37:44 UNRAID kernel: </IRQ>
Feb  7 08:37:44 UNRAID kernel: <TASK>
Feb  7 08:37:44 UNRAID kernel: __local_bh_enable_ip+0x4c/0x6b
Feb  7 08:37:44 UNRAID kernel: netif_rx+0x52/0x5a
Feb  7 08:37:44 UNRAID kernel: macvlan_broadcast+0x10a/0x150 [macvlan]
Feb  7 08:37:44 UNRAID kernel: macvlan_process_broadcast+0xbc/0x12f [macvlan]
Feb  7 08:37:44 UNRAID kernel: process_one_work+0x1a8/0x295
Feb  7 08:37:44 UNRAID kernel: worker_thread+0x18b/0x244
Feb  7 08:37:44 UNRAID kernel: ? rescuer_thread+0x281/0x281
Feb  7 08:37:44 UNRAID kernel: kthread+0xe4/0xef
Feb  7 08:37:44 UNRAID kernel: ? kthread_complete_and_exit+0x1b/0x1b
Feb  7 08:37:44 UNRAID kernel: ret_from_fork+0x1f/0x30
Feb  7 08:37:44 UNRAID kernel: </TASK>
Feb  7 08:37:44 UNRAID kernel: ---[ end trace 0000000000000000 ]---

 

The rest of the logs are normal lines (spin up/downs, accessing the UI, mover items, etc).

The CPU may change numbers but the overall items in the snippets are similar.

This portion of the log doesn't coincide with a crash though.  It could be hours or days after seeing something like this before a crash.

I've also run a memtest and all sticks are good.

 

Machine specs are:

M/B:Supermicro X9DR3-F - s/n: XXXXXXXXXX

BIOS:American Megatrends Inc. Version 3.3. Dated: 07/12/2018

CPU:Dual Intel® Xeon® CPU E5-2650 v2 @ 2.60GHz

HVM:Enabled

IOMMU:Enabled

Cache:512 KiB, 2 MB, 20 MB, 512 KiB, 2 MB, 20 MB

Memory:96 GiB DDR3 Multi-bit ECC (max. installable capacity 192 GiB)

Network:eth0: 1000 Mbps, full duplex, mtu 1500

Kernel:Linux 5.19.17-Unraid x86_64

OpenSSL:1.1.1s

 

 

Macvlan call traces are usually the result of having dockers with a custom IP address and will end up crashing the server, upgrading to v6.10 or later and switching to ipvlan should fix it (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

So I can switch it from macvlan to ipvlan with no other changes needed in my individual dockers?

Usually yes.

So this resolved the issue of those lines in the log but I've had a complete freeze since then.  I had the logs written to the flash drive and on the cache drive as well.

All I see right before the freeze is normal spin down lines.  The log times stop and come back when I find the frozen machine and restart it back up.

 

So it appears as though the logs aren't catching the error for the hard freeze.  Any other suggestions?

attach diagnostics to your NEXT post in this thread

I appreciate the help.

xorunraid-diagnostics-20230213-2158.zip

11 hours ago, thegurujim said:

I had the logs written to the flash drive

Post this also please.

Here are those.

Feb 6th at 3:10AM is when it froze in this one.

syslog-192.168.2.70.log

Edited February 15, 20233 yr by thegurujim

Unfortunately there's nothing relevant logged:

 

Feb  6 03:08:55 XORUNRAID  emhttpd: spinning down /dev/sdv
Feb  6 03:10:36 XORUNRAID  emhttpd: spinning down /dev/sdu
Feb  6 05:00:01 XORUNRAID Recycle Bin: Scheduled: Files older than 3 days have been removed

 

This usually points to a hardware problem.

That's unfortunate.  I have the BIOs logging now as well, maybe that will catch something.