Hello All,

 

I have been seeing random freezes on my Unraid server (6.9.2). I started capturing logs; however, I am unable to pinpoint the error. There has been no hardware changes whatsoever. Will someone be kind enough to look at the attached log and provide some guidance?

 

Thank you!

syslog-192.168.1.11.log.zip

It would be more useful to provide your diagnostics.

However, from your signature I see the memory at 3600MHz, it's way off the supported speed :

 

 

Thank for ChatNoir for the prompt reply. I have attached the diagnostic in the above post. Is that not the right one? As for the RAM, I am not using any xmp profiles. It is just stock speed. Is there any way in Unraid for me to see the RAM timings? Also, I have had the same setup for over a year. This just started happening fairly recently. The only thing I update are the dockers when there is an update.

No, thats the syslog, the diagnostics provide much more information and includes the syslog. Check the link for details on how to get the diagnostics.

 

As for your question about actual memory speed, its included in the diags.  

Thank you for guiding me in the right direction. I have attached the diagnostic here; however, I am not sure if it will have any insight as to why the server froze this morning. I appreciate your guidance.

 

diagnostics-20220413-0851.zip

Edited April 13, 20224 yr by beinghitesh

We can at least confirm that your memory speed is OK (even lower than supported).

 

To confirm, when you say that your system freezes, it cannot recover and you have to reboot ?

 

If that's the case, we will not be able to that on the diagnostics after reboot since the log is stored in RAM and lost on reboot. You should set up a syslog server and attach this log file after the next crash.

That's what I thought. That's why I set up the syslog server so I can see what might be the culprit. In the syslog attached in the original post, I believe the server froze at round 4am EDT. When it freezes, it is completely unresponsive. I have to do a hard shutdown and restart to bring it back online.

 

Your syslog is filling with these:

Apr 13 07:55:53 teshomatic kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Apr 13 07:55:53 teshomatic kernel: caller _nv000651rm+0x1ad/0x200 [nvidia] mapping multiple BARs

I don't use nvidia and maybe it isn't related, but doesn't seem normal to have your syslog spammed like that. What happens if you remove the nvidia plugin?

 

Unrelated

Apr 13 08:05:01 teshomatic root: Fix Common Problems: Warning: Share appdata set to cache-only, but files / folders exist on the array
Apr 13 08:05:01 teshomatic root: Fix Common Problems: Warning: Share system set to cache-only, but files / folders exist on the array

 

 

 

There is a 11 hours gap at the begining, so maybe it cashed, can't say.

 

However, there are several call traces like that :

Apr 12 09:56:21 teshomatic kernel: ------------[ cut here ]------------
Apr 12 09:56:21 teshomatic kernel: WARNING: CPU: 34 PID: 0 at net/netfilter/nf_conntrack_core.c:1120 __nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Apr 12 09:56:21 teshomatic kernel: Modules linked in: nvidia_uvm(PO) veth xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_nat ip6table_mangle ip6table_nat xt_tcpudp iptable_mangle nf_tables vhost_net tun vhost vhost_iotlb tap macvlan xt_conntrack nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xfs md_mod nvidia_drm(PO) nvidia_modeset(PO) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nvidia(PO) drm backlight agpgart k10temp it87 hwmon_vid iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s blake2s_x86_64 libblake2s_generic libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables bonding ixgbe mdio edac_mce_amd mxm_wmi wmi_bmof amd_energy btusb btrtl btbcm btintel kvm_amd bluetooth kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel ecdh_generic crypto_simd ecc cryptd glue_helper ahci mpt3sas
Apr 12 09:56:21 teshomatic kernel: i2c_piix4 ccp i2c_core libahci rapl raid_class nvme scsi_transport_sas nvme_core wmi thermal button acpi_cpufreq [last unloaded: mdio]
Apr 12 09:56:21 teshomatic kernel: CPU: 34 PID: 0 Comm: swapper/34 Tainted: P           O      5.10.28-Unraid #1
Apr 12 09:56:21 teshomatic kernel: Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS XTREME/TRX40 AORUS XTREME, BIOS F5a 10/20/2021
Apr 12 09:56:21 teshomatic kernel: RIP: 0010:__nf_conntrack_confirm+0x9b/0x1e6 [nf_conntrack]
Apr 12 09:56:21 teshomatic kernel: Code: e8 dc f8 ff ff 44 89 fa 89 c6 41 89 c4 48 c1 eb 20 89 df 41 89 de e8 36 f6 ff ff 84 c0 75 bb 48 8b 85 80 00 00 00 a8 08 74 18 <0f> 0b 89 df 44 89 e6 31 db e8 6d f3 ff ff e8 35 f5 ff ff e9 22 01
Apr 12 09:56:21 teshomatic kernel: RSP: 0018:ffffc90000cb4898 EFLAGS: 00010202
Apr 12 09:56:21 teshomatic kernel: RAX: 0000000000000188 RBX: 0000000000000421 RCX: 00000000575dd61d
Apr 12 09:56:21 teshomatic kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa015ff4c
Apr 12 09:56:21 teshomatic kernel: RBP: ffff88816e525cc0 R08: 00000000a9f3c50e R09: ffff888101ee0560
Apr 12 09:56:21 teshomatic kernel: R10: 0000000000000158 R11: ffff888101191400 R12: 000000000000a353
Apr 12 09:56:21 teshomatic kernel: R13: ffffffff8210b440 R14: 0000000000000421 R15: 0000000000000000
Apr 12 09:56:21 teshomatic kernel: FS:  0000000000000000(0000) GS:ffff889fdd880000(0000) knlGS:0000000000000000
Apr 12 09:56:21 teshomatic kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 12 09:56:21 teshomatic kernel: CR2: 0000000000607318 CR3: 00000001566fa000 CR4: 0000000000350ee0
Apr 12 09:56:21 teshomatic kernel: Call Trace:
Apr 12 09:56:21 teshomatic kernel: <IRQ>
Apr 12 09:56:21 teshomatic kernel: nf_conntrack_confirm+0x2f/0x36 [nf_conntrack]
Apr 12 09:56:21 teshomatic kernel: nf_hook_slow+0x39/0x8e
Apr 12 09:56:21 teshomatic kernel: nf_hook.constprop.0+0xb1/0xd8
Apr 12 09:56:21 teshomatic kernel: ? ip_protocol_deliver_rcu+0xfe/0xfe
Apr 12 09:56:21 teshomatic kernel: ip_local_deliver+0x49/0x75
Apr 12 09:56:21 teshomatic kernel: ip_sabotage_in+0x43/0x4d [br_netfilter]
Apr 12 09:56:21 teshomatic kernel: nf_hook_slow+0x39/0x8e
Apr 12 09:56:21 teshomatic kernel: nf_hook.constprop.0+0xb1/0xd8
Apr 12 09:56:21 teshomatic kernel: ? l3mdev_l3_rcv.constprop.0+0x50/0x50
Apr 12 09:56:21 teshomatic kernel: ip_rcv+0x41/0x61
Apr 12 09:56:21 teshomatic kernel: __netif_receive_skb_one_core+0x74/0x95
Apr 12 09:56:21 teshomatic kernel: netif_receive_skb+0x79/0xa1
Apr 12 09:56:21 teshomatic kernel: br_handle_frame_finish+0x30d/0x351
Apr 12 09:56:21 teshomatic kernel: ? ipt_do_table+0x570/0x5c0 [ip_tables]
Apr 12 09:56:21 teshomatic kernel: ? br_pass_frame_up+0xda/0xda
Apr 12 09:56:21 teshomatic kernel: br_nf_hook_thresh+0xa3/0xc3 [br_netfilter]
Apr 12 09:56:21 teshomatic kernel: ? br_pass_frame_up+0xda/0xda
Apr 12 09:56:21 teshomatic kernel: br_nf_pre_routing_finish+0x23d/0x264 [br_netfilter]
Apr 12 09:56:21 teshomatic kernel: ? br_pass_frame_up+0xda/0xda
Apr 12 09:56:21 teshomatic kernel: ? br_handle_frame_finish+0x351/0x351
Apr 12 09:56:21 teshomatic kernel: ? nf_nat_ipv4_pre_routing+0x1e/0x4a [nf_nat]
Apr 12 09:56:21 teshomatic kernel: ? br_nf_forward_finish+0xd0/0xd0 [br_netfilter]
Apr 12 09:56:21 teshomatic kernel: ? br_handle_frame_finish+0x351/0x351
Apr 12 09:56:21 teshomatic kernel: NF_HOOK+0xd7/0xf7 [br_netfilter]
Apr 12 09:56:21 teshomatic kernel: ? br_nf_forward_finish+0xd0/0xd0 [br_netfilter]
Apr 12 09:56:21 teshomatic kernel: br_nf_pre_routing+0x229/0x239 [br_netfilter]
Apr 12 09:56:21 teshomatic kernel: ? br_nf_forward_finish+0xd0/0xd0 [br_netfilter]
Apr 12 09:56:21 teshomatic kernel: br_handle_frame+0x25e/0x2a6
Apr 12 09:56:21 teshomatic kernel: ? br_pass_frame_up+0xda/0xda
Apr 12 09:56:21 teshomatic kernel: __netif_receive_skb_core+0x335/0x4e7
Apr 12 09:56:21 teshomatic kernel: __netif_receive_skb_list_core+0x78/0x104
Apr 12 09:56:21 teshomatic kernel: netif_receive_skb_list_internal+0x1bf/0x1f2
Apr 12 09:56:21 teshomatic kernel: ? dev_gro_receive+0x55d/0x578
Apr 12 09:56:21 teshomatic kernel: gro_normal_list+0x1d/0x39
Apr 12 09:56:21 teshomatic kernel: napi_complete_done+0x79/0x104
Apr 12 09:56:21 teshomatic kernel: ixgbe_poll+0xc95/0xd4f [ixgbe]
Apr 12 09:56:21 teshomatic kernel: net_rx_action+0xf4/0x29d
Apr 12 09:56:21 teshomatic kernel: __do_softirq+0xc4/0x1c2
Apr 12 09:56:21 teshomatic kernel: asm_call_irq_on_stack+0x12/0x20
Apr 12 09:56:21 teshomatic kernel: </IRQ>
Apr 12 09:56:21 teshomatic kernel: do_softirq_own_stack+0x2c/0x39
Apr 12 09:56:21 teshomatic kernel: __irq_exit_rcu+0x45/0x80
Apr 12 09:56:21 teshomatic kernel: common_interrupt+0x119/0x12e
Apr 12 09:56:21 teshomatic kernel: asm_common_interrupt+0x1e/0x40
Apr 12 09:56:21 teshomatic kernel: RIP: 0010:arch_local_irq_enable+0x7/0x8
Apr 12 09:56:21 teshomatic kernel: Code: 00 48 83 c4 28 4c 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 9c 58 0f 1f 44 00 00 c3 fa 66 0f 1f 44 00 00 c3 fb 66 0f 1f 44 00 00 <c3> 55 8b af 28 04 00 00 b8 01 00 00 00 45 31 c9 53 45 31 d2 39 c5
Apr 12 09:56:21 teshomatic kernel: RSP: 0018:ffffc90000367ea0 EFLAGS: 00000246
Apr 12 09:56:21 teshomatic kernel: RAX: ffff889fdd8a2380 RBX: 0000000000000002 RCX: 000000000000001f
Apr 12 09:56:21 teshomatic kernel: RDX: 0000000000000000 RSI: 0000000021aed5ee RDI: 0000000000000000
Apr 12 09:56:21 teshomatic kernel: RBP: ffff8881444ad800 R08: 0000072a3daf2fcc R09: 0000072a42e4415f
Apr 12 09:56:21 teshomatic kernel: R10: 000000007fffffff R11: 071c71c71c71c71c R12: 0000072a3daf2fcc
Apr 12 09:56:21 teshomatic kernel: R13: ffffffff820c8c40 R14: 0000000000000002 R15: 0000000000000000
Apr 12 09:56:21 teshomatic kernel: cpuidle_enter_state+0x101/0x1c4
Apr 12 09:56:21 teshomatic kernel: cpuidle_enter+0x25/0x31
Apr 12 09:56:21 teshomatic kernel: do_idle+0x1a6/0x214
Apr 12 09:56:21 teshomatic kernel: cpu_startup_entry+0x18/0x1a
Apr 12 09:56:21 teshomatic kernel: secondary_startup_64_no_verify+0xb0/0xbb
Apr 12 09:56:21 teshomatic kernel: ---[ end trace 4d9e100d280d6eaa ]---

 

MACVLAN is a known issue with docker containers with a custom IP.

 

I suggest that you update to 6.10.0 RC4 and switch your docker network to IPVLAN.

1 minute ago, trurl said:

I don't use nvidia and maybe it isn't related, but doesn't seem normal to have your syslog spammed like that. What happens if you remove the nvidia plugin?

It makes it difficult to scan the log for issues at least.

1 hour ago, trurl said:

Your syslog is filling with these:

Apr 13 07:55:53 teshomatic kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Apr 13 07:55:53 teshomatic kernel: caller _nv000651rm+0x1ad/0x200 [nvidia] mapping multiple BARs

I don't use nvidia and maybe it isn't related, but doesn't seem normal to have your syslog spammed like that. What happens if you remove the nvidia plugin?

 

 

I noticed that as well, but wasn't sure if it was normal or not. I use the Nvidia Driver plugin by ich777 for Plex transcoding. I can remove it and have the cpu do the transcoding and see if that fixes the issue.

 

Quick update: I did some more digging and found those errors? are caused by GPU statistics by b3rs3rk. I removed the plugin and the errors have now disappeared.

Edited April 13, 20224 yr by beinghitesh

23 minutes ago, ChatNoir said:

MACVLAN is a known issue with docker containers with a custom IP.

 

I suggest that you update to 6.10.0 RC4 and switch your docker network to IPVLAN.

I am going to read up on RC4 and issues to ensure it won't break anything else before upgrading

Same here. I have the HM90 with AMD Ryzen 9 4900H.

 

Everytime I "play" with Dockers my system freezes.

 

I also got a crash while installing a VM (Debian). First I thought my USB-Drives causing the error, so I disconnected all of them. In BIOS I disabled C States. I also did a fresh Unraid Installation, but the time I install some Docker I starts again.. mostly Plex. I don't know what to do.  I spend 5 days trying to figure out what the problem is. HELP!

 

p.s. maybe it has something to do with the network config? I tried to set up Phiole again, clicked on the Button to finish .. and just like that - system crashed again.

unraid-diagnostics-20230126-2253.zip syslog-192.168.1.174.log

Edited January 26, 20233 yr by jc82517

Short update: I disabled "bonding" in the network settings. Since that no more freezes. I will continue testing. Wish me luck.

Edited January 27, 20233 yr by jc82517