Unraid Version: 6.12.10

Issue: Unraid will run fine for anywhere from several days up to a month or so but at some point i'll have no access in via web/ssh, cannot ping IP. I reboot and regain webgui and sometimes i will have to xfs repair drives that are unmountable but generally i can get everything working again. I have syslog server and can see that last night at 3:30 the following happened before the server died.

Request: Does anyone understand what is possibly going on here? I've looked at the logs and i can see a kernel error but i am not sure the cause. I have tried different unraid versions. All my drives are ok, some have smart errors (2) but would not expect this to cause a loss of connection to the server.

 

Jul  9 03:30:25 Tower2 kernel: CPU: 4 PID: 12903 Comm: apps.plugin Tainted: P           O       6.1.79-Unraid #1
Jul  9 03:30:25 Tower2 kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./970A-DS3P, BIOS FD 02/26/2016
Jul  9 03:30:25 Tower2 kernel: RIP: 0010:do_task_stat+0x232/0xa1e
Jul  9 03:30:25 Tower2 kernel: Code: 48 89 44 24 60 49 8b 86 d0 01 00 00 48 89 44 24 68 49 8b 86 f0 02 00 00 48 89 44 24 70 74 5d 48 89 d8 31 d2 45 31 e4 45 31 ed <4c> 03 a8 c8 05 00 00 4c 03 a0 d0 05 00 00 48 03 90 88 05 00 00 48
Jul  9 03:30:25 Tower2 kernel: RSP: 0018:ffffc9002237fc38 EFLAGS: 00010097
Jul  9 03:30:25 Tower2 kernel: RAX: fffffffffffffac8 RBX: ffff8883d9a52f40 RCX: 0000000000000040
Jul  9 03:30:25 Tower2 kernel: RDX: 0000000000000000 RSI: ffffc9002237fce8 RDI: ffff8882a0257380
Jul  9 03:30:25 Tower2 kernel: RBP: ffff88810280d500 R08: 0000000000000001 R09: 0000000000000000
Jul  9 03:30:25 Tower2 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000002000009
Jul  9 03:30:25 Tower2 kernel: R13: 00000000000095a6 R14: ffff88819f409c00 R15: 0000000080000001
Jul  9 03:30:25 Tower2 kernel: FS:  00007f3f2c92c600(0000) GS:ffff88850ad00000(0000) knlGS:0000000000000000
Jul  9 03:30:25 Tower2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  9 03:30:25 Tower2 kernel: CR2: 0000000000000090 CR3: 00000003e1e74000 CR4: 00000000000406e0
Jul  9 03:30:25 Tower2 kernel: Call Trace:
Jul  9 03:30:25 Tower2 kernel: <TASK>
Jul  9 03:30:25 Tower2 kernel: ? __die_body+0x1a/0x5c
Jul  9 03:30:25 Tower2 kernel: ? page_fault_oops+0x329/0x376
Jul  9 03:30:25 Tower2 kernel: ? fixup_exception+0x22/0x24b
Jul  9 03:30:25 Tower2 kernel: ? exc_page_fault+0xfb/0x11d
Jul  9 03:30:25 Tower2 kernel: ? asm_exc_page_fault+0x22/0x30
Jul  9 03:30:25 Tower2 kernel: ? do_task_stat+0x232/0xa1e
Jul  9 03:30:25 Tower2 kernel: ? do_task_stat+0x1d8/0xa1e
Jul  9 03:30:25 Tower2 kernel: ? slab_post_alloc_hook+0x11c/0x15e
Jul  9 03:30:25 Tower2 kernel: proc_single_show+0x54/0x73
Jul  9 03:30:25 Tower2 kernel: seq_read_iter+0x16c/0x346
Jul  9 03:30:25 Tower2 kernel: seq_read+0x92/0xbc
Jul  9 03:30:25 Tower2 kernel: vfs_read+0xa7/0x19f
Jul  9 03:30:25 Tower2 kernel: ? __fget+0x33/0x41
Jul  9 03:30:25 Tower2 kernel: ksys_read+0x76/0xc2
Jul  9 03:30:25 Tower2 kernel: do_syscall_64+0x6b/0x81
Jul  9 03:30:25 Tower2 kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
Jul  9 03:30:25 Tower2 kernel: RIP: 0033:0x7f3f2c2061dc
Jul  9 03:30:25 Tower2 kernel: Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 d9 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 2f d6 f8 ff 48
Jul  9 03:30:25 Tower2 kernel: RSP: 002b:00007fff683a1b40 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jul  9 03:30:25 Tower2 kernel: RAX: ffffffffffffffda RBX: 0000558362390430 RCX: 00007f3f2c2061dc
Jul  9 03:30:25 Tower2 kernel: RDX: 0000000000003776 RSI: 0000558362390560 RDI: 0000000000000004
Jul  9 03:30:25 Tower2 kernel: RBP: 0000000000000193 R08: 0000000000000000 R09: 00000000000051c9
Jul  9 03:30:25 Tower2 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
Jul  9 03:30:25 Tower2 kernel: R13: 0000000000000130 R14: 0000558361d32a68 R15: 0000558361d33050
Jul  9 03:30:25 Tower2 kernel: </TASK>
Jul  9 03:30:25 Tower2 kernel: Modules linked in: ipvlan md_mod tls veth xt_nat xt_tcpudp xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter xfs nfsd auth_rpcgss lockd grace sunrpc cmac zfs(PO) cifs asn1_decoder cifs_arc4 cifs_md4 oid_registry dns_resolver zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) tcp_diag inet_diag it87 hwmon_vid iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc edac_mce_amd edac_core kvm_amd ccp kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 mvsas aesni_intel libsas crypto_simd i2c_piix4 r8169 ahci cryptd fam15h_power i2c_core k10temp scsi_transport_sas pata_atiixp libahci realtek button acpi_cpufreq unix
Jul  9 03:30:25 Tower2 kernel: [last unloaded: md_mod]
Jul  9 03:30:25 Tower2 kernel: CR2: 0000000000000090
Jul  9 03:30:25 Tower2 kernel: ---[ end trace 0000000000000000 ]---
Jul  9 03:30:25 Tower2 kernel: RIP: 0010:do_task_stat+0x232/0xa1e
Jul  9 03:30:25 Tower2 kernel: Code: 48 89 44 24 60 49 8b 86 d0 01 00 00 48 89 44 24 68 49 8b 86 f0 02 00 00 48 89 44 24 70 74 5d 48 89 d8 31 d2 45 31 e4 45 31 ed <4c> 03 a8 c8 05 00 00 4c 03 a0 d0 05 00 00 48 03 90 88 05 00 00 48
Jul  9 03:30:25 Tower2 kernel: RSP: 0018:ffffc9002237fc38 EFLAGS: 00010097
Jul  9 03:30:25 Tower2 kernel: RAX: fffffffffffffac8 RBX: ffff8883d9a52f40 RCX: 0000000000000040
Jul  9 03:30:25 Tower2 kernel: RDX: 0000000000000000 RSI: ffffc9002237fce8 RDI: ffff8882a0257380
Jul  9 03:30:25 Tower2 kernel: RBP: ffff88810280d500 R08: 0000000000000001 R09: 0000000000000000
Jul  9 03:30:25 Tower2 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000002000009
Jul  9 03:30:25 Tower2 kernel: R13: 00000000000095a6 R14: ffff88819f409c00 R15: 0000000080000001
Jul  9 03:30:25 Tower2 kernel: FS:  00007f3f2c92c600(0000) GS:ffff88850ad00000(0000) knlGS:0000000000000000
Jul  9 03:30:25 Tower2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul  9 03:30:25 Tower2 kernel: CR2: 0000000000000090 CR3: 00000003e1e74000 CR4: 00000000000406e0
Jul  9 03:30:25 Tower2 kernel: note: apps.plugin[12903] exited with irqs disabled
Jul  9 03:30:25 Tower2 kernel: note: apps.plugin[12903] exited with preempt_count 1
Jul  9 03:30:27 Tower2 emhttpd: read SMART /dev/sdf
Jul  9 03:30:38 Tower2 emhttpd: read SMART /dev/sde
Jul  9 03:30:47 Tower2 emhttpd: read SMART /dev/sdd
Jul  9 03:30:55 Tower2 emhttpd: read SMART /dev/sdh
Jul  9 03:31:30 Tower2 kernel: CIFS: VFS: \\192.168.2.44\media Close unmatched open for MID:8520916
Jul  9 03:32:03 Tower2 kernel: CIFS: VFS: \\192.168.2.44\Media Close unmatched open for MID:8521813

syslog-192.168.2.88 - Copy.log tower2-diagnostics-20240710-1257.zip

Edited July 10, 20241 yr by ivez
some part did not make sense

There are multiple call traces logged, including the Unraid driver crashing, that is almost always a hardware issue, though if the server can run for a month it may be more difficult to find the issue, if you have multiple sticks try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.