Dagills22191

Yea, I have everything shut down prior to running mover. Confirmed they are all on the array

Ended up being 1 stick of memory causing issues. Replaced that, deleted and recreated docker image and set appdata and system back to cache, ran mover and all has been well since. Just wanted to close the loop on this issue.

One fear I have is it is the cache drive (nvme). So, in preparation for the possibility of having to replace that I temporarily set the appdata (and other shares) to cache-> array so I can clear out the cache drive if that is what I need to do. That is not how I normally run it. I also just ran a memtest and it is getting loads of errors (i hadn't run that before my first post). I'm going to start with replacing the memory and see where that gets me.

The next crash is coming soon I assume as the current syslog is showing errors already: text error warn system array login Mar 24 12:24:23 UnraidGill kernel: CPU: 0 PID: 239 Comm: kswapd0 Tainted: P O 6.6.78-Unraid #2 Mar 24 12:24:23 UnraidGill kernel: Hardware name: Micro-Star International Co., Ltd. MS-7E06/PRO Z790-P WIFI DDR4 (MS-7E06), BIOS 1.E0 09/27/2024 Mar 24 12:24:23 UnraidGill kernel: RIP: 0010:clear_inode+0x40/0x7c Mar 24 12:24:23 UnraidGill kernel: Code: 66 78 00 48 83 bb c8 01 00 00 00 74 02 0f 0b 48 89 ef e8 cf 67 78 00 48 8b 93 18 02 00 00 48 8d 83 18 02 00 00 48 39 c2 74 02 <0f> 0b 48 8b 83 a0 00 00 00 a8 20 75 02 0f 0b a8 40 74 02 0f 0b 48 Mar 24 12:24:23 UnraidGill kernel: RSP: 0000:ffffc900049b7a50 EFLAGS: 00010287 Mar 24 12:24:23 UnraidGill kernel: RAX: ffff8882b9f0fb50 RBX: ffff8882b9f0f938 RCX: 0000000000000000 Mar 24 12:24:23 UnraidGill kernel: RDX: ffff8882b9f0fb10 RSI: 0000000000000000 RDI: 00000000ffffffff Mar 24 12:24:23 UnraidGill kernel: RBP: ffff8882b9f0fab0 R08: ffffffff824066f0 R09: ffff8881aef8a2c0 Mar 24 12:24:23 UnraidGill kernel: R10: 0000000000000000 R11: 00000000000320c0 R12: ffff8882b9f0f9d8 Mar 24 12:24:23 UnraidGill kernel: R13: ffff8881045496d8 R14: ffffffff81e2b6c0 R15: 0000000000000000 Mar 24 12:24:23 UnraidGill kernel: FS: 0000000000000000(0000) GS:ffff88885f200000(0000) knlGS:0000000000000000 Mar 24 12:24:23 UnraidGill kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 24 12:24:23 UnraidGill kernel: CR2: ffff88887f7ff000 CR3: 0000000005416000 CR4: 0000000000750ef0 Mar 24 12:24:23 UnraidGill kernel: PKRU: 55555554 Mar 24 12:24:23 UnraidGill kernel: Call Trace: Mar 24 12:24:23 UnraidGill kernel: <TASK> Mar 24 12:24:23 UnraidGill kernel: ? __die_body+0x1a/0x5c Mar 24 12:24:23 UnraidGill kernel: ? die+0x30/0x49 Mar 24 12:24:23 UnraidGill kernel: ? do_trap+0x7b/0xfe Mar 24 12:24:23 UnraidGill kernel: ? clear_inode+0x40/0x7c Mar 24 12:24:23 UnraidGill kernel: ? clear_inode+0x40/0x7c Mar 24 12:24:23 UnraidGill kernel: ? do_error_trap+0x6e/0x98 Mar 24 12:24:23 UnraidGill kernel: ? clear_inode+0x40/0x7c Mar 24 12:24:23 UnraidGill kernel: ? exc_invalid_op+0x4c/0x60 Mar 24 12:24:23 UnraidGill kernel: ? clear_inode+0x40/0x7c Mar 24 12:24:23 UnraidGill kernel: ? asm_exc_invalid_op+0x16/0x20 Mar 24 12:24:23 UnraidGill kernel: ? clear_inode+0x40/0x7c Mar 24 12:24:23 UnraidGill kernel: proc_evict_inode+0x1d/0x65 Mar 24 12:24:23 UnraidGill kernel: evict+0x179/0x224 Mar 24 12:24:23 UnraidGill kernel: ? preempt_latency_start+0x2b/0x46 Mar 24 12:24:23 UnraidGill kernel: ? _raw_spin_lock+0x13/0x1c Mar 24 12:24:23 UnraidGill kernel: ? _atomic_dec_and_lock+0x27/0x42 Mar 24 12:24:23 UnraidGill kernel: __dentry_kill+0xcb/0x131 Mar 24 12:24:23 UnraidGill kernel: shrink_dentry_list+0x9b/0xad Mar 24 12:24:23 UnraidGill kernel: prune_dcache_sb+0x52/0x75 Mar 24 12:24:23 UnraidGill kernel: super_cache_scan+0xf1/0x174 Mar 24 12:24:23 UnraidGill kernel: do_shrink_slab+0x187/0x29e Mar 24 12:24:23 UnraidGill kernel: shrink_slab+0x1f7/0x265 Mar 24 12:24:23 UnraidGill kernel: shrink_node+0x4df/0x734 Mar 24 12:24:23 UnraidGill kernel: balance_pgdat+0x419/0x5e3 Mar 24 12:24:23 UnraidGill kernel: kswapd+0x2d6/0x319 Mar 24 12:24:23 UnraidGill kernel: ? __pfx_autoremove_wake_function+0x10/0x10 Mar 24 12:24:23 UnraidGill kernel: ? __pfx_kswapd+0x10/0x10 Mar 24 12:24:23 UnraidGill kernel: kthread+0xf1/0xfc Mar 24 12:24:23 UnraidGill kernel: ? __pfx_kthread+0x10/0x10 Mar 24 12:24:23 UnraidGill kernel: ret_from_fork+0x21/0x36 Mar 24 12:24:23 UnraidGill kernel: ? __pfx_kthread+0x10/0x10 Mar 24 12:24:23 UnraidGill kernel: ret_from_fork_asm+0x1b/0x30 Mar 24 12:24:23 UnraidGill kernel: </TASK> Mar 24 12:24:23 UnraidGill kernel: Modules linked in: veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo ip6table_nat iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype br_netfilter nfsd auth_rpcgss lockd grace sunrpc md_mod ntfs3 tcp_diag inet_diag nct6683 ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs macvtap macvlan tap bridge stp llc atlantic igc iwlmvm mac80211 intel_rapl_common x86_pkg_temp_thermal intel_powerclamp libarc4 zfs(PO) i915 coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 sha256_ssse3 iosf_mbi sha1_ssse3 drm_buddy ttm aesni_intel i2c_algo_bit crypto_simd drm_display_helper btusb spl(O) cryptd iwlwifi btrtl drm_kms_helper btbcm btintel input_leds rapl bluetooth mei_hdcp mei_pxp drm mpt3sas wmi_bmof mxm_wmi led_class cfg80211 joydev ecdh_generic nvme ecc i2c_i801 intel_cstate raid_class intel_gtt mei_me i2c_smbus nvme_core intel_uncore agpgart ahci scsi_transport_sas mei i2c_core Mar 24 12:24:23 UnraidGill kernel: tpm_crb libahci rfkill thermal fan tpm_tis video tpm_tis_core tpm wmi backlight acpi_tad acpi_pad button [last unloaded: atlantic] Mar 24 12:24:23 UnraidGill kernel: ---[ end trace 0000000000000000 ]--- Mar 24 12:24:23 UnraidGill kernel: pstore: backend (efi_pstore) writing error (-5) Mar 24 12:24:23 UnraidGill kernel: RIP: 0010:clear_inode+0x40/0x7c Mar 24 12:24:23 UnraidGill kernel: Code: 66 78 00 48 83 bb c8 01 00 00 00 74 02 0f 0b 48 89 ef e8 cf 67 78 00 48 8b 93 18 02 00 00 48 8d 83 18 02 00 00 48 39 c2 74 02 <0f> 0b 48 8b 83 a0 00 00 00 a8 20 75 02 0f 0b a8 40 74 02 0f 0b 48 Mar 24 12:24:23 UnraidGill kernel: RSP: 0000:ffffc900049b7a50 EFLAGS: 00010287 Mar 24 12:24:23 UnraidGill kernel: RAX: ffff8882b9f0fb50 RBX: ffff8882b9f0f938 RCX: 0000000000000000 Mar 24 12:24:23 UnraidGill kernel: RDX: ffff8882b9f0fb10 RSI: 0000000000000000 RDI: 00000000ffffffff Mar 24 12:24:23 UnraidGill kernel: RBP: ffff8882b9f0fab0 R08: ffffffff824066f0 R09: ffff8881aef8a2c0 Mar 24 12:24:23 UnraidGill kernel: R10: 0000000000000000 R11: 00000000000320c0 R12: ffff8882b9f0f9d8 Mar 24 12:24:23 UnraidGill kernel: R13: ffff8881045496d8 R14: ffffffff81e2b6c0 R15: 0000000000000000 Mar 24 12:24:23 UnraidGill kernel: FS: 0000000000000000(0000) GS:ffff88885f200000(0000) knlGS:0000000000000000 Mar 24 12:24:23 UnraidGill kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Mar 24 12:24:23 UnraidGill kernel: CR2: ffff88887f7ff000 CR3: 0000000005416000 CR4: 0000000000750ef0 Mar 24 12:24:23 UnraidGill kernel: PKRU: 55555554 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 90227560 op 0x0:(READ) flags 0x80700 phys_seg 7 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 90227936 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 90227992 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 90227688 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 90227688 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 90227688 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 90227688 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 6947248 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 3413200 op 0x0:(READ) flags 0x80700 phys_seg 10 prio class 2 Mar 24 12:37:19 UnraidGill kernel: I/O error, dev loop2, sector 3413304 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2 Mar 24 12:37:19 UnraidGill kernel: loop: Write error at byte offset 16831184896, length 4096. Mar 24 12:37:21 UnraidGill kernel: loop2: writeback error on inode 35768288, offset 8024064, sector 32873408 Mar 24 12:37:21 UnraidGill kernel: loop: Write error at byte offset 46246760448, length 4096. Mar 24 12:37:21 UnraidGill kernel: loop2: writeback error on inode 103756659, offset 0, sector 90325704 Mar 24 12:37:21 UnraidGill kernel: loop: Write error at byte offset 15888646144, length 4096. Mar 24 12:37:21 UnraidGill kernel: loop2: writeback error on inode 35768296, offset 0, sector 31032512 Mar 24 12:37:21 UnraidGill kernel: loop: Write error at byte offset 16831184896, length 4096. Mar 24 13:04:04 UnraidGill emhttpd: shcmd (166): rm -f '/boot/config/shares/syslog.cfg' Mar 24 13:04:04 UnraidGill shfs: /usr/sbin/zfs create 'm2cache/syslog' 2>&1 Mar 24 13:04:51 UnraidGill ool www[372421]: /usr/local/emhttp/plugins/dynamix/scripts/rsyslog_config Mar 24 13:04:52 UnraidGill rc.rsyslogd: Restarting Syslog server daemon... Mar 24 13:04:52 UnraidGill rc.rsyslogd: Stopping Syslog server daemon... Mar 24 13:04:52 UnraidGill rc.rsyslogd: killall --ns 373098 rsyslogd Mar 24 13:04:55 UnraidGill rc.rsyslogd: Syslog server daemon... Started. Mar 24 13:04:56 UnraidGill rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="373210" x-info="https://www.rsyslog.com"] start

Got it. will do and will post back here after next crash.

Basically just that. Pretty much daily, if not more often, my Unraid is throwing lots of errors (diags attached) and becoming unresponsive. The latest change made to the system was replacing a failing drive about a week ago and letting it rebuild. That seemed to go just fine. The m2cache started showing errors but have since scrubbed it and now not showing any errors. I have rebuilt the docker image several times because generally that is the first thing to "lock up". Ideas? Hints? amusing anecdotes? Thanks in advance, Tim unraidgill-diagnostics-20250324-1117.zip