MammothJerk Posted August 5 Share Posted August 5 unraid: 6.12.11 asus Pro WS W680-ACE IPMI: bios 3701 14900k I noticed my server started hanging a couple of days ago with no configuration changes on my end. i restarted the server and a couple hours later it crashed again. I had recently heard that some intel CPUs had some instability, i went to the mobo manufacturers website and installed the latest BIOS update 3701. a couple hours later it crashed again. I tried downloading diagnostics but it seems to hang on the flash drive Quote Samsung_Flash_Drive_0376622050006768-0-0-2024-04-03 flash (sda).txt' /usr/sbin/zpool status 2>/dev/null|todos >>'/venasaur-diagnostics-20240805-0635/system/zfs-info.txt' ! Not valid! i have a syslog server setup on another server and it stops updating as well. Quote Aug 4 23:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 00:01:48 VeNASaur sshd-session[2643]: Connection closed by 192.168.1.100 port 55778 Aug 5 00:01:48 VeNASaur sshd-session[2643]: Close session: user root from 192.168.1.100 port 55778 id 0 Aug 5 00:01:48 VeNASaur sshd-session[2643]: Transferred: sent 8000, received 3288 bytes Aug 5 00:01:48 VeNASaur sshd-session[2643]: Closing connection to 192.168.1.100 port 55778 Aug 5 00:01:48 VeNASaur sshd-session[2636]: pam_unix(sshd:session): session closed for user root Aug 5 00:04:46 VeNASaur kernel: br-ee7a2a4499c0: port 11(vethb375cf0) entered disabled state Aug 5 00:04:46 VeNASaur kernel: veth3e2b2b8: renamed from eth0 Aug 5 00:04:47 VeNASaur kernel: br-ee7a2a4499c0: port 11(vethb375cf0) entered disabled state Aug 5 00:04:47 VeNASaur kernel: device vethb375cf0 left promiscuous mode Aug 5 00:04:47 VeNASaur kernel: br-ee7a2a4499c0: port 11(vethb375cf0) entered disabled state Aug 5 00:04:47 VeNASaur kernel: br-ee7a2a4499c0: port 11(vetha1e6e2f) entered blocking state Aug 5 00:04:47 VeNASaur kernel: br-ee7a2a4499c0: port 11(vetha1e6e2f) entered disabled state Aug 5 00:04:47 VeNASaur kernel: device vetha1e6e2f entered promiscuous mode Aug 5 00:04:47 VeNASaur kernel: br-ee7a2a4499c0: port 11(vetha1e6e2f) entered blocking state Aug 5 00:04:47 VeNASaur kernel: br-ee7a2a4499c0: port 11(vetha1e6e2f) entered forwarding state Aug 5 00:04:48 VeNASaur kernel: br-ee7a2a4499c0: port 11(vetha1e6e2f) entered disabled state Aug 5 00:04:48 VeNASaur kernel: eth0: renamed from vethcdc6110 Aug 5 00:04:48 VeNASaur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vetha1e6e2f: link becomes ready Aug 5 00:04:48 VeNASaur kernel: br-ee7a2a4499c0: port 11(vetha1e6e2f) entered blocking state Aug 5 00:04:48 VeNASaur kernel: br-ee7a2a4499c0: port 11(vetha1e6e2f) entered forwarding state Aug 5 00:30:26 VeNASaur kernel: .NET Long Runni[27838]: segfault at 2 ip 0000152cd9a32f37 sp 0000152c3d53acf0 error 4 in libcoreclr.so[152cd98d1000+2fc000] likely on CPU 8 (core 16, socket 0) Aug 5 00:30:26 VeNASaur kernel: Code: e8 7e fe 2e 00 0f 57 c0 0f 29 45 a0 4c 89 f7 e8 cf f1 e9 ff 48 89 c7 e8 17 0a ea ff 0f 1f 80 00 00 00 00 55 48 89 e5 48 8b 07 <f6> 40 02 10 74 48 48 89 fe 48 8d 05 79 22 51 00 48 8b 38 48 8b 07 Aug 5 00:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 01:00:07 VeNASaur kernel: mdcmd (38): check NOCORRECT Aug 5 01:00:07 VeNASaur kernel: Aug 5 01:00:07 VeNASaur kernel: md: recovery thread: check P Q ... Aug 5 01:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 02:15:42 VeNASaur sshd-session[15599]: Connection closed by 192.168.1.100 port 57024 Aug 5 02:15:43 VeNASaur sshd-session[15599]: Close session: user root from 192.168.1.100 port 57024 id 0 Aug 5 02:15:43 VeNASaur sshd-session[15599]: Transferred: sent 104167776, received 701608 bytes Aug 5 02:15:43 VeNASaur sshd-session[15599]: Closing connection to 192.168.1.100 port 57024 Aug 5 02:15:43 VeNASaur sshd-session[15593]: pam_unix(sshd:session): session closed for user root Aug 5 02:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 03:00:01 VeNASaur Plugin Auto Update: Checking for available plugin updates Aug 5 03:00:08 VeNASaur Plugin Auto Update: folder.view.plg version 2024.08.04 does not meet age requirements to update - 0 days old Aug 5 03:00:08 VeNASaur Plugin Auto Update: Checking for language updates Aug 5 03:00:08 VeNASaur Plugin Auto Update: Community Applications Plugin Auto Update finished Aug 5 03:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 04:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 04:58:59 VeNASaur kernel: BUG: kernel NULL pointer dereference, address: 0000000000000038 Aug 5 04:58:59 VeNASaur kernel: #PF: supervisor read access in kernel mode Aug 5 04:58:59 VeNASaur kernel: #PF: error_code(0x0000) - not-present page Aug 5 04:58:59 VeNASaur kernel: PGD 12420d2067 P4D 12420d2067 PUD 1171b93067 PMD 0 Aug 5 04:58:59 VeNASaur kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI Aug 5 04:58:59 VeNASaur kernel: CPU: 6 PID: 15797 Comm: zfs Tainted: P O 6.1.99-Unraid #1 Aug 5 04:58:59 VeNASaur kernel: Hardware name: ASUSTeK COMPUTER INC. System Product Name/Pro WS W680-ACE IPMI, BIOS 3701 06/28/2024 Aug 5 04:58:59 VeNASaur kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf Aug 5 04:58:59 VeNASaur kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41 Aug 5 04:58:59 VeNASaur kernel: RSP: 0018:ffffc9005922f338 EFLAGS: 00010202 Aug 5 04:58:59 VeNASaur kernel: RAX: 0000000000000001 RBX: ffff8881040f2c00 RCX: 0000000000000001 Aug 5 04:58:59 VeNASaur kernel: RDX: ffffc9005922f388 RSI: 0000000000000000 RDI: ffff8881040f2c00 Aug 5 04:58:59 VeNASaur kernel: RBP: 0000000000000000 R08: ffff889369728400 R09: ffffffffa0ef76a9 Aug 5 04:58:59 VeNASaur kernel: R10: ffff889369728200 R11: ffff889369728400 R12: 0000000000000000 Aug 5 04:58:59 VeNASaur kernel: R13: ffff888f794330c0 R14: ffffc9005922f388 R15: 0000000000000000 Aug 5 04:58:59 VeNASaur kernel: FS: 0000149316472800(0000) GS:ffff889fff980000(0000) knlGS:0000000000000000 Aug 5 04:58:59 VeNASaur kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 5 04:58:59 VeNASaur kernel: CR2: 0000000000000038 CR3: 0000000fef86c002 CR4: 0000000000772ee0 Aug 5 04:58:59 VeNASaur kernel: PKRU: 55555554 Aug 5 04:58:59 VeNASaur kernel: Call Trace: Aug 5 04:58:59 VeNASaur kernel: <TASK> Aug 5 04:58:59 VeNASaur kernel: ? __die_body+0x1a/0x5c Aug 5 04:58:59 VeNASaur kernel: ? page_fault_oops+0x329/0x376 Aug 5 04:58:59 VeNASaur kernel: ? do_user_addr_fault+0x12e/0x465 Aug 5 04:58:59 VeNASaur kernel: ? exc_page_fault+0xfb/0x11d Aug 5 04:58:59 VeNASaur kernel: ? asm_exc_page_fault+0x22/0x30 Aug 5 04:58:59 VeNASaur kernel: ? spl_kmem_cache_free+0x3a/0x1a5 [spl] Aug 5 04:58:59 VeNASaur kernel: ? memcg_slab_free_hook+0x28/0xcf Aug 5 04:58:59 VeNASaur kernel: ? memcg_slab_free_hook+0x20/0xcf Aug 5 04:58:59 VeNASaur kernel: ? slab_post_alloc_hook+0x4d/0x15e Aug 5 04:58:59 VeNASaur kernel: kmem_cache_free+0xb7/0x154 Aug 5 04:58:59 VeNASaur kernel: ? spl_kmem_cache_free+0x3a/0x1a5 [spl] Aug 5 04:58:59 VeNASaur kernel: spl_kmem_cache_free+0x3a/0x1a5 [spl] Aug 5 04:58:59 VeNASaur kernel: abd_free+0x100/0x15c [zfs] Aug 5 04:58:59 VeNASaur kernel: zio_pop_transforms+0x41/0x77 [zfs] Aug 5 04:58:59 VeNASaur kernel: zio_done+0x2eb/0xc79 [zfs] Aug 5 04:58:59 VeNASaur kernel: zio_nowait+0xed/0x10a [zfs] Aug 5 04:58:59 VeNASaur kernel: arc_read+0xd78/0xf60 [zfs] Aug 5 04:58:59 VeNASaur kernel: ? dbuf_rele_and_unlock+0x4ef/0x4ef [zfs] Aug 5 04:58:59 VeNASaur kernel: dbuf_read_impl.constprop.0+0x49d/0x51c [zfs] Aug 5 04:58:59 VeNASaur kernel: dbuf_read+0x2c6/0x4da [zfs] Aug 5 04:58:59 VeNASaur kernel: dmu_buf_hold+0x4c/0x75 [zfs] Aug 5 04:58:59 VeNASaur kernel: zap_lockdir+0x4e/0xaf [zfs] Aug 5 04:58:59 VeNASaur kernel: ? _raw_spin_lock+0x13/0x1c Aug 5 04:58:59 VeNASaur kernel: zap_lookup_norm+0x5a/0xcb [zfs] Aug 5 04:58:59 VeNASaur kernel: zap_contains+0x1a/0x2f [zfs] Aug 5 04:58:59 VeNASaur kernel: dsl_dataset_hold_obj+0x2f3/0x811 [zfs] Aug 5 04:58:59 VeNASaur kernel: dsl_dataset_hold_obj_flags+0x1f/0x57 [zfs] Aug 5 04:58:59 VeNASaur kernel: dsl_dataset_hold_flags+0x82/0x225 [zfs] Aug 5 04:58:59 VeNASaur kernel: zcp_dataset_hold+0x37/0x6d [zfs] Aug 5 04:58:59 VeNASaur kernel: ? zprop_name_to_prop+0x4b/0x71 [zcommon] Aug 5 04:58:59 VeNASaur kernel: ? prop_valid_for_ds+0x99/0x99 [zfs] Aug 5 04:58:59 VeNASaur kernel: zcp_get_prop+0x3ee/0x541 [zfs] Aug 5 04:58:59 VeNASaur kernel: ? luaB_select+0x93/0x93 [zlua] Aug 5 04:58:59 VeNASaur kernel: ? luaD_precall+0xe5/0x30e [zlua] Aug 5 04:58:59 VeNASaur kernel: ? prop_valid_for_ds+0x99/0x99 [zfs] Aug 5 04:58:59 VeNASaur kernel: luaD_precall+0xcd/0x30e [zlua] Aug 5 04:58:59 VeNASaur kernel: luaV_execute+0xb99/0x1110 [zlua] Aug 5 04:58:59 VeNASaur kernel: luaD_call+0xca/0x110 [zlua] Aug 5 04:58:59 VeNASaur kernel: luaD_rawrunprotected+0x98/0xa6 [zlua] Aug 5 04:58:59 VeNASaur kernel: ? lua_setmetatable+0xe9/0xe9 [zlua] Aug 5 04:58:59 VeNASaur kernel: ? luaD_rawrunprotected+0x52/0xa6 [zlua] Aug 5 04:58:59 VeNASaur kernel: luaD_pcall+0x31/0x87 [zlua] Aug 5 04:58:59 VeNASaur kernel: lua_pcallk+0x82/0x10c [zlua] Aug 5 04:58:59 VeNASaur kernel: zcp_eval_impl+0x104/0x40d [zfs] Aug 5 04:58:59 VeNASaur kernel: zcp_eval+0x6fc/0x78b [zfs] Aug 5 04:58:59 VeNASaur kernel: ? nvlist_lookup_nvpair_ei_sep+0x22d/0x33f [znvpair] Aug 5 04:58:59 VeNASaur kernel: ? nvt_lookup_name_type.isra.0+0x44/0x6e [znvpair] Aug 5 04:58:59 VeNASaur kernel: zfs_ioc_channel_program+0x115/0x13a [zfs] Aug 5 04:58:59 VeNASaur kernel: zfsdev_ioctl_common+0x518/0x726 [zfs] Aug 5 04:58:59 VeNASaur kernel: zfsdev_ioctl+0x5b/0xb4 [zfs] Aug 5 04:58:59 VeNASaur kernel: vfs_ioctl+0x1b/0x2f Aug 5 04:58:59 VeNASaur kernel: __do_sys_ioctl+0x52/0x78 Aug 5 04:58:59 VeNASaur kernel: do_syscall_64+0x65/0x7b Aug 5 04:58:59 VeNASaur kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Aug 5 04:58:59 VeNASaur kernel: RIP: 0033:0x1493167154e8 Aug 5 04:58:59 VeNASaur kernel: Code: 00 00 48 8d 44 24 08 48 89 54 24 e0 48 89 44 24 c0 48 8d 44 24 d0 48 89 44 24 c8 b8 10 00 00 00 c7 44 24 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 07 89 d0 c3 0f 1f 40 00 48 8b 15 f9 e8 0d Aug 5 04:58:59 VeNASaur kernel: RSP: 002b:00007ffc20c52c28 EFLAGS: 00000206 ORIG_RAX: 0000000000000010 Aug 5 04:58:59 VeNASaur kernel: RAX: ffffffffffffffda RBX: 0000000000005a48 RCX: 00001493167154e8 Aug 5 04:58:59 VeNASaur kernel: RDX: 00007ffc20c52c50 RSI: 0000000000005a48 RDI: 000000000000000c Aug 5 04:58:59 VeNASaur kernel: RBP: 00007ffc20c56230 R08: 00000000ffffffff R09: 0000000000000000 Aug 5 04:58:59 VeNASaur kernel: R10: 000014931661f370 R11: 0000000000000206 R12: 00007ffc20c52c50 Aug 5 04:58:59 VeNASaur kernel: R13: 0000000000005a48 R14: 0000000000427b01 R15: 00007ffc20c56308 Aug 5 04:58:59 VeNASaur kernel: </TASK> Aug 5 04:58:59 VeNASaur kernel: Modules linked in: udp_diag nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat vhost_net tun vhost vhost_iotlb tap ipvlan veth xt_nat xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo br_netfilter xt_REDIRECT xt_tcpudp xfs md_mod tcp_diag inet_diag ipmi_devintf xt_connmark xt_mark iptable_mangle xt_comment xt_addrtype iptable_raw iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc igc nvidia_drm(PO) nvidia_modeset(PO) zfs(PO) zunicode(PO) intel_rapl_msr intel_rapl_common iosf_mbi zzstd(O) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel zlua(O) kvm zavl(PO) icp(PO) nvidia(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 ast sha256_ssse3 sha1_ssse3 Aug 5 04:58:59 VeNASaur kernel: drm_vram_helper i2c_algo_bit drm_ttm_helper aesni_intel ttm zcommon(PO) crypto_simd znvpair(PO) drm_kms_helper cryptd rapl spl(O) ipmi_ssif intel_cstate mei_hdcp mei_pxp i2c_i801 wmi_bmof drm mpt3sas agpgart i2c_smbus nvme mei_me sr_mod raid_class intel_uncore cdc_ether nvme_core scsi_transport_sas joydev cdrom ahci input_leds i2c_core mei usbnet led_class syscopyarea cp210x libahci mii sysfillrect tpm_crb acpi_ipmi sysimgblt usbserial video fb_sys_fops vmd thermal fan tpm_tis tpm_tis_core wmi ipmi_si backlight tpm intel_pmc_core acpi_pad acpi_tad button unix [last unloaded: igc] Aug 5 04:58:59 VeNASaur kernel: CR2: 0000000000000038 Aug 5 04:58:59 VeNASaur kernel: ---[ end trace 0000000000000000 ]--- Aug 5 04:58:59 VeNASaur kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf Aug 5 04:58:59 VeNASaur kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41 Aug 5 04:58:59 VeNASaur kernel: RSP: 0018:ffffc9005922f338 EFLAGS: 00010202 Aug 5 04:58:59 VeNASaur kernel: RAX: 0000000000000001 RBX: ffff8881040f2c00 RCX: 0000000000000001 Aug 5 04:58:59 VeNASaur kernel: RDX: ffffc9005922f388 RSI: 0000000000000000 RDI: ffff8881040f2c00 Aug 5 04:58:59 VeNASaur kernel: RBP: 0000000000000000 R08: ffff889369728400 R09: ffffffffa0ef76a9 Aug 5 04:58:59 VeNASaur kernel: R10: ffff889369728200 R11: ffff889369728400 R12: 0000000000000000 Aug 5 04:58:59 VeNASaur kernel: R13: ffff888f794330c0 R14: ffffc9005922f388 R15: 0000000000000000 Aug 5 04:58:59 VeNASaur kernel: FS: 0000149316472800(0000) GS:ffff889fff980000(0000) knlGS:0000000000000000 Aug 5 04:58:59 VeNASaur kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 5 04:58:59 VeNASaur kernel: CR2: 0000000000000038 CR3: 0000000fef86c002 CR4: 0000000000772ee0 Aug 5 05:00:01 VeNASaur Docker Auto Update: Community Applications Docker Autoupdate running Aug 5 05:00:01 VeNASaur Docker Auto Update: Checking for available updates Aug 5 05:00:40 VeNASaur kernel: mdcmd (39): nocheck PAUSE Aug 5 05:00:40 VeNASaur kernel: Aug 5 05:00:40 VeNASaur kernel: md: recovery thread: exit status: -4 Aug 5 05:00:50 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:01:10 VeNASaur Docker Auto Update: Installing Updates for Jellyfin Aug 5 05:04:22 VeNASaur Docker Auto Update: Community Applications Docker Autoupdate finished Aug 5 05:06:25 VeNASaur kernel: mdcmd (40): nocheck PAUSE Aug 5 05:06:25 VeNASaur kernel: Aug 5 05:06:35 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:12:44 VeNASaur kernel: mdcmd (41): nocheck PAUSE Aug 5 05:12:44 VeNASaur kernel: Aug 5 05:12:54 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:18:36 VeNASaur kernel: mdcmd (42): nocheck PAUSE Aug 5 05:18:36 VeNASaur kernel: Aug 5 05:18:46 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:24:38 VeNASaur kernel: mdcmd (43): nocheck PAUSE Aug 5 05:24:38 VeNASaur kernel: Aug 5 05:24:48 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:25:57 VeNASaur kernel: BUG: kernel NULL pointer dereference, address: 0000000000000000 Aug 5 05:25:57 VeNASaur kernel: #PF: supervisor write access in kernel mode Aug 5 05:25:57 VeNASaur kernel: #PF: error_code(0x0002) - not-present page Aug 5 05:25:57 VeNASaur kernel: PGD b0bdf5067 P4D b0bdf5067 PUD 4b0a25067 PMD 0 Aug 5 05:25:57 VeNASaur kernel: Oops: 0002 [#2] PREEMPT SMP NOPTI Aug 5 05:25:57 VeNASaur kernel: CPU: 8 PID: 28816 Comm: lsof Tainted: P D O 6.1.99-Unraid #1 Aug 5 05:25:57 VeNASaur kernel: Hardware name: ASUSTeK COMPUTER INC. System Product Name/Pro WS W680-ACE IPMI, BIOS 3701 06/28/2024 Aug 5 05:25:57 VeNASaur kernel: RIP: 0010:generic_permission+0x23/0x1d2 Aug 5 05:25:57 VeNASaur kernel: Code: d0 5b c3 cc cc cc cc 0f 1f 44 00 00 41 57 41 56 41 89 d6 41 55 49 89 fd 41 54 55 53 48 89 f3 44 0f b7 26 e8 c3 f1 ff ff 65 48 <8b> 14 25 80 cb 01 00 48 8b 92 70 06 00 00 39 42 20 75 15 41 c1 ec Aug 5 05:25:57 VeNASaur kernel: RSP: 0018:ffffc90043593c38 EFLAGS: 00010246 Aug 5 05:25:57 VeNASaur kernel: RAX: 0000000000000000 RBX: ffff888525051878 RCX: 0000000000200000 Aug 5 05:25:57 VeNASaur kernel: RDX: 0000000000000000 RSI: ffffffff8223ea20 RDI: ffffffff8223ea20 Aug 5 05:25:57 VeNASaur kernel: RBP: 0000000000000081 R08: 392f000000000000 R09: 392f6f666e696466 Aug 5 05:25:57 VeNASaur kernel: R10: 0000000000000000 R11: 0000000000000fe0 R12: 000000000000416d Aug 5 05:25:57 VeNASaur kernel: R13: ffffffff8223ea20 R14: 0000000000000081 R15: ffff8887621b1032 Aug 5 05:25:57 VeNASaur kernel: FS: 000014bbfd570e00(0000) GS:ffff889fffa00000(0000) knlGS:0000000000000000 Aug 5 05:25:57 VeNASaur kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 5 05:25:57 VeNASaur kernel: CR2: 0000000000000000 CR3: 0000000aad378005 CR4: 0000000000772ee0 Aug 5 05:25:57 VeNASaur kernel: PKRU: 55555554 Aug 5 05:25:57 VeNASaur kernel: Call Trace: Aug 5 05:25:57 VeNASaur kernel: <TASK> Aug 5 05:25:57 VeNASaur kernel: ? __die_body+0x1a/0x5c Aug 5 05:25:57 VeNASaur kernel: ? page_fault_oops+0x329/0x376 Aug 5 05:25:57 VeNASaur kernel: ? do_user_addr_fault+0x12e/0x465 Aug 5 05:25:57 VeNASaur kernel: ? __d_alloc+0x27/0x185 Aug 5 05:25:57 VeNASaur kernel: ? exc_page_fault+0xfb/0x11d Aug 5 05:25:57 VeNASaur kernel: ? asm_exc_page_fault+0x22/0x30 Aug 5 05:25:57 VeNASaur kernel: ? generic_permission+0x23/0x1d2 Aug 5 05:25:57 VeNASaur kernel: ? generic_permission+0x21/0x1d2 Aug 5 05:25:57 VeNASaur kernel: inode_permission+0xbe/0x131 Aug 5 05:25:57 VeNASaur kernel: link_path_walk+0x9d/0x308 Aug 5 05:25:57 VeNASaur kernel: ? path_init+0x107/0x2fc Aug 5 05:25:57 VeNASaur kernel: path_openat+0x129/0xa4d Aug 5 05:25:57 VeNASaur kernel: do_filp_open+0x55/0xb8 Aug 5 05:25:57 VeNASaur kernel: ? getname_flags+0x29/0x152 Aug 5 05:25:57 VeNASaur kernel: ? kmem_cache_alloc+0x122/0x14d Aug 5 05:25:57 VeNASaur kernel: ? _raw_spin_unlock+0x14/0x29 Aug 5 05:25:57 VeNASaur kernel: do_sys_openat2+0x6c/0xd9 Aug 5 05:25:57 VeNASaur kernel: do_sys_open+0x3a/0x5a Aug 5 05:25:57 VeNASaur kernel: do_syscall_64+0x65/0x7b Aug 5 05:25:57 VeNASaur kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Aug 5 05:25:57 VeNASaur kernel: RIP: 0033:0x14bbfd7fc8a1 Aug 5 05:25:57 VeNASaur kernel: Code: 75 37 89 f0 25 00 00 41 00 3d 00 00 41 00 74 29 80 3d 4a cd 0e 00 00 74 4d 89 da 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 77 48 83 c4 68 5b 5d c3 48 8d 84 24 80 00 00 Aug 5 05:25:57 VeNASaur kernel: RSP: 002b:00007ffe68f56c30 EFLAGS: 00000202 ORIG_RAX: 0000000000000101 Aug 5 05:25:57 VeNASaur kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000014bbfd7fc8a1 Aug 5 05:25:57 VeNASaur kernel: RDX: 0000000000000000 RSI: 000000000049f440 RDI: 00000000ffffff9c Aug 5 05:25:57 VeNASaur kernel: RBP: 000000000049f440 R08: 0000000000000008 R09: 0000000000000001 Aug 5 05:25:57 VeNASaur kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 000000000042b72f Aug 5 05:25:57 VeNASaur kernel: R13: 000000000049f440 R14: 0000000000433dd0 R15: 000014bbfd965000 Aug 5 05:25:57 VeNASaur kernel: </TASK> Aug 5 05:25:57 VeNASaur kernel: Modules linked in: udp_diag nvidia_uvm(PO) xt_CHECKSUM ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat vhost_net tun vhost vhost_iotlb tap ipvlan veth xt_nat xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo br_netfilter xt_REDIRECT xt_tcpudp xfs md_mod tcp_diag inet_diag ipmi_devintf xt_connmark xt_mark iptable_mangle xt_comment xt_addrtype iptable_raw iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc igc nvidia_drm(PO) nvidia_modeset(PO) zfs(PO) zunicode(PO) intel_rapl_msr intel_rapl_common iosf_mbi zzstd(O) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel zlua(O) kvm zavl(PO) icp(PO) nvidia(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 ast sha256_ssse3 sha1_ssse3 Aug 5 05:25:57 VeNASaur kernel: drm_vram_helper i2c_algo_bit drm_ttm_helper aesni_intel ttm zcommon(PO) crypto_simd znvpair(PO) drm_kms_helper cryptd rapl spl(O) ipmi_ssif intel_cstate mei_hdcp mei_pxp i2c_i801 wmi_bmof drm mpt3sas agpgart i2c_smbus nvme mei_me sr_mod raid_class intel_uncore cdc_ether nvme_core scsi_transport_sas joydev cdrom ahci input_leds i2c_core mei usbnet led_class syscopyarea cp210x libahci mii sysfillrect tpm_crb acpi_ipmi sysimgblt usbserial video fb_sys_fops vmd thermal fan tpm_tis tpm_tis_core wmi ipmi_si backlight tpm intel_pmc_core acpi_pad acpi_tad button unix [last unloaded: igc] Aug 5 05:25:57 VeNASaur kernel: CR2: 0000000000000000 Aug 5 05:25:57 VeNASaur kernel: ---[ end trace 0000000000000000 ]--- Aug 5 05:25:57 VeNASaur kernel: RIP: 0010:memcg_slab_free_hook+0x28/0xcf Aug 5 05:25:57 VeNASaur kernel: Code: cc cc 41 57 41 56 49 89 d6 41 55 41 54 55 48 89 f5 53 48 89 fb 48 83 ec 10 89 4c 24 0c e8 5a e1 ff ff 84 c0 0f 84 94 00 00 00 <4c> 8b 65 38 49 83 fc 03 0f 86 86 00 00 00 49 83 e4 fc 45 31 ed 41 Aug 5 05:25:57 VeNASaur kernel: RSP: 0018:ffffc9005922f338 EFLAGS: 00010202 Aug 5 05:25:57 VeNASaur kernel: RAX: 0000000000000001 RBX: ffff8881040f2c00 RCX: 0000000000000001 Aug 5 05:25:57 VeNASaur kernel: RDX: ffffc9005922f388 RSI: 0000000000000000 RDI: ffff8881040f2c00 Aug 5 05:25:57 VeNASaur kernel: RBP: 0000000000000000 R08: ffff889369728400 R09: ffffffffa0ef76a9 Aug 5 05:25:57 VeNASaur kernel: R10: ffff889369728200 R11: ffff889369728400 R12: 0000000000000000 Aug 5 05:25:57 VeNASaur kernel: R13: ffff888f794330c0 R14: ffffc9005922f388 R15: 0000000000000000 Aug 5 05:25:57 VeNASaur kernel: FS: 000014bbfd570e00(0000) GS:ffff889fffa00000(0000) knlGS:0000000000000000 Aug 5 05:25:57 VeNASaur kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 5 05:25:57 VeNASaur kernel: CR2: 0000000000000000 CR3: 0000000aad378005 CR4: 0000000000772ee0 Aug 5 05:25:57 VeNASaur kernel: PKRU: 55555554 Aug 5 05:25:57 VeNASaur kernel: note: lsof[28816] exited with irqs disabled Aug 5 05:30:21 VeNASaur kernel: mdcmd (44): nocheck PAUSE Aug 5 05:30:21 VeNASaur kernel: Aug 5 05:30:31 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:36:42 VeNASaur kernel: mdcmd (45): nocheck PAUSE Aug 5 05:36:42 VeNASaur kernel: Aug 5 05:36:52 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:42:43 VeNASaur kernel: mdcmd (46): nocheck PAUSE Aug 5 05:42:43 VeNASaur kernel: Aug 5 05:42:53 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:46:00 VeNASaur kernel: python3[17966]: segfault at 28 ip 000014af4fcc6f68 sp 000014aef53210b0 error 4 in libpython3.12.so.1.0[14af4fac2000+20b000] likely on CPU 6 (core 12, socket 0) Aug 5 05:46:00 VeNASaur kernel: Code: 89 44 24 38 31 c0 48 89 56 38 4d 8d bc 24 c0 00 00 00 49 8b 8c 24 a8 00 00 00 4c 29 fa 48 89 d0 48 d1 f8 4c 63 f0 4b 8d 2c 36 <48> 03 69 28 83 7f 2c 00 74 27 0f b6 45 00 48 8b 54 24 38 64 48 2b Aug 5 05:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 05:47:01 VeNASaur root: Fix Common Problems: Warning: Docker Application Jellyfin has an update available for it ** Ignored Aug 5 05:48:24 VeNASaur kernel: mdcmd (47): nocheck PAUSE Aug 5 05:48:24 VeNASaur kernel: Aug 5 05:48:34 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 05:54:47 VeNASaur kernel: mdcmd (48): nocheck PAUSE Aug 5 05:54:47 VeNASaur kernel: Aug 5 05:54:57 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) Aug 5 06:00:45 VeNASaur kernel: mdcmd (49): nocheck PAUSE Aug 5 06:00:45 VeNASaur kernel: Aug 5 06:00:55 VeNASaur Parity Check Tuning: Send notification: Paused: Scheduled Non-Correcting Parity-Check (3.7% completed) (type=normal link=/Settings/Scheduler) in this so called hung state i can see the disk activity on /main and i can navigate to /settings and /tools but if i go to /dashboard the webui is blank and it will not load, at this point i cannot navigate back to /main without having to reload php with /etc/rc.d/rc.php-fpm reload even weirder is i have homeassistant running in a VM and some docker containers that don't appear to be affected at all. but even then there are a couple of docker containers that seem to load indefinitely even though they dont seem to outright crash. From what i can see when trying to create the diagnostics it stops on the zpool status which seems to indicate there is an issue there. But then in the syslog there seems to be a lot of different stuff like segfaults, tainted kernel, and call traces. any ideas what issues i'm having here? Quote Link to comment
JorgeB Posted August 5 Share Posted August 5 Try backing up and reformatting that zfs filesystem. Quote Link to comment
MammothJerk Posted August 6 Author Share Posted August 6 overnight it did another "real crash" where i wasn't able to connect via ssh, webui, or even the physical keyboard. the syslog looked like this: Quote Aug 5 13:44:53 VeNASaur kernel: device veth49f0cbb entered promiscuous mode Aug 5 13:44:53 VeNASaur kernel: eth0: renamed from veth96fcb5a Aug 5 13:44:53 VeNASaur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth49f0cbb: link becomes ready Aug 5 13:44:53 VeNASaur kernel: docker0: port 23(veth49f0cbb) entered blocking state Aug 5 13:44:53 VeNASaur kernel: docker0: port 23(veth49f0cbb) entered forwarding state Aug 5 13:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 14:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 15:47:02 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 16:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 17:47:02 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 18:47:02 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 19:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 20:03:17 VeNASaur kernel: ffmpeg[9298]: segfault at 14f1c3d09280 ip 000014f1c8120ea2 sp 00007ffc1f8ab0c0 error 6 in libavcodec.so.60[14f1c7e0b000+cef000] likely on CPU 6 (core 12, socket 0) Aug 5 20:03:17 VeNASaur kernel: Code: 0f 11 87 e0 70 00 00 c1 e2 02 48 63 ff 4c 63 d2 48 01 f9 4a 8d 34 96 4d 63 d0 4e 8d 1c 95 00 00 00 00 41 0f 28 87 80 70 00 00 <0f> 29 06 4c 01 de 41 0f 28 87 a0 70 00 00 0f 29 06 4c 01 de 41 0f Aug 5 20:06:28 VeNASaur kernel: ffmpeg[22299]: segfault at 150b307ed140 ip 0000150b34d4fea2 sp 00007ffc40842100 error 6 in libavcodec.so.60[150b34a3a000+cef000] likely on CPU 6 (core 12, socket 0) Aug 5 20:06:28 VeNASaur kernel: Code: 0f 11 87 e0 70 00 00 c1 e2 02 48 63 ff 4c 63 d2 48 01 f9 4a 8d 34 96 4d 63 d0 4e 8d 1c 95 00 00 00 00 41 0f 28 87 80 70 00 00 <0f> 29 06 4c 01 de 41 0f 28 87 a0 70 00 00 0f 29 06 4c 01 de 41 0f Aug 5 20:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 21:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 5 22:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 ill reformat the zfs cache and see if that helps. Quote Link to comment
MammothJerk Posted August 6 Author Share Posted August 6 (edited) trying to move data off of the cache to the array using Mover my server has crashed twice Quote Aug 6 05:38:33 VeNASaur kernel: mdcmd (37): nocheck cancel Aug 6 05:38:39 VeNASaur kernel: md: recovery thread: exit status: -4 Aug 6 05:39:44 VeNASaur emhttpd: shcmd (164): /usr/local/sbin/mover &> /dev/null & Aug 6 05:47:00 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 6 05:51:12 VeNASaur kernel: BUG: unable to handle page fault for address: 0000000081265fc9 Aug 6 05:51:12 VeNASaur kernel: #PF: supervisor instruction fetch in kernel mode Aug 6 05:51:12 VeNASaur kernel: #PF: error_code(0x0010) - not-present page Aug 6 05:51:12 VeNASaur kernel: PGD 440916067 P4D 440916067 PUD 0 Aug 6 05:51:12 VeNASaur kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI Aug 6 05:51:12 VeNASaur kernel: CPU: 10 PID: 25256 Comm: lsof Tainted: P O 6.1.99-Unraid #1 Aug 6 05:51:12 VeNASaur kernel: Hardware name: ASUSTeK COMPUTER INC. System Product Name/Pro WS W680-ACE IPMI, BIOS 3701 06/28/2024 Aug 6 05:51:12 VeNASaur kernel: RIP: 0010:0x81265fc9 Aug 6 05:51:12 VeNASaur kernel: Code: Unable to access opcode bytes at 0x81265f9f. Aug 6 05:51:12 VeNASaur kernel: RSP: 0018:ffffc90009a2fbf8 EFLAGS: 00010246 Aug 6 05:51:12 VeNASaur kernel: RAX: 0000000000000000 RBX: ffff888100fb8000 RCX: 0000000000000064 Aug 6 05:51:12 VeNASaur kernel: RDX: 0000000000000003 RSI: 0000000000000002 RDI: ffff888100fb8028 Aug 6 05:51:12 VeNASaur kernel: RBP: ffffc90009a2fc98 R08: 6576696500000000 R09: 0000000000746e6d Aug 6 05:51:12 VeNASaur kernel: R10: 0000000000000000 R11: 0000000000000fe0 R12: ffffffff81e25960 Aug 6 05:51:12 VeNASaur kernel: R13: ffffc90009a2fd68 R14: ffff88817e105398 R15: ffffffff812b2bec Aug 6 05:51:12 VeNASaur kernel: FS: 000014571228ee00(0000) GS:ffff889fffa80000(0000) knlGS:0000000000000000 Aug 6 05:51:12 VeNASaur kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 6 05:51:12 VeNASaur kernel: CR2: 0000000081265fc9 CR3: 0000000634692006 CR4: 0000000000772ee0 Aug 6 05:51:12 VeNASaur kernel: PKRU: 55555554 Aug 6 05:51:12 VeNASaur kernel: Call Trace: Aug 6 05:51:12 VeNASaur kernel: <TASK> Aug 6 05:51:12 VeNASaur kernel: ? __die_body+0x1a/0x5c Aug 6 05:51:12 VeNASaur kernel: ? page_fault_oops+0x329/0x376 Aug 6 05:51:12 VeNASaur kernel: ? get_obj_cgroup_from_current+0xd7/0xe0 Aug 6 05:51:12 VeNASaur kernel: ? slab_post_alloc_hook+0x4d/0x15e Aug 6 05:51:12 VeNASaur kernel: ? exc_page_fault+0xfb/0x11d Aug 6 05:51:12 VeNASaur kernel: ? asm_exc_page_fault+0x22/0x30 Aug 6 05:51:12 VeNASaur kernel: ? proc_ns_dir_lookup+0xb5/0xb5 Aug 6 05:51:12 VeNASaur kernel: ? proc_ns_get_link+0x36/0xa6 Aug 6 05:51:12 VeNASaur kernel: ? step_into+0x48e/0x512 Aug 6 05:51:12 VeNASaur kernel: ? lookup_fast+0x70/0xc0 Aug 6 05:51:12 VeNASaur kernel: path_lookupat+0x78/0xfe Aug 6 05:51:12 VeNASaur kernel: filename_lookup+0x5f/0xbc Aug 6 05:51:12 VeNASaur kernel: ? call_rcu+0x4f0/0x5ab Aug 6 05:51:12 VeNASaur kernel: ? slab_post_alloc_hook+0x4d/0x15e Aug 6 05:51:12 VeNASaur kernel: vfs_statx+0x62/0x126 Aug 6 05:51:12 VeNASaur kernel: vfs_fstatat+0x46/0x62 Aug 6 05:51:12 VeNASaur kernel: __do_sys_newfstatat+0x26/0x5c Aug 6 05:51:12 VeNASaur kernel: ? fpregs_assert_state_consistent+0x20/0x44 Aug 6 05:51:12 VeNASaur kernel: ? exit_to_user_mode_prepare+0xd8/0x112 Aug 6 05:51:12 VeNASaur kernel: do_syscall_64+0x65/0x7b Aug 6 05:51:12 VeNASaur kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8 Aug 6 05:51:12 VeNASaur kernel: RIP: 0033:0x14571251a1ca Aug 6 05:51:12 VeNASaur kernel: Code: 48 89 f2 b9 00 01 00 00 48 89 fe bf 9c ff ff ff e9 0b 00 00 00 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca b8 06 01 00 00 0f 05 <3d> 00 f0 ff ff 77 07 31 c0 c3 0f 1f 40 00 48 8b 15 19 4c 0e 00 f7 Aug 6 05:51:12 VeNASaur kernel: RSP: 002b:00007fff608e3748 EFLAGS: 00000246 ORIG_RAX: 0000000000000106 Aug 6 05:51:12 VeNASaur kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000014571251a1ca Aug 6 05:51:12 VeNASaur kernel: RDX: 00007fff608e3760 RSI: 00007fff608e3880 RDI: 00000000ffffff9c Aug 6 05:51:12 VeNASaur kernel: RBP: 00007fff608e38f0 R08: 0000000000000064 R09: 0000000000000000 Aug 6 05:51:12 VeNASaur kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 Aug 6 05:51:12 VeNASaur kernel: R13: 00007fff608e92f0 R14: 0000000000433dd0 R15: 0000145712683000 Aug 6 05:51:12 VeNASaur kernel: </TASK> Aug 6 05:51:12 VeNASaur kernel: Modules linked in: xt_CHECKSUM xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat vhost_net tun vhost vhost_iotlb tap xt_REDIRECT xt_tcpudp xfs md_mod tcp_diag inet_diag ipmi_devintf xt_connmark xt_mark iptable_mangle xt_comment xt_addrtype iptable_raw iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha ip6table_filter ip6_tables iptable_filter ip_tables x_tables efivarfs bridge stp llc igc nvidia_drm(PO) nvidia_modeset(PO) zfs(PO) intel_rapl_msr intel_rapl_common zunicode(PO) iosf_mbi x86_pkg_temp_thermal intel_powerclamp zzstd(O) coretemp kvm_intel kvm zlua(O) zavl(PO) icp(PO) nvidia(PO) crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel sha512_ssse3 ast sha256_ssse3 sha1_ssse3 drm_vram_helper i2c_algo_bit drm_ttm_helper aesni_intel ttm zcommon(PO) crypto_simd znvpair(PO) drm_kms_helper Aug 6 05:51:12 VeNASaur kernel: cryptd rapl spl(O) ipmi_ssif intel_cstate i2c_i801 mei_pxp mei_hdcp drm agpgart cdc_ether nvme i2c_smbus wmi_bmof mpt3sas sr_mod usbnet input_leds intel_uncore mei_me raid_class tpm_crb joydev i2c_core cdrom led_class nvme_core mii scsi_transport_sas syscopyarea ahci cp210x sysfillrect mei sysimgblt libahci usbserial acpi_ipmi fb_sys_fops vmd thermal fan tpm_tis video tpm_tis_core wmi ipmi_si tpm backlight intel_pmc_core acpi_pad acpi_tad button unix [last unloaded: igc] Aug 6 05:51:12 VeNASaur kernel: CR2: 0000000081265fc9 Aug 6 05:51:12 VeNASaur kernel: ---[ end trace 0000000000000000 ]--- Aug 6 05:51:12 VeNASaur kernel: RIP: 0010:0x81265fc9 Aug 6 05:51:12 VeNASaur kernel: Code: Unable to access opcode bytes at 0x81265f9f. Aug 6 05:51:12 VeNASaur kernel: RSP: 0018:ffffc90009a2fbf8 EFLAGS: 00010246 Aug 6 05:51:12 VeNASaur kernel: RAX: 0000000000000000 RBX: ffff888100fb8000 RCX: 0000000000000064 Aug 6 05:51:12 VeNASaur kernel: RDX: 0000000000000003 RSI: 0000000000000002 RDI: ffff888100fb8028 Aug 6 05:51:12 VeNASaur kernel: RBP: ffffc90009a2fc98 R08: 6576696500000000 R09: 0000000000746e6d Aug 6 05:51:12 VeNASaur kernel: R10: 0000000000000000 R11: 0000000000000fe0 R12: ffffffff81e25960 Aug 6 05:51:12 VeNASaur kernel: R13: ffffc90009a2fd68 R14: ffff88817e105398 R15: ffffffff812b2bec Aug 6 05:51:12 VeNASaur kernel: FS: 000014571228ee00(0000) GS:ffff889fffa80000(0000) knlGS:0000000000000000 Aug 6 05:51:12 VeNASaur kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Aug 6 05:51:12 VeNASaur kernel: CR2: 0000000081265fc9 CR3: 0000000634692006 CR4: 0000000000772ee0 Aug 6 05:51:12 VeNASaur kernel: PKRU: 55555554 Aug 6 05:51:12 VeNASaur kernel: note: lsof[25256] exited with irqs disabled Edited August 6 by MammothJerk Quote Link to comment
JorgeB Posted August 6 Share Posted August 6 There may be other issues, I would start by running memtest. Quote Link to comment
MammothJerk Posted August 6 Author Share Posted August 6 (edited) I moved everything off using mc, then reformatted the cache, no issues. I then started the mover to move everything back to the cache, and the server crashed 🤔 I'll run memtest and get back to you... Edited August 6 by MammothJerk Quote Link to comment
Solution JorgeB Posted August 6 Solution Share Posted August 6 I see you are using 14900K, could be the Intel instability issue. Quote Link to comment
MammothJerk Posted August 6 Author Share Posted August 6 27 minutes ago, JorgeB said: I see you are using 14900K, could be the Intel instability issue. yea thats what i'm afraid of. i updated the bios to 3701 Quote Updated with microcode 0x125 to ensure eTVB operates within Intel specifications but it mightve been too late and damaged the CPU. any ways i can test for it or would i need to start a return? memtest ran 1 pass with no errors i let it run overnight and it got up to 5 passes without errors Whats next? Quote Link to comment
MammothJerk Posted August 6 Author Share Posted August 6 (edited) Formatted the cache as btrfs and started mover to try to move stuff back over... instantly crashed would appear that the issue is with either unraid or my storage... Edited August 6 by MammothJerk Quote Link to comment
ogr Posted August 7 Share Posted August 7 (edited) I'm no expert and only here cause I'm experiencing my own crashes, but maybe roll back to 6.12.10? Or did you experience these crashes also before the latest update? Edited August 7 by ogr Quote Link to comment
MammothJerk Posted August 7 Author Share Posted August 7 26 minutes ago, ogr said: I'm no expert and only here cause I'm experiencing my own crashes, but maybe roll back to 6.12.10? Or did you experience these crashes also before the latest update? i updated to 6.12.11 when it was released and my server started crashing this friday so i asssume that is not the issue, i will most likely do a rollback if my other troubleshooting does not fix the problem. Quote Link to comment
JorgeB Posted August 7 Share Posted August 7 9 hours ago, MammothJerk said: any ways i can test for it or would i need to start a return? Unless you have a different CPU to test with it may not be easy to confirm. Quote Link to comment
MammothJerk Posted August 8 Author Share Posted August 8 Since it seemed like the mover was causing the crash i searched around for previous issues pertaining to the mover crashing the server. I tried running xfs repair on all the disks in maintenance mode, a couple of them seem to have needed that with 10~ inodes being fixed. running "docker safe new perms" which seemed to get stuck at /disk13/Storage for over 15 hours before i decided cancel it. increasing the setting "minimum free space" for all my shares, i believe i set them when i created the server and didn't touch since. I then enabled mover logging and it ran for 5 minutes before crashing the server, stopping on disk13/appdata/Radarr/MediaCover/2018/poster-500.jpg so disk13 is mentioned in 2 issues, might be a coincidence. the disc is only about 3 months old and doesn't seem to have any obvious issues 1 Raw read error rate 0x000b 100 100 001 Pre-fail Always Never 0 2 Throughput performance 0x0004 148 148 000 Old age Offline Never 49 3 Spin up time 0x0007 094 094 001 Pre-fail Always Never 145 (average 137) 4 Start stop count 0x0012 100 100 000 Old age Always Never 62 5 Reallocated sector count 0x0033 100 100 001 Pre-fail Always Never 0 7 Seek error rate 0x000a 100 100 000 Old age Always Never 0 8 Seek time performance 0x0004 140 140 000 Old age Offline Never 15 9 Power on hours 0x0012 100 100 000 Old age Always Never 2610 (3m, 17d, 18h) 10 Spin retry count 0x0012 100 100 000 Old age Always Never 0 12 Power cycle count 0x0032 100 100 000 Old age Always Never 62 22 Helium level 0x0023 100 100 025 Pre-fail Always Never 6553700 90 NAND master 0x0031 100 100 001 Pre-fail Offline Never 0x004400000000 192 Power-off retract count 0x0032 100 100 000 Old age Always Never 214 193 Load cycle count 0x0012 100 100 000 Old age Always Never 214 194 Temperature celsius 0x0002 049 049 000 Old age Always Never 45 (min/max 25/56) 196 Reallocated event count 0x0032 100 100 000 Old age Always Never 0 197 Current pending sector 0x0022 100 100 000 Old age Always Never 0 198 Offline uncorrectable 0x0008 100 100 000 Old age Offline Never 0 199 UDMA CRC error count 0x000a 100 100 000 Old age Always Never 0 I'm currently running an extended self-test but im doubtful that will tell me anything. Anything else i can do to troubleshoot Mover if that indeed is the issue? Quote Link to comment
JorgeB Posted August 8 Share Posted August 8 Unlikely that a disk issue would crash the server, and don't see anything logged to suggest a disk is the problem, but you can disable the mover for a few days and see if it still crashes. Quote Link to comment
MammothJerk Posted August 12 Author Share Posted August 12 (edited) The server ran fine for 4 days and 12 hours~ and then out of nowhere it did another hard crash where keyboard, webui, ssh, containers unresponsive, etc. Had to do a hard reset Syslog for the past couple hours: Quote Aug 12 10:35:29 VeNASaur kernel: docker0: port 24(veth5a26b80) entered blocking state Aug 12 10:35:29 VeNASaur kernel: docker0: port 24(veth5a26b80) entered disabled state Aug 12 10:35:29 VeNASaur kernel: device veth5a26b80 entered promiscuous mode Aug 12 10:35:32 VeNASaur kernel: eth0: renamed from veth9d04202 Aug 12 10:35:32 VeNASaur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth5a26b80: link becomes ready Aug 12 10:35:32 VeNASaur kernel: docker0: port 24(veth5a26b80) entered blocking state Aug 12 10:35:32 VeNASaur kernel: docker0: port 24(veth5a26b80) entered forwarding state Aug 12 10:35:45 VeNASaur kernel: vethdd3a03c: renamed from eth0 Aug 12 10:35:45 VeNASaur kernel: docker0: port 2(veth51fd1ef) entered disabled state Aug 12 10:35:45 VeNASaur kernel: docker0: port 2(veth51fd1ef) entered disabled state Aug 12 10:35:45 VeNASaur kernel: device veth51fd1ef left promiscuous mode Aug 12 10:35:45 VeNASaur kernel: docker0: port 2(veth51fd1ef) entered disabled state Aug 12 10:35:45 VeNASaur kernel: docker0: port 2(vethedd0394) entered blocking state Aug 12 10:35:45 VeNASaur kernel: docker0: port 2(vethedd0394) entered disabled state Aug 12 10:35:45 VeNASaur kernel: device vethedd0394 entered promiscuous mode Aug 12 10:35:45 VeNASaur kernel: docker0: port 2(vethedd0394) entered blocking state Aug 12 10:35:45 VeNASaur kernel: docker0: port 2(vethedd0394) entered forwarding state Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:45 VeNASaur kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: AE_ALREADY_EXISTS, CreateBufferField failure (20220331/dswload2-477) Aug 12 10:35:45 VeNASaur kernel: ACPI Error: Aborting method \_SB.PC00.PEG1.PEGP._DSM due to previous error (AE_ALREADY_EXISTS) (20220331/psparse-529) Aug 12 10:35:46 VeNASaur kernel: eth0: renamed from vethbaa2ed1 Aug 12 10:35:46 VeNASaur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethedd0394: link becomes ready Aug 12 10:35:52 VeNASaur kernel: docker0: port 25(vethb00c81f) entered blocking state Aug 12 10:35:52 VeNASaur kernel: docker0: port 25(vethb00c81f) entered disabled state Aug 12 10:35:52 VeNASaur kernel: device vethb00c81f entered promiscuous mode Aug 12 10:35:52 VeNASaur kernel: docker0: port 25(vethb00c81f) entered blocking state Aug 12 10:35:52 VeNASaur kernel: docker0: port 25(vethb00c81f) entered forwarding state Aug 12 10:35:52 VeNASaur kernel: docker0: port 25(vethb00c81f) entered disabled state Aug 12 10:35:52 VeNASaur kernel: eth0: renamed from veth03d5c13 Aug 12 10:35:52 VeNASaur kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethb00c81f: link becomes ready Aug 12 10:35:52 VeNASaur kernel: docker0: port 25(vethb00c81f) entered blocking state Aug 12 10:35:52 VeNASaur kernel: docker0: port 25(vethb00c81f) entered forwarding state Aug 12 10:47:02 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 12 10:47:14 VeNASaur root: Fix Common Problems: Other Warning: Mover logging is enabled Aug 12 11:12:27 VeNASaur kernel: ffmpeg[15794]: segfault at 149e841632d0 ip 0000149e863f9ea2 sp 00007fffa7d73030 error 7 in libavcodec.so.60[149e860e4000+cef000] likely on CPU 6 (core 12, socket 0) Aug 12 11:12:27 VeNASaur kernel: Code: 0f 11 87 e0 70 00 00 c1 e2 02 48 63 ff 4c 63 d2 48 01 f9 4a 8d 34 96 4d 63 d0 4e 8d 1c 95 00 00 00 00 41 0f 28 87 80 70 00 00 <0f> 29 06 4c 01 de 41 0f 28 87 a0 70 00 00 0f 29 06 4c 01 de 41 0f Aug 12 11:44:39 VeNASaur sshd-session[17281]: Connection from 192.168.1.100 port 51318 on 192.168.1.111 port 22 rdomain "" Aug 12 11:44:39 VeNASaur sshd-session[17281]: Accepted key Aug 12 11:44:39 VeNASaur sshd-session[17281]: Postponed publickey for root from 192.168.1.100 port 51318 ssh2 [preauth] Aug 12 11:44:39 VeNASaur sshd-session[17281]: Accepted key Aug 12 11:44:39 VeNASaur sshd-session[17281]: Accepted publickey Aug 12 11:44:39 VeNASaur sshd-session[17281]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) Aug 12 11:44:39 VeNASaur sshd-session[17281]: User child is on pid 17290 Aug 12 11:44:39 VeNASaur sshd-session[17290]: Starting session: shell on pts/1 for root from 192.168.1.100 port 51318 id 0 Aug 12 11:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 12 11:47:23 VeNASaur root: Fix Common Problems: Other Warning: Mover logging is enabled Aug 12 12:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 12 12:47:15 VeNASaur root: Fix Common Problems: Other Warning: Mover logging is enabled Aug 12 13:45:44 VeNASaur sshd-session[17290]: Connection closed by 192.168.1.100 port 51318 Aug 12 13:45:44 VeNASaur sshd-session[17290]: Close session: user root from 192.168.1.100 port 51318 id 0 Aug 12 13:45:44 VeNASaur sshd-session[17290]: Transferred: sent 69495280, received 146312 bytes Aug 12 13:45:44 VeNASaur sshd-session[17290]: Closing connection to 192.168.1.100 port 51318 Aug 12 13:45:44 VeNASaur sshd-session[17281]: pam_unix(sshd:session): session closed for user root Aug 12 13:47:01 VeNASaur root: Fix Common Problems Version 2024.07.18 Aug 12 13:47:13 VeNASaur root: Fix Common Problems: Other Warning: Mover logging is enabled The crash happened a couple of minutes after i finished manually moving a couple of things (400GB~) off of the cache to the array. The mover is disabled. the self test didn't find any errors either. Could it be a bad PSU? edit: just saw my motherboard got ANOTHER BIOS update (beta 3802) though it specifically mentions a fix for non-k processors - I'll install that as well and see what happens... Edited August 12 by MammothJerk Quote Link to comment
MammothJerk Posted August 15 Author Share Posted August 15 Another full crash after 2 days 3 hours of uptime, no special activities were taking place (no mover, no manual moves, no parity check). I did get a troubling message yesterday about the 3.3V voltage sensor going off Quote August 2024 27 minutes ago ID: 94 3.3V Voltage sensor of type voltage logged a lower non critical going low deasserted on Wednesday, August 14th 2024, 5:54:30 pm 27 minutes ago ID: 93 3.3V Voltage sensor of type voltage logged a lower critical going low deasserted on Wednesday, August 14th 2024, 5:54:30 pm 27 minutes ago ID: 92 3.3V Voltage sensor of type voltage logged a lower critical going low asserted on Wednesday, August 14th 2024, 5:54:29 pm 27 minutes ago ID: 91 3.3V Voltage sensor of type voltage logged a lower non critical going low asserted on Wednesday, August 14th 2024, 5:54:29 pm after searching the logs it appears i got a warning on the 28th of july as well, this time on the 12V voltage sensor Quote July 2024 17 days ago ID: 10 12V Voltage sensor of type voltage logged a lower non critical going low deasserted on Sunday, July 28th 2024, 2:56:20 pm 17 days ago ID: 9 12V Voltage sensor of type voltage logged a lower critical going low deasserted on Sunday, July 28th 2024, 2:56:20 pm 17 days ago ID: 8 12V Voltage sensor of type voltage logged a lower critical going low asserted on Sunday, July 28th 2024, 2:56:19 pm 17 days ago ID: 7 12V Voltage sensor of type voltage logged a lower non critical going low asserted on Sunday, July 28th 2024, 2:56:19 pm I do have an extra PSU lying around, ill change it out and see if that fixes it... Quote Link to comment
MammothJerk Posted August 16 Author Share Posted August 16 different PSU... still hard crash after 30~ hours 😬 Quote Link to comment
MammothJerk Posted August 17 Author Share Posted August 17 I have an inkling that it's related to my HA VM. i think i've had it twice now that i trigger a lightswitch and shortly thereafter the server crashes. it might be the "USB manager" plugin i use to add a zwave and zigbee dongle to the same VM even though they have the same name in unraid, ill just add them in xml instead and see. Quote Link to comment
MammothJerk Posted August 18 Author Share Posted August 18 was a long shot and it didnt work. I cant really think of anything else to try... It does seem like the server crashes more frequently now so that might hint that the CPU is even further degraded? the only error message i'm receiving is fro the IPMI Quote ID: 115 Unknown sensor of type os_stop_or_shutdown logged a run time critical stop but that doesn't really tell me what the issue is... Any other ideas @JorgeB? Quote Link to comment
JorgeB Posted August 18 Share Posted August 18 I still think it's a hardware problem, possibly the Intel 13/14th gen stability issue. 1 Quote Link to comment
MammothJerk Posted August 19 Author Share Posted August 19 (edited) I started the RMA process and should receive the replacement CPU within the week.. I tried disabling docker and VMs and just running mover for the past 24 hours and it did not result in a crash, i turn on VMs (HAOS) and it results in a "hung" state within 4 hours. This time i was able to grab diagnostics since i destroyed the zpool. Not sure if it shows anything or if it's a different issue altogether. Edited August 19 by MammothJerk Quote Link to comment
JorgeB Posted August 19 Share Posted August 19 Shows a kvm related crash, which can or not be hardware related. 1 Quote Link to comment
MammothJerk Posted August 23 Author Share Posted August 23 New cpu installed. i'll try to run a parity check now and see what happens. Quote Link to comment
MammothJerk Posted August 26 Author Share Posted August 26 haven't had any crashes since, mover has mover 3TB and parity check finished (with lots of errors 🥲) and the server has not crashed. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.