sens_dep Posted September 16, 2024 Posted September 16, 2024 (edited) updated earlier tonight and now my system fails to mount. Tried rolling back, disabling docker, removing a stick of ram, etc... not sure how to proceed. diag attached. showing weird drive data in the dashboard too: Intel 14th gen failure? here is the panic when starting the array: Sep 15 19:48:02 GhostNAS emhttpd: shcmd (432): udevadm settle Sep 15 19:48:02 GhostNAS emhttpd: Opening encrypted volumes... Sep 15 19:48:02 GhostNAS emhttpd: shcmd (433): touch /boot/config/forcesync Sep 15 19:48:02 GhostNAS emhttpd: Mounting disks... Sep 15 19:48:02 GhostNAS emhttpd: mounting /mnt/disk1 Sep 15 19:48:02 GhostNAS emhttpd: shcmd (434): mkdir -p /mnt/disk1 Sep 15 19:48:02 GhostNAS emhttpd: shcmd (435): mount -t xfs -o noatime,nouuid /dev/md1p1 /mnt/disk1 Sep 15 19:48:02 GhostNAS kernel: SGI XFS with ACLs, security attributes, no debug enabled Sep 15 19:48:02 GhostNAS kernel: XFS (md1p1): Mounting V5 Filesystem Sep 15 19:48:03 GhostNAS kernel: XFS (md1p1): Starting recovery (logdev: internal) Sep 15 19:48:03 GhostNAS emhttpd: shcmd (436): xfs_growfs /mnt/disk1 Sep 15 19:48:03 GhostNAS kernel: XFS (md1p1): Ending recovery (logdev: internal) Sep 15 19:48:03 GhostNAS root: meta-data=/dev/md1p1 isize=512 agcount=13, agsize=268435455 blks Sep 15 19:48:03 GhostNAS root: = sectsz=512 attr=2, projid32bit=1 Sep 15 19:48:03 GhostNAS root: = crc=1 finobt=1, sparse=1, rmapbt=0 Sep 15 19:48:03 GhostNAS root: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 Sep 15 19:48:03 GhostNAS root: data = bsize=4096 blocks=3418095603, imaxpct=5 Sep 15 19:48:03 GhostNAS root: = sunit=0 swidth=0 blks Sep 15 19:48:03 GhostNAS root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Sep 15 19:48:03 GhostNAS root: log =internal log bsize=4096 blocks=521728, version=2 Sep 15 19:48:03 GhostNAS root: = sectsz=512 sunit=0 blks, lazy-count=1 Sep 15 19:48:03 GhostNAS root: realtime =none extsz=4096 blocks=0, rtextents=0 Sep 15 19:48:03 GhostNAS emhttpd: mounting /mnt/disk3 Sep 15 19:48:03 GhostNAS emhttpd: shcmd (437): mkdir -p /mnt/disk3 Sep 15 19:48:03 GhostNAS emhttpd: shcmd (438): mount -t xfs -o noatime,nouuid /dev/md3p1 /mnt/disk3 Sep 15 19:48:03 GhostNAS kernel: XFS (md3p1): Mounting V5 Filesystem Sep 15 19:48:03 GhostNAS kernel: XFS (md3p1): Starting recovery (logdev: internal) Sep 15 19:48:03 GhostNAS emhttpd: shcmd (439): xfs_growfs /mnt/disk3 Sep 15 19:48:03 GhostNAS kernel: XFS (md3p1): Ending recovery (logdev: internal) Sep 15 19:48:03 GhostNAS root: meta-data=/dev/md3p1 isize=512 agcount=8, agsize=268435455 blks Sep 15 19:48:03 GhostNAS root: = sectsz=512 attr=2, projid32bit=1 Sep 15 19:48:03 GhostNAS root: = crc=1 finobt=1, sparse=1, rmapbt=0 Sep 15 19:48:03 GhostNAS root: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 Sep 15 19:48:03 GhostNAS root: data = bsize=4096 blocks=1953506633, imaxpct=5 Sep 15 19:48:03 GhostNAS root: = sunit=0 swidth=0 blks Sep 15 19:48:03 GhostNAS root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Sep 15 19:48:03 GhostNAS root: log =internal log bsize=4096 blocks=521728, version=2 Sep 15 19:48:03 GhostNAS root: = sectsz=512 sunit=0 blks, lazy-count=1 Sep 15 19:48:03 GhostNAS root: realtime =none extsz=4096 blocks=0, rtextents=0 Sep 15 19:48:03 GhostNAS emhttpd: mounting /mnt/disk4 Sep 15 19:48:03 GhostNAS emhttpd: shcmd (440): mkdir -p /mnt/disk4 Sep 15 19:48:03 GhostNAS emhttpd: shcmd (441): mount -t xfs -o noatime,nouuid /dev/md4p1 /mnt/disk4 Sep 15 19:48:03 GhostNAS kernel: XFS (md4p1): Mounting V5 Filesystem Sep 15 19:48:04 GhostNAS kernel: XFS (md4p1): Starting recovery (logdev: internal) Sep 15 19:48:04 GhostNAS emhttpd: shcmd (442): xfs_growfs /mnt/disk4 Sep 15 19:48:04 GhostNAS kernel: XFS (md4p1): Ending recovery (logdev: internal) Sep 15 19:48:04 GhostNAS root: meta-data=/dev/md4p1 isize=512 agcount=8, agsize=268435455 blks Sep 15 19:48:04 GhostNAS root: = sectsz=512 attr=2, projid32bit=1 Sep 15 19:48:04 GhostNAS root: = crc=1 finobt=1, sparse=1, rmapbt=0 Sep 15 19:48:04 GhostNAS root: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 Sep 15 19:48:04 GhostNAS root: data = bsize=4096 blocks=1953506633, imaxpct=5 Sep 15 19:48:04 GhostNAS root: = sunit=0 swidth=0 blks Sep 15 19:48:04 GhostNAS root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Sep 15 19:48:04 GhostNAS root: log =internal log bsize=4096 blocks=521728, version=2 Sep 15 19:48:04 GhostNAS root: = sectsz=512 sunit=0 blks, lazy-count=1 Sep 15 19:48:04 GhostNAS root: realtime =none extsz=4096 blocks=0, rtextents=0 Sep 15 19:48:04 GhostNAS emhttpd: mounting /mnt/cache Sep 15 19:48:04 GhostNAS emhttpd: shcmd (443): mkdir -p /mnt/cache Sep 15 19:48:04 GhostNAS emhttpd: shcmd (444): /usr/sbin/zpool import -f -N -o autoexpand=on -d /dev/nvme0n1p1 -d /dev/nvme1n1p1 811097023252368398 cache Sep 15 19:48:10 GhostNAS kernel: VERIFY3(size <= rt->rt_space) failed (281442900340736 <= 16139841536) Sep 15 19:48:10 GhostNAS kernel: PANIC at range_tree.c:436:range_tree_remove_impl() Sep 15 19:48:10 GhostNAS kernel: Showing stack for process 19845 Sep 15 19:48:10 GhostNAS kernel: CPU: 8 PID: 19845 Comm: z_wr_iss Tainted: P O 6.1.79-Unraid #1 Sep 15 19:48:10 GhostNAS kernel: Hardware name: Micro-Star International Co., Ltd. MS-7D29/MEG Z690I UNIFY (MS-7D29), BIOS 1.I0 07/26/2024 Sep 15 19:48:10 GhostNAS kernel: Call Trace: Sep 15 19:48:10 GhostNAS kernel: <TASK> Sep 15 19:48:10 GhostNAS kernel: dump_stack_lvl+0x44/0x5c Sep 15 19:48:10 GhostNAS kernel: spl_panic+0xd0/0xe8 [spl] Sep 15 19:48:10 GhostNAS kernel: ? bt_grow_leaf+0xc3/0xd6 [zfs] Sep 15 19:48:10 GhostNAS kernel: ? zfs_btree_find_in_buf+0x4c/0x94 [zfs] Sep 15 19:48:10 GhostNAS kernel: ? zfs_btree_find+0x16d/0x1b0 [zfs] Sep 15 19:48:10 GhostNAS kernel: ? rs_get_start+0xc/0x1d [zfs] Sep 15 19:48:10 GhostNAS kernel: range_tree_remove_impl+0x77/0x406 [zfs] Sep 15 19:48:10 GhostNAS kernel: ? zio_wait+0x1ee/0x1fd [zfs] Sep 15 19:48:10 GhostNAS kernel: space_map_load_callback+0x70/0x79 [zfs] Sep 15 19:48:10 GhostNAS kernel: space_map_iterate+0x2d3/0x324 [zfs] Sep 15 19:48:10 GhostNAS kernel: ? spa_stats_destroy+0x16c/0x16c [zfs] Sep 15 19:48:10 GhostNAS kernel: space_map_load_length+0x93/0xcb [zfs] Sep 15 19:48:10 GhostNAS kernel: metaslab_load+0x33b/0x6e3 [zfs] Sep 15 19:48:10 GhostNAS kernel: ? slab_post_alloc_hook+0x4d/0x15e Sep 15 19:48:10 GhostNAS kernel: ? rs_get_start+0xc/0x1d [zfs] Sep 15 19:48:10 GhostNAS kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl] Sep 15 19:48:10 GhostNAS kernel: ? __kmem_cache_alloc_node+0x118/0x147 Sep 15 19:48:10 GhostNAS kernel: metaslab_activate+0x36/0x1f1 [zfs] Sep 15 19:48:10 GhostNAS kernel: metaslab_alloc_dva+0x8bc/0xfce [zfs] Sep 15 19:48:10 GhostNAS kernel: ? preempt_latency_start+0x2b/0x46 Sep 15 19:48:10 GhostNAS kernel: metaslab_alloc+0x107/0x1fd [zfs] Sep 15 19:48:10 GhostNAS kernel: zio_dva_allocate+0xee/0x73f [zfs] Sep 15 19:48:10 GhostNAS kernel: ? kmem_cache_free+0xc9/0x154 Sep 15 19:48:10 GhostNAS kernel: ? spl_kmem_cache_free+0x3a/0x1a5 [spl] Sep 15 19:48:10 GhostNAS kernel: ? preempt_latency_start+0x2b/0x46 Sep 15 19:48:10 GhostNAS kernel: ? _raw_spin_lock+0x13/0x1c Sep 15 19:48:10 GhostNAS kernel: ? _raw_spin_unlock+0x14/0x29 Sep 15 19:48:10 GhostNAS kernel: ? tsd_hash_search+0x70/0x7d [spl] Sep 15 19:48:10 GhostNAS kernel: zio_execute+0xb1/0xdf [zfs] Sep 15 19:48:10 GhostNAS kernel: taskq_thread+0x266/0x38a [spl] Sep 15 19:48:10 GhostNAS kernel: ? wake_up_q+0x44/0x44 Sep 15 19:48:10 GhostNAS kernel: ? zio_subblock+0x22/0x22 [zfs] Sep 15 19:48:10 GhostNAS kernel: ? taskq_dispatch_delay+0x106/0x106 [spl] Sep 15 19:48:10 GhostNAS kernel: kthread+0xe4/0xef Sep 15 19:48:10 GhostNAS kernel: ? kthread_complete_and_exit+0x1b/0x1b Sep 15 19:48:10 GhostNAS kernel: ret_from_fork+0x1f/0x30 Sep 15 19:48:10 GhostNAS kernel: </TASK> ghostnas-diagnostics-20240915-2140.zip Edited September 16, 2024 by sens_dep Quote
sens_dep Posted September 16, 2024 Author Posted September 16, 2024 Update. Was able to start the array after mounting the cache in read only disable auto start of array reboot mount cache read only: zpool import -o readonly=on cache start the array seems fine? would the path here be to backup all the cache and reformat? thanks Quote
JorgeB Posted September 16, 2024 Posted September 16, 2024 6 hours ago, sens_dep said: would the path here be to backup all the cache and reformat? Yep Quote
sens_dep Posted September 16, 2024 Author Posted September 16, 2024 2 hours ago, JorgeB said: Yep What causes this? It seemed nominal before I triggered the update. Any way to monitor this / be more prepared in the future? Quote
JorgeB Posted September 16, 2024 Posted September 16, 2024 Difficult to say for certain, could be a hardware issue, like bad RAM, device/controller firmware, or a ZFS issue/bug, but if it keeps happening I would bet on the former. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.