July 24, 20232 yr After a power loss, I have not been able to start the array, i have isolated the problem to when the system is trying to mount the "mainzfs" pool and have successfullyu started the array with all drives in "mainzfs" unassigned Jul 24 19:08:07 KWUN emhttpd: shcmd (249): touch /boot/config/forcesync Jul 24 19:08:07 KWUN emhttpd: Mounting disks... Jul 24 19:08:07 KWUN emhttpd: mounting /mnt/disk1 Jul 24 19:08:07 KWUN emhttpd: shcmd (250): mkdir -p /mnt/disk1 Jul 24 19:08:07 KWUN emhttpd: shcmd (251): mount -t xfs -o noatime,nouuid /dev/md1p1 /mnt/disk1 Jul 24 19:08:08 KWUN kernel: SGI XFS with ACLs, security attributes, no debug enabled Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Mounting V5 Filesystem Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Starting recovery (logdev: internal) Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Ending recovery (logdev: internal) Jul 24 19:08:08 KWUN emhttpd: shcmd (252): xfs_growfs /mnt/disk1 Jul 24 19:08:08 KWUN root: meta-data=/dev/md1p1 isize=512 agcount=4, agsize=7551902 blks Jul 24 19:08:08 KWUN root: = sectsz=512 attr=2, projid32bit=1 Jul 24 19:08:08 KWUN root: = crc=1 finobt=1, sparse=1, rmapbt=0 Jul 24 19:08:08 KWUN root: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 Jul 24 19:08:08 KWUN root: data = bsize=4096 blocks=30207606, imaxpct=25 Jul 24 19:08:08 KWUN root: = sunit=0 swidth=0 blks Jul 24 19:08:08 KWUN root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jul 24 19:08:08 KWUN root: log =internal log bsize=4096 blocks=16384, version=2 Jul 24 19:08:08 KWUN root: = sectsz=512 sunit=0 blks, lazy-count=1 Jul 24 19:08:08 KWUN root: realtime =none extsz=4096 blocks=0, rtextents=0 Jul 24 19:08:08 KWUN emhttpd: mounting /mnt/big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (253): mkdir -p /mnt/big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (254): /usr/sbin/zpool import -N -o autoexpand=on -d /dev/sdc1 -d /dev/sdd1 8947813539389484345 big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (255): /usr/sbin/zfs set mountpoint=/mnt/big_cache big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (256): /usr/sbin/zfs set atime=off big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (257): /usr/sbin/zfs mount big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (258): /usr/sbin/zpool set autotrim=on big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (259): /usr/sbin/zfs set compression=on big_cache Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root profile: Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root groups: 2 Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root width: 1 Jul 24 19:08:08 KWUN emhttpd: /usr/sbin/zpool status -PL big_cache 2>&1 Jul 24 19:08:08 KWUN emhttpd: pool: big_cache Jul 24 19:08:08 KWUN emhttpd: state: ONLINE Jul 24 19:08:08 KWUN emhttpd: config: Jul 24 19:08:08 KWUN emhttpd: NAME STATE READ WRITE CKSUM Jul 24 19:08:08 KWUN emhttpd: big_cache ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: /dev/sdc1 ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: /dev/sdd1 ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: errors: No known data errors Jul 24 19:08:08 KWUN emhttpd: mounting /mnt/mainzfs Jul 24 19:08:08 KWUN emhttpd: shcmd (260): mkdir -p /mnt/mainzfs Jul 24 19:08:08 KWUN emhttpd: /sbin/btrfs filesystem show /dev/sdj1 2>&1 Jul 24 19:08:08 KWUN emhttpd: ERROR: no btrfs on /dev/sdj1 Jul 24 19:08:08 KWUN emhttpd: /usr/sbin/zpool import -d /dev/sdj1 2>&1 Jul 24 19:08:09 KWUN emhttpd: pool: mainzfs Jul 24 19:08:09 KWUN emhttpd: id: 17865510998351045675 Jul 24 19:08:09 KWUN emhttpd: shcmd (261): /usr/sbin/zpool import -N -o autoexpand=on -d /dev/sdj1 -d /dev/sdf1 -d /dev/sdg1 -d /dev/sdh1 -d /dev/sdm1 -d /dev/sdk1 -d /dev/sdi1 -d /dev/sdl1 17865510998351045675 mainzfs Jul 24 19:08:12 KWUN kernel: VERIFY3(range_tree_space(smla->smla_rt) + sme->sme_run <= smla->smla_sm->sm_size) failed (17237200896 <= 17179869184) Jul 24 19:08:12 KWUN kernel: PANIC at space_map.c:405:space_map_load_callback() Jul 24 19:08:12 KWUN kernel: Showing stack for process 6653 Jul 24 19:08:12 KWUN kernel: CPU: 1 PID: 6653 Comm: z_wr_iss Tainted: P O 6.1.38-Unraid #2 Jul 24 19:08:12 KWUN kernel: Hardware name: Micro-Star International Co., Ltd MS-7C02/B450 TOMAHAWK MAX II (MS-7C02), BIOS H.90 07/25/2022 Jul 24 19:08:12 KWUN kernel: Call Trace: Jul 24 19:08:12 KWUN kernel: <TASK> Jul 24 19:08:12 KWUN kernel: dump_stack_lvl+0x44/0x5c Jul 24 19:08:12 KWUN kernel: spl_panic+0xd0/0xe8 [spl] Jul 24 19:08:12 KWUN kernel: ? rs_get_start+0xc/0x1d [zfs] Jul 24 19:08:12 KWUN kernel: ? range_tree_stat_incr+0x28/0x43 [zfs] Jul 24 19:08:12 KWUN kernel: ? range_tree_remove_impl+0x3b7/0x406 [zfs] Jul 24 19:08:12 KWUN kernel: ? zio_wait+0x1ee/0x1fd [zfs] Jul 24 19:08:12 KWUN kernel: space_map_load_callback+0x50/0x79 [zfs] Jul 24 19:08:12 KWUN kernel: space_map_iterate+0x2d6/0x324 [zfs] Jul 24 19:08:12 KWUN kernel: ? spa_stats_destroy+0x16c/0x16c [zfs] Jul 24 19:08:12 KWUN kernel: space_map_load_length+0x93/0xcb [zfs] Jul 24 19:08:12 KWUN kernel: metaslab_load+0x33b/0x6e3 [zfs] Jul 24 19:08:12 KWUN kernel: ? slab_post_alloc_hook+0x4d/0x15e Jul 24 19:08:12 KWUN kernel: ? rs_get_start+0xc/0x1d [zfs] Jul 24 19:08:12 KWUN kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl] Jul 24 19:08:12 KWUN kernel: ? __kmem_cache_alloc_node+0x118/0x147 Jul 24 19:08:12 KWUN kernel: metaslab_activate+0x36/0x1f1 [zfs] Jul 24 19:08:12 KWUN kernel: metaslab_alloc_dva+0x8bc/0xfce [zfs] Jul 24 19:08:12 KWUN kernel: ? preempt_latency_start+0x2b/0x46 Jul 24 19:08:12 KWUN kernel: metaslab_alloc+0x107/0x1fd [zfs] Jul 24 19:08:12 KWUN kernel: zio_dva_allocate+0xee/0x73f [zfs] Jul 24 19:08:12 KWUN kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl] Jul 24 19:08:12 KWUN kernel: ? preempt_latency_start+0x2b/0x46 Jul 24 19:08:12 KWUN kernel: ? _raw_spin_lock+0x13/0x1c Jul 24 19:08:12 KWUN kernel: ? _raw_spin_unlock+0x14/0x29 Jul 24 19:08:12 KWUN kernel: ? tsd_hash_search+0x70/0x7d [spl] Jul 24 19:08:12 KWUN kernel: zio_execute+0xb4/0xdf [zfs] Jul 24 19:08:12 KWUN kernel: taskq_thread+0x269/0x38a [spl] Jul 24 19:08:12 KWUN kernel: ? wake_up_q+0x44/0x44 Jul 24 19:08:12 KWUN kernel: ? zio_subblock+0x22/0x22 [zfs] Jul 24 19:08:12 KWUN kernel: ? taskq_dispatch_delay+0x106/0x106 [spl] Jul 24 19:08:12 KWUN kernel: kthread+0xe7/0xef Jul 24 19:08:12 KWUN kernel: ? kthread_complete_and_exit+0x1b/0x1b Jul 24 19:08:12 KWUN kernel: ret_from_fork+0x22/0x30 Jul 24 19:08:12 KWUN kernel: </TASK> Jul 24 19:11:05 KWUN nginx: 2023/07/24 19:11:05 [error] 2375#2375: *1071 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.2.2, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "kwun.local", referrer: "http://kwun.local/Main" root@KWUN:~# zpool import pool: mainzfs id: 17865510998351045675 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: mainzfs ONLINE raidz1-0 ONLINE sdj1 ONLINE sdf1 ONLINE sdg1 ONLINE sdh1 ONLINE raidz1-1 ONLINE sdm1 ONLINE sdk1 ONLINE sdi1 ONLINE sdl1 ONLINE I could not get the diagnostics while it was stuck in mounting but here is the diagnostics with the unassigned disks I have snapshots, are they usable? Can anyone help me fix this problem please? kwun-diagnostics-20230724-2027.zip
July 31, 20232 yr Community Expert If you still need help see if the pool can be imported read only: zpool import -o readonly=on mainzfs
August 2, 20232 yr Author Yep that worked on read only, going to back up all the data then try the magical -F -X -a -d to see if that fixes things.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.