Nokedr Posted July 24, 2023 Share Posted July 24, 2023 After a power loss, I have not been able to start the array, i have isolated the problem to when the system is trying to mount the "mainzfs" pool and have successfullyu started the array with all drives in "mainzfs" unassigned Jul 24 19:08:07 KWUN emhttpd: shcmd (249): touch /boot/config/forcesync Jul 24 19:08:07 KWUN emhttpd: Mounting disks... Jul 24 19:08:07 KWUN emhttpd: mounting /mnt/disk1 Jul 24 19:08:07 KWUN emhttpd: shcmd (250): mkdir -p /mnt/disk1 Jul 24 19:08:07 KWUN emhttpd: shcmd (251): mount -t xfs -o noatime,nouuid /dev/md1p1 /mnt/disk1 Jul 24 19:08:08 KWUN kernel: SGI XFS with ACLs, security attributes, no debug enabled Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Mounting V5 Filesystem Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Starting recovery (logdev: internal) Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Ending recovery (logdev: internal) Jul 24 19:08:08 KWUN emhttpd: shcmd (252): xfs_growfs /mnt/disk1 Jul 24 19:08:08 KWUN root: meta-data=/dev/md1p1 isize=512 agcount=4, agsize=7551902 blks Jul 24 19:08:08 KWUN root: = sectsz=512 attr=2, projid32bit=1 Jul 24 19:08:08 KWUN root: = crc=1 finobt=1, sparse=1, rmapbt=0 Jul 24 19:08:08 KWUN root: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 Jul 24 19:08:08 KWUN root: data = bsize=4096 blocks=30207606, imaxpct=25 Jul 24 19:08:08 KWUN root: = sunit=0 swidth=0 blks Jul 24 19:08:08 KWUN root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jul 24 19:08:08 KWUN root: log =internal log bsize=4096 blocks=16384, version=2 Jul 24 19:08:08 KWUN root: = sectsz=512 sunit=0 blks, lazy-count=1 Jul 24 19:08:08 KWUN root: realtime =none extsz=4096 blocks=0, rtextents=0 Jul 24 19:08:08 KWUN emhttpd: mounting /mnt/big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (253): mkdir -p /mnt/big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (254): /usr/sbin/zpool import -N -o autoexpand=on -d /dev/sdc1 -d /dev/sdd1 8947813539389484345 big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (255): /usr/sbin/zfs set mountpoint=/mnt/big_cache big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (256): /usr/sbin/zfs set atime=off big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (257): /usr/sbin/zfs mount big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (258): /usr/sbin/zpool set autotrim=on big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (259): /usr/sbin/zfs set compression=on big_cache Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root profile: Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root groups: 2 Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root width: 1 Jul 24 19:08:08 KWUN emhttpd: /usr/sbin/zpool status -PL big_cache 2>&1 Jul 24 19:08:08 KWUN emhttpd: pool: big_cache Jul 24 19:08:08 KWUN emhttpd: state: ONLINE Jul 24 19:08:08 KWUN emhttpd: config: Jul 24 19:08:08 KWUN emhttpd: NAME STATE READ WRITE CKSUM Jul 24 19:08:08 KWUN emhttpd: big_cache ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: /dev/sdc1 ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: /dev/sdd1 ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: errors: No known data errors Jul 24 19:08:08 KWUN emhttpd: mounting /mnt/mainzfs Jul 24 19:08:08 KWUN emhttpd: shcmd (260): mkdir -p /mnt/mainzfs Jul 24 19:08:08 KWUN emhttpd: /sbin/btrfs filesystem show /dev/sdj1 2>&1 Jul 24 19:08:08 KWUN emhttpd: ERROR: no btrfs on /dev/sdj1 Jul 24 19:08:08 KWUN emhttpd: /usr/sbin/zpool import -d /dev/sdj1 2>&1 Jul 24 19:08:09 KWUN emhttpd: pool: mainzfs Jul 24 19:08:09 KWUN emhttpd: id: 17865510998351045675 Jul 24 19:08:09 KWUN emhttpd: shcmd (261): /usr/sbin/zpool import -N -o autoexpand=on -d /dev/sdj1 -d /dev/sdf1 -d /dev/sdg1 -d /dev/sdh1 -d /dev/sdm1 -d /dev/sdk1 -d /dev/sdi1 -d /dev/sdl1 17865510998351045675 mainzfs Jul 24 19:08:12 KWUN kernel: VERIFY3(range_tree_space(smla->smla_rt) + sme->sme_run <= smla->smla_sm->sm_size) failed (17237200896 <= 17179869184) Jul 24 19:08:12 KWUN kernel: PANIC at space_map.c:405:space_map_load_callback() Jul 24 19:08:12 KWUN kernel: Showing stack for process 6653 Jul 24 19:08:12 KWUN kernel: CPU: 1 PID: 6653 Comm: z_wr_iss Tainted: P O 6.1.38-Unraid #2 Jul 24 19:08:12 KWUN kernel: Hardware name: Micro-Star International Co., Ltd MS-7C02/B450 TOMAHAWK MAX II (MS-7C02), BIOS H.90 07/25/2022 Jul 24 19:08:12 KWUN kernel: Call Trace: Jul 24 19:08:12 KWUN kernel: <TASK> Jul 24 19:08:12 KWUN kernel: dump_stack_lvl+0x44/0x5c Jul 24 19:08:12 KWUN kernel: spl_panic+0xd0/0xe8 [spl] Jul 24 19:08:12 KWUN kernel: ? rs_get_start+0xc/0x1d [zfs] Jul 24 19:08:12 KWUN kernel: ? range_tree_stat_incr+0x28/0x43 [zfs] Jul 24 19:08:12 KWUN kernel: ? range_tree_remove_impl+0x3b7/0x406 [zfs] Jul 24 19:08:12 KWUN kernel: ? zio_wait+0x1ee/0x1fd [zfs] Jul 24 19:08:12 KWUN kernel: space_map_load_callback+0x50/0x79 [zfs] Jul 24 19:08:12 KWUN kernel: space_map_iterate+0x2d6/0x324 [zfs] Jul 24 19:08:12 KWUN kernel: ? spa_stats_destroy+0x16c/0x16c [zfs] Jul 24 19:08:12 KWUN kernel: space_map_load_length+0x93/0xcb [zfs] Jul 24 19:08:12 KWUN kernel: metaslab_load+0x33b/0x6e3 [zfs] Jul 24 19:08:12 KWUN kernel: ? slab_post_alloc_hook+0x4d/0x15e Jul 24 19:08:12 KWUN kernel: ? rs_get_start+0xc/0x1d [zfs] Jul 24 19:08:12 KWUN kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl] Jul 24 19:08:12 KWUN kernel: ? __kmem_cache_alloc_node+0x118/0x147 Jul 24 19:08:12 KWUN kernel: metaslab_activate+0x36/0x1f1 [zfs] Jul 24 19:08:12 KWUN kernel: metaslab_alloc_dva+0x8bc/0xfce [zfs] Jul 24 19:08:12 KWUN kernel: ? preempt_latency_start+0x2b/0x46 Jul 24 19:08:12 KWUN kernel: metaslab_alloc+0x107/0x1fd [zfs] Jul 24 19:08:12 KWUN kernel: zio_dva_allocate+0xee/0x73f [zfs] Jul 24 19:08:12 KWUN kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl] Jul 24 19:08:12 KWUN kernel: ? preempt_latency_start+0x2b/0x46 Jul 24 19:08:12 KWUN kernel: ? _raw_spin_lock+0x13/0x1c Jul 24 19:08:12 KWUN kernel: ? _raw_spin_unlock+0x14/0x29 Jul 24 19:08:12 KWUN kernel: ? tsd_hash_search+0x70/0x7d [spl] Jul 24 19:08:12 KWUN kernel: zio_execute+0xb4/0xdf [zfs] Jul 24 19:08:12 KWUN kernel: taskq_thread+0x269/0x38a [spl] Jul 24 19:08:12 KWUN kernel: ? wake_up_q+0x44/0x44 Jul 24 19:08:12 KWUN kernel: ? zio_subblock+0x22/0x22 [zfs] Jul 24 19:08:12 KWUN kernel: ? taskq_dispatch_delay+0x106/0x106 [spl] Jul 24 19:08:12 KWUN kernel: kthread+0xe7/0xef Jul 24 19:08:12 KWUN kernel: ? kthread_complete_and_exit+0x1b/0x1b Jul 24 19:08:12 KWUN kernel: ret_from_fork+0x22/0x30 Jul 24 19:08:12 KWUN kernel: </TASK> Jul 24 19:11:05 KWUN nginx: 2023/07/24 19:11:05 [error] 2375#2375: *1071 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.2.2, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "kwun.local", referrer: "http://kwun.local/Main" root@KWUN:~# zpool import pool: mainzfs id: 17865510998351045675 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: mainzfs ONLINE raidz1-0 ONLINE sdj1 ONLINE sdf1 ONLINE sdg1 ONLINE sdh1 ONLINE raidz1-1 ONLINE sdm1 ONLINE sdk1 ONLINE sdi1 ONLINE sdl1 ONLINE I could not get the diagnostics while it was stuck in mounting but here is the diagnostics with the unassigned disks I have snapshots, are they usable? Can anyone help me fix this problem please? kwun-diagnostics-20230724-2027.zip Quote Link to comment
JorgeB Posted July 31, 2023 Share Posted July 31, 2023 If you still need help see if the pool can be imported read only: zpool import -o readonly=on mainzfs Quote Link to comment
Nokedr Posted August 2, 2023 Author Share Posted August 2, 2023 Yep that worked on read only, going to back up all the data then try the magical -F -X -a -d to see if that fixes things. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.