Nokedr

Yep that worked on read only, going to back up all the data then try the magical -F -X -a -d to see if that fixes things.

After a power loss, I have not been able to start the array, i have isolated the problem to when the system is trying to mount the "mainzfs" pool and have successfullyu started the array with all drives in "mainzfs" unassigned Jul 24 19:08:07 KWUN emhttpd: shcmd (249): touch /boot/config/forcesync Jul 24 19:08:07 KWUN emhttpd: Mounting disks... Jul 24 19:08:07 KWUN emhttpd: mounting /mnt/disk1 Jul 24 19:08:07 KWUN emhttpd: shcmd (250): mkdir -p /mnt/disk1 Jul 24 19:08:07 KWUN emhttpd: shcmd (251): mount -t xfs -o noatime,nouuid /dev/md1p1 /mnt/disk1 Jul 24 19:08:08 KWUN kernel: SGI XFS with ACLs, security attributes, no debug enabled Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Mounting V5 Filesystem Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Starting recovery (logdev: internal) Jul 24 19:08:08 KWUN kernel: XFS (md1p1): Ending recovery (logdev: internal) Jul 24 19:08:08 KWUN emhttpd: shcmd (252): xfs_growfs /mnt/disk1 Jul 24 19:08:08 KWUN root: meta-data=/dev/md1p1 isize=512 agcount=4, agsize=7551902 blks Jul 24 19:08:08 KWUN root: = sectsz=512 attr=2, projid32bit=1 Jul 24 19:08:08 KWUN root: = crc=1 finobt=1, sparse=1, rmapbt=0 Jul 24 19:08:08 KWUN root: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 Jul 24 19:08:08 KWUN root: data = bsize=4096 blocks=30207606, imaxpct=25 Jul 24 19:08:08 KWUN root: = sunit=0 swidth=0 blks Jul 24 19:08:08 KWUN root: naming =version 2 bsize=4096 ascii-ci=0, ftype=1 Jul 24 19:08:08 KWUN root: log =internal log bsize=4096 blocks=16384, version=2 Jul 24 19:08:08 KWUN root: = sectsz=512 sunit=0 blks, lazy-count=1 Jul 24 19:08:08 KWUN root: realtime =none extsz=4096 blocks=0, rtextents=0 Jul 24 19:08:08 KWUN emhttpd: mounting /mnt/big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (253): mkdir -p /mnt/big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (254): /usr/sbin/zpool import -N -o autoexpand=on -d /dev/sdc1 -d /dev/sdd1 8947813539389484345 big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (255): /usr/sbin/zfs set mountpoint=/mnt/big_cache big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (256): /usr/sbin/zfs set atime=off big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (257): /usr/sbin/zfs mount big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (258): /usr/sbin/zpool set autotrim=on big_cache Jul 24 19:08:08 KWUN emhttpd: shcmd (259): /usr/sbin/zfs set compression=on big_cache Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root profile: Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root groups: 2 Jul 24 19:08:08 KWUN emhttpd: /mnt/big_cache root width: 1 Jul 24 19:08:08 KWUN emhttpd: /usr/sbin/zpool status -PL big_cache 2>&1 Jul 24 19:08:08 KWUN emhttpd: pool: big_cache Jul 24 19:08:08 KWUN emhttpd: state: ONLINE Jul 24 19:08:08 KWUN emhttpd: config: Jul 24 19:08:08 KWUN emhttpd: NAME STATE READ WRITE CKSUM Jul 24 19:08:08 KWUN emhttpd: big_cache ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: /dev/sdc1 ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: /dev/sdd1 ONLINE 0 0 0 Jul 24 19:08:08 KWUN emhttpd: errors: No known data errors Jul 24 19:08:08 KWUN emhttpd: mounting /mnt/mainzfs Jul 24 19:08:08 KWUN emhttpd: shcmd (260): mkdir -p /mnt/mainzfs Jul 24 19:08:08 KWUN emhttpd: /sbin/btrfs filesystem show /dev/sdj1 2>&1 Jul 24 19:08:08 KWUN emhttpd: ERROR: no btrfs on /dev/sdj1 Jul 24 19:08:08 KWUN emhttpd: /usr/sbin/zpool import -d /dev/sdj1 2>&1 Jul 24 19:08:09 KWUN emhttpd: pool: mainzfs Jul 24 19:08:09 KWUN emhttpd: id: 17865510998351045675 Jul 24 19:08:09 KWUN emhttpd: shcmd (261): /usr/sbin/zpool import -N -o autoexpand=on -d /dev/sdj1 -d /dev/sdf1 -d /dev/sdg1 -d /dev/sdh1 -d /dev/sdm1 -d /dev/sdk1 -d /dev/sdi1 -d /dev/sdl1 17865510998351045675 mainzfs Jul 24 19:08:12 KWUN kernel: VERIFY3(range_tree_space(smla->smla_rt) + sme->sme_run <= smla->smla_sm->sm_size) failed (17237200896 <= 17179869184) Jul 24 19:08:12 KWUN kernel: PANIC at space_map.c:405:space_map_load_callback() Jul 24 19:08:12 KWUN kernel: Showing stack for process 6653 Jul 24 19:08:12 KWUN kernel: CPU: 1 PID: 6653 Comm: z_wr_iss Tainted: P O 6.1.38-Unraid #2 Jul 24 19:08:12 KWUN kernel: Hardware name: Micro-Star International Co., Ltd MS-7C02/B450 TOMAHAWK MAX II (MS-7C02), BIOS H.90 07/25/2022 Jul 24 19:08:12 KWUN kernel: Call Trace: Jul 24 19:08:12 KWUN kernel: <TASK> Jul 24 19:08:12 KWUN kernel: dump_stack_lvl+0x44/0x5c Jul 24 19:08:12 KWUN kernel: spl_panic+0xd0/0xe8 [spl] Jul 24 19:08:12 KWUN kernel: ? rs_get_start+0xc/0x1d [zfs] Jul 24 19:08:12 KWUN kernel: ? range_tree_stat_incr+0x28/0x43 [zfs] Jul 24 19:08:12 KWUN kernel: ? range_tree_remove_impl+0x3b7/0x406 [zfs] Jul 24 19:08:12 KWUN kernel: ? zio_wait+0x1ee/0x1fd [zfs] Jul 24 19:08:12 KWUN kernel: space_map_load_callback+0x50/0x79 [zfs] Jul 24 19:08:12 KWUN kernel: space_map_iterate+0x2d6/0x324 [zfs] Jul 24 19:08:12 KWUN kernel: ? spa_stats_destroy+0x16c/0x16c [zfs] Jul 24 19:08:12 KWUN kernel: space_map_load_length+0x93/0xcb [zfs] Jul 24 19:08:12 KWUN kernel: metaslab_load+0x33b/0x6e3 [zfs] Jul 24 19:08:12 KWUN kernel: ? slab_post_alloc_hook+0x4d/0x15e Jul 24 19:08:12 KWUN kernel: ? rs_get_start+0xc/0x1d [zfs] Jul 24 19:08:12 KWUN kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl] Jul 24 19:08:12 KWUN kernel: ? __kmem_cache_alloc_node+0x118/0x147 Jul 24 19:08:12 KWUN kernel: metaslab_activate+0x36/0x1f1 [zfs] Jul 24 19:08:12 KWUN kernel: metaslab_alloc_dva+0x8bc/0xfce [zfs] Jul 24 19:08:12 KWUN kernel: ? preempt_latency_start+0x2b/0x46 Jul 24 19:08:12 KWUN kernel: metaslab_alloc+0x107/0x1fd [zfs] Jul 24 19:08:12 KWUN kernel: zio_dva_allocate+0xee/0x73f [zfs] Jul 24 19:08:12 KWUN kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl] Jul 24 19:08:12 KWUN kernel: ? preempt_latency_start+0x2b/0x46 Jul 24 19:08:12 KWUN kernel: ? _raw_spin_lock+0x13/0x1c Jul 24 19:08:12 KWUN kernel: ? _raw_spin_unlock+0x14/0x29 Jul 24 19:08:12 KWUN kernel: ? tsd_hash_search+0x70/0x7d [spl] Jul 24 19:08:12 KWUN kernel: zio_execute+0xb4/0xdf [zfs] Jul 24 19:08:12 KWUN kernel: taskq_thread+0x269/0x38a [spl] Jul 24 19:08:12 KWUN kernel: ? wake_up_q+0x44/0x44 Jul 24 19:08:12 KWUN kernel: ? zio_subblock+0x22/0x22 [zfs] Jul 24 19:08:12 KWUN kernel: ? taskq_dispatch_delay+0x106/0x106 [spl] Jul 24 19:08:12 KWUN kernel: kthread+0xe7/0xef Jul 24 19:08:12 KWUN kernel: ? kthread_complete_and_exit+0x1b/0x1b Jul 24 19:08:12 KWUN kernel: ret_from_fork+0x22/0x30 Jul 24 19:08:12 KWUN kernel: </TASK> Jul 24 19:11:05 KWUN nginx: 2023/07/24 19:11:05 [error] 2375#2375: *1071 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.2.2, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "kwun.local", referrer: "http://kwun.local/Main" root@KWUN:~# zpool import pool: mainzfs id: 17865510998351045675 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: mainzfs ONLINE raidz1-0 ONLINE sdj1 ONLINE sdf1 ONLINE sdg1 ONLINE sdh1 ONLINE raidz1-1 ONLINE sdm1 ONLINE sdk1 ONLINE sdi1 ONLINE sdl1 ONLINE I could not get the diagnostics while it was stuck in mounting but here is the diagnostics with the unassigned disks I have snapshots, are they usable? Can anyone help me fix this problem please? kwun-diagnostics-20230724-2027.zip

Data rebuild successfully on the same disk (I had a backup of all the data), it was just an unproved shutdown nothing wrong with the disk, thanks a lot for the help

Thanks for the link, all the old links on the older thread seem to just redirect back to the main page. The disk is being emulated correctly. kwun-diagnostics-20230616-1035.zip

After messing up an assigning my only gpu to a vm i had to hard power off. Upon rebooting one of my disk was disabled, I followed the storage management on the manual to check / repair a disabled disk Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 1 - agno = 3 - agno = 2 - agno = 4 - agno = 0 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done Restarting the array normally still showed the disk as disabled, the .trash file on the disk was also empty

+1

Hey @GonjaT, I've been running your RagnamodVI, and it's been great, I just had one question: Is there a way to issue server commands? As in "list" or "op" or "say" through the server terminal. The log doesn't seem to respond to anything I type, and the console is a system console and not the one of the server, so I can find the server process, but I don't know how to access it. Thanks in advance for any help you can give. Nokedr