Disks wont Mount

[resident-rummy3849]

Hi,

 

I had a brief power outage yesterday that lasted about 15-20 minutes. Server lost power but didnt get turned on for several hours later. Nothing was being written or read from the server at the time of the outage.

 

Now my server starts but I wont mount any of the disks in the array and will just hang there as "mounting".

 

I can start the server in maintenance mode without issue.

 

Doing a google search I read something that said to erase the cache drive and that would fix the issue. That did not work.

 

I then tried to remove the cache drive from the array. That also did not work.

 

Everything is formatted as ZFS.

 

The only thing that I have not tried is to remove the disks from the array and then re-add them all.

 

Any input would be greatly appreciated.

 

-B

[JorgeB]

Get the syslog once it's stuck to se which filesystem is the problem:

 

cp /var/log/syslog /boot/syslog.txt

 

[resident-rummy3849]

Apr 26 11:01:46 streamengine kernel: mdcmd (30): import 29
Apr 26 11:01:46 streamengine kernel: md: import_slot: 29 empty
Apr 26 11:01:46 streamengine emhttpd: import 30 cache device: (sdd) Samsung_SSD_870_EVO_2TB_S6PNNS0T603387M
Apr 26 11:01:46 streamengine emhttpd: import 31 cache device: no device
Apr 26 11:01:46 streamengine emhttpd: import flash device: sda
Apr 26 11:01:46 streamengine root: Submitting SysDrivers Build
Apr 26 11:01:46 streamengine SysDrivers: SysDrivers Build Starting
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdj
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdk
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdh
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdg
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdd
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sde
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdb
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdf
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdc
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdl
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sdi
Apr 26 11:01:46 streamengine emhttpd: read SMART /dev/sda
Apr 26 11:01:56 streamengine kernel: mdcmd (31): set md_num_stripes 1280
Apr 26 11:01:56 streamengine kernel: mdcmd (32): set md_queue_limit 80
Apr 26 11:01:56 streamengine kernel: mdcmd (33): set md_sync_limit 5
Apr 26 11:01:56 streamengine kernel: mdcmd (34): set md_write_method
Apr 26 11:02:13 streamengine SysDrivers: SysDrivers Build Complete
Apr 26 14:54:13 streamengine kernel: mdcmd (35): set md_num_stripes 1280
Apr 26 14:54:13 streamengine kernel: mdcmd (36): set md_queue_limit 80
Apr 26 14:54:13 streamengine kernel: mdcmd (37): set md_sync_limit 5
Apr 26 14:54:13 streamengine kernel: mdcmd (38): set md_write_method
Apr 26 14:54:13 streamengine kernel: mdcmd (39): start STOPPED
Apr 26 14:54:13 streamengine kernel: unraid: allocating 51590K for 1280 stripes (10 disks)
Apr 26 14:54:13 streamengine kernel: md1p1: running, size: 19531825100 blocks
Apr 26 14:54:13 streamengine kernel: md2p1: running, size: 7814026532 blocks
Apr 26 14:54:13 streamengine kernel: md3p1: running, size: 7814026532 blocks
Apr 26 14:54:13 streamengine kernel: md4p1: running, size: 7814026532 blocks
Apr 26 14:54:13 streamengine kernel: md5p1: running, size: 7814026532 blocks
Apr 26 14:54:13 streamengine kernel: md6p1: running, size: 11718885324 blocks
Apr 26 14:54:13 streamengine kernel: md7p1: running, size: 11718885324 blocks
Apr 26 14:54:14 streamengine kernel: md8p1: running, size: 11718885324 blocks
Apr 26 14:54:14 streamengine emhttpd: shcmd (25494): udevadm settle
Apr 26 14:54:14 streamengine emhttpd: Opening encrypted volumes...
Apr 26 14:54:14 streamengine emhttpd: shcmd (25495): touch /boot/config/forcesync
Apr 26 14:54:14 streamengine emhttpd: Mounting disks...
Apr 26 14:54:14 streamengine emhttpd: mounting /mnt/disk1
Apr 26 14:54:14 streamengine emhttpd: shcmd (25496): mkdir -p /mnt/disk1
Apr 26 14:54:14 streamengine emhttpd: /usr/sbin/zpool import -f -d /dev/md1p1 2>&1
Apr 26 14:54:17 streamengine emhttpd:    pool: disk1
Apr 26 14:54:17 streamengine emhttpd:      id: 1522930789915103990
Apr 26 14:54:17 streamengine emhttpd: shcmd (25497): /usr/sbin/zpool import -f -N -o autoexpand=on  -d /dev/md1p1 1522930789915103990 disk1
Apr 26 14:54:22 streamengine kernel: VERIFY3(range_tree_space(smla->smla_rt) + sme->sme_run <= smla->smla_sm->sm_size) failed (281460079722496 <= 17179869184)
Apr 26 14:54:22 streamengine kernel: PANIC at space_map.c:405:space_map_load_callback()
Apr 26 14:54:22 streamengine kernel: Showing stack for process 25399
Apr 26 14:54:22 streamengine kernel: CPU: 0 PID: 25399 Comm: z_wr_iss Tainted: P           O       6.1.79-Unraid #1
Apr 26 14:54:22 streamengine kernel: Hardware name: System manufacturer System Product Name/ROG STRIX B450-F GAMING, BIOS 4901 07/25/2022
Apr 26 14:54:22 streamengine kernel: Call Trace:
Apr 26 14:54:22 streamengine kernel: <TASK>
Apr 26 14:54:22 streamengine kernel: dump_stack_lvl+0x44/0x5c
Apr 26 14:54:22 streamengine kernel: spl_panic+0xd0/0xe8 [spl]
Apr 26 14:54:22 streamengine kernel: ? rs_get_start+0xc/0x1d [zfs]
Apr 26 14:54:22 streamengine kernel: ? range_tree_stat_incr+0x28/0x43 [zfs]
Apr 26 14:54:22 streamengine kernel: ? range_tree_remove_impl+0x3b7/0x406 [zfs]
Apr 26 14:54:22 streamengine kernel: ? zio_wait+0x1ee/0x1fd [zfs]
Apr 26 14:54:22 streamengine kernel: space_map_load_callback+0x50/0x79 [zfs]
Apr 26 14:54:22 streamengine kernel: space_map_iterate+0x2d6/0x324 [zfs]
Apr 26 14:54:22 streamengine kernel: ? spa_stats_destroy+0x16c/0x16c [zfs]
Apr 26 14:54:22 streamengine kernel: space_map_load_length+0x93/0xcb [zfs]
Apr 26 14:54:22 streamengine kernel: metaslab_load+0x33b/0x6e3 [zfs]
Apr 26 14:54:22 streamengine kernel: ? slab_post_alloc_hook+0x4d/0x15e
Apr 26 14:54:22 streamengine kernel: ? spl_kmem_alloc_impl+0xc1/0xf2 [spl]
Apr 26 14:54:22 streamengine kernel: ? __kmem_cache_alloc_node+0x118/0x147
Apr 26 14:54:22 streamengine kernel: metaslab_activate+0x36/0x1f1 [zfs]
Apr 26 14:54:22 streamengine kernel: metaslab_alloc_dva+0x8bc/0xfce [zfs]
Apr 26 14:54:22 streamengine kernel: ? preempt_latency_start+0x2b/0x46
Apr 26 14:54:22 streamengine kernel: metaslab_alloc+0x107/0x1fd [zfs]
Apr 26 14:54:22 streamengine kernel: zio_dva_allocate+0xee/0x73f [zfs]
Apr 26 14:54:22 streamengine kernel: ? kmem_cache_free+0xc9/0x154
Apr 26 14:54:22 streamengine kernel: ? spl_kmem_cache_free+0x3a/0x1a5 [spl]
Apr 26 14:54:22 streamengine kernel: ? preempt_latency_start+0x2b/0x46
Apr 26 14:54:22 streamengine kernel: ? _raw_spin_lock+0x13/0x1c
Apr 26 14:54:22 streamengine kernel: ? _raw_spin_unlock+0x14/0x29
Apr 26 14:54:22 streamengine kernel: ? tsd_hash_search+0x70/0x7d [spl]
Apr 26 14:54:22 streamengine kernel: zio_execute+0xb4/0xdf [zfs]
Apr 26 14:54:22 streamengine kernel: taskq_thread+0x269/0x38a [spl]
Apr 26 14:54:22 streamengine kernel: ? wake_up_q+0x44/0x44
Apr 26 14:54:22 streamengine kernel: ? zio_subblock+0x22/0x22 [zfs]
Apr 26 14:54:22 streamengine kernel: ? taskq_dispatch_delay+0x106/0x106 [spl]
Apr 26 14:54:22 streamengine kernel: kthread+0xe7/0xef
Apr 26 14:54:22 streamengine kernel: ? kthread_complete_and_exit+0x1b/0x1b
Apr 26 14:54:22 streamengine kernel: ret_from_fork+0x22/0x30
Apr 26 14:54:22 streamengine kernel: </TASK>
Apr 26 14:54:43 streamengine network: reload service: nginx

[resident-rummy3849]

I noticed this:

 

Apr 26 14:54:22 streamengine kernel: PANIC at space_map.c:405:space_map_load_callback()

 

[JorgeB]

Reboot and don't start the array, then try to import disk1 read-only, since that won't read space maps, among other things:

 

zpool import -o readonly=on disk1

 

If it works, start the array, disk1 will show unmountable in the GUI, but the data should be under /mnt/disk1, if yes backup that disk and re-format

[resident-rummy3849]

Okay,

 

Array started and disk 1 shows unmountable as you said in the GUI. Going to the field explorer I can see evertything that is on Disk 1. I will work on moving those to other disks and will reformat.

 

Also, Docker is not saying Failed to start?

[JorgeB]

17 hours ago, resident-rummy3849 said:

Also, Docker is not saying Failed to start?

Post new diags.

[resident-rummy3849]

I am pretty sure that I figured it out for the Docker. My chace drive needs to be formatted as well. 

 

I am still waiting on everything to finish moving from Disk 1 that crashed to Disk 6. Going to take a while.