February 27, 20242 yr I am stuck between a rock and a hard place with my Unraid server and wondering if someone can help me and point me in the right direction. I have the following system. Intel® Xeon® Gold 6126 CPU @ 2.60GHz 192 GiB DDR4 Single-bit ECC Parity Disk 1 - Seagate 12TB NAS Parity Disk 2 - Seagate 10TB NAS Array Disk 1 to 5 - Seagate 10TB NAS Array Disk 6 - Seagate 6TB NAS 4 Cache array - 3 x 2 x Samsung 1TB SSD - 1 x Samsung 500GB SSD The history - I recently converted disk 1 and all my cache disks to ZFS file system, without any problems following Spaceinvader tutorials. Since then disk 5 had failed due to an old disk and a second time due to a fault cable (Which I have replaced now). I have not run a new parity check since then and it is still valid. The problem - I am trying to rebuild disk 5 from the valid parity, but the system is hanging for days without any read/write, but one cpu core is pegged 100%. At one point I got some wrong information and I have wiped all my cache disks, this will not a problem if I can save disk 1 as it has the ZFS snapshots on it. What Happening - Start in maintenance mode, I am able to download the diagnostics files without a problem (See attached). unassigned devices is working normally. I am able to stop the array normally. But when I start the array normally, it gets stuck "Array Starting - Mounting disks" and I can not do anything with array. Once again unassigned devices is working normally. Starting array normally - Feb 27 19:15:35 unraidbox avahi-daemon[17112]: Server startup complete. Host name is unraidbox.local. Local service cookie is 3262916861. Feb 27 19:15:36 unraidbox avahi-daemon[17112]: Service "unraidbox" (/services/ssh.service) successfully established. Feb 27 19:15:36 unraidbox avahi-daemon[17112]: Service "unraidbox" (/services/smb.service) successfully established. Feb 27 19:15:36 unraidbox avahi-daemon[17112]: Service "unraidbox" (/services/sftp-ssh.service) successfully established. Feb 27 19:15:44 unraidbox SysDrivers: SysDrivers Build Complete Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538513): /usr/local/sbin/set_ncq sdi 1 Feb 27 19:15:47 unraidbox kernel: mdcmd (31): set md_num_stripes 1280 Feb 27 19:15:47 unraidbox kernel: mdcmd (32): set md_queue_limit 80 Feb 27 19:15:47 unraidbox kernel: mdcmd (33): set md_sync_limit 5 Feb 27 19:15:47 unraidbox kernel: mdcmd (34): set md_write_method Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538514): echo 128 > /sys/block/sdi/queue/nr_requests Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538515): /usr/local/sbin/set_ncq sdk 1 Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538516): echo 128 > /sys/block/sdk/queue/nr_requests Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538517): /usr/local/sbin/set_ncq sdj 1 Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538518): echo 128 > /sys/block/sdj/queue/nr_requests Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538519): /usr/local/sbin/set_ncq sdq 1 Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538520): echo 128 > /sys/block/sdq/queue/nr_requests Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538521): /usr/local/sbin/set_ncq sdr 1 Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538522): echo 128 > /sys/block/sdr/queue/nr_requests Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538523): /usr/local/sbin/set_ncq sdo 1 Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538524): echo 128 > /sys/block/sdo/queue/nr_requests Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538525): /usr/local/sbin/set_ncq sdn 1 Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538526): echo 128 > /sys/block/sdn/queue/nr_requests Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538527): /usr/local/sbin/set_ncq sdl 1 Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538528): echo 128 > /sys/block/sdl/queue/nr_requests Feb 27 19:15:47 unraidbox kernel: mdcmd (35): start STOPPED Feb 27 19:15:47 unraidbox kernel: unraid: allocating 41350K for 1280 stripes (8 disks) Feb 27 19:15:47 unraidbox kernel: md1p1: running, size: 9766436812 blocks Feb 27 19:15:47 unraidbox kernel: md2p1: running, size: 9766436812 blocks Feb 27 19:15:47 unraidbox kernel: md3p1: running, size: 9766436812 blocks Feb 27 19:15:47 unraidbox kernel: md4p1: running, size: 9766436812 blocks Feb 27 19:15:47 unraidbox kernel: md5p1: running, size: 9766436812 blocks Feb 27 19:15:47 unraidbox kernel: md6p1: running, size: 5860522532 blocks Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538529): udevadm settle Feb 27 19:15:47 unraidbox emhttpd: Opening encrypted volumes... Feb 27 19:15:47 unraidbox emhttpd: shcmd (1538530): touch /boot/config/forcesync Feb 27 19:15:48 unraidbox emhttpd: Mounting disks... Feb 27 19:15:48 unraidbox emhttpd: mounting /mnt/disk1 Feb 27 19:15:48 unraidbox emhttpd: shcmd (1538531): mkdir -p /mnt/disk1 Feb 27 19:15:48 unraidbox emhttpd: /usr/sbin/zpool import -f -d /dev/md1p1 2>&1 Feb 27 19:15:51 unraidbox emhttpd: pool: disk1 Feb 27 19:15:51 unraidbox emhttpd: id: ????????????????? Feb 27 19:15:51 unraidbox emhttpd: shcmd (1538532): /usr/sbin/zpool import -f -N -o autoexpand=on -d /dev/md1p1 ????????????????? disk1 Feb 27 19:15:56 unraidbox kernel: VERIFY3(0 == dmu_buf_hold_array(os, object, offset, size, FALSE, FTAG, &numbufs, &dbp)) failed (0 == 5) Feb 27 19:15:56 unraidbox kernel: PANIC at dmu.c:1123:dmu_write() Feb 27 19:15:56 unraidbox kernel: Showing stack for process 22191 Feb 27 19:15:56 unraidbox kernel: CPU: 2 PID: 22191 Comm: txg_sync Tainted: P O 6.1.74-Unraid #1 Feb 27 19:15:56 unraidbox kernel: Hardware name: ASUSTeK COMPUTER INC. Z11PA-U12 Series/Z11PA-U12 Series, BIOS 6802 09/19/2022 Feb 27 19:15:56 unraidbox kernel: Call Trace: Feb 27 19:15:56 unraidbox kernel: <TASK> Feb 27 19:15:56 unraidbox kernel: dump_stack_lvl+0x44/0x5c Feb 27 19:15:56 unraidbox kernel: spl_panic+0xd0/0xe8 [spl] Feb 27 19:15:56 unraidbox kernel: ? dmu_buf_hold_array_by_dnode+0x279/0x41c [zfs] Feb 27 19:15:56 unraidbox kernel: ? preempt_latency_start+0x2b/0x46 Feb 27 19:15:56 unraidbox kernel: ? _raw_spin_lock+0x13/0x1c Feb 27 19:15:56 unraidbox kernel: ? _raw_spin_unlock+0x14/0x29 Feb 27 19:15:56 unraidbox kernel: ? dnode_rele_and_unlock+0x67/0xb6 [zfs] Feb 27 19:15:56 unraidbox kernel: ? dmu_buf_hold_array.constprop.0+0x7b/0xa2 [zfs] Feb 27 19:15:56 unraidbox kernel: ? dmu_buf_hold_array_by_dnode+0x24d/0x41c [zfs] Feb 27 19:15:56 unraidbox kernel: dmu_write+0x77/0xf0 [zfs] Feb 27 19:15:56 unraidbox kernel: ? _raw_spin_lock+0x13/0x1c Feb 27 19:15:56 unraidbox kernel: space_map_write+0x19c/0x605 [zfs] Feb 27 19:15:56 unraidbox kernel: ? zap_lookup_norm+0x9b/0xcb [zfs] Feb 27 19:15:56 unraidbox kernel: metaslab_flush+0x199/0x326 [zfs] Feb 27 19:15:56 unraidbox kernel: spa_flush_metaslabs+0x2bf/0x334 [zfs] Feb 27 19:15:56 unraidbox kernel: spa_sync+0x7c9/0xd12 [zfs] Feb 27 19:15:56 unraidbox kernel: txg_sync_thread+0x1f3/0x375 [zfs] Feb 27 19:15:56 unraidbox kernel: ? __kmem_cache_free+0xb0/0xf5 Feb 27 19:15:56 unraidbox kernel: ? txg_fini+0x1c2/0x1c2 [zfs] Feb 27 19:15:56 unraidbox kernel: ? __thread_exit+0x13/0x13 [spl] Feb 27 19:15:56 unraidbox kernel: thread_generic_wrapper+0x57/0x65 [spl] Feb 27 19:15:56 unraidbox kernel: kthread+0xe4/0xef Feb 27 19:15:56 unraidbox kernel: ? kthread_complete_and_exit+0x1b/0x1b Feb 27 19:15:56 unraidbox kernel: ret_from_fork+0x1f/0x30 Feb 27 19:15:56 unraidbox kernel: </TASK> Running diagnostics download during normal start - it hangs in the same place each time - But completes during maintenance mode. Finally, I not know if this is the cause or problem, but the "zfs-info.txt" file has no information in it unraidbox-diagnostics-20240227-0755.zip unraidbox-syslog-20240226-2050.zip Edited February 27, 20242 yr by Thunder Add Log details
February 27, 20242 yr Community Expert Disk1 filesystem appears to be corrupt, see if you can mount it read only, start the array in maintenance mode and type: zpool import -o readonly=on disk1
February 28, 20242 yr Author I type this command in maintenance mode and here is the result and I didn't see any change in the syslog. unraidbox-syslog-20240228-0726.zip unraidbox-diagnostics-20240228-1827.zip Edited February 28, 20242 yr by Thunder Add diagnostics & syslog
February 28, 20242 yr Community Expert Now you should be able to backup any data there to somewhere else and reformat the disk, the data will be under /mnt/disk1, you can copy for example to a UD device or SMB share.
February 28, 20242 yr Author Sorry, does this mean that I drop this disk 1 out of array and use "unassigned disk" to mount the disk out of maintenance mode? Or do I start to the array as normal and access this disk
February 28, 20242 yr Community Expert Whatever is easier for you, if you prefer to copy the data to the array, export the pool first zpool export disk1 stop the array re-import the pool read-only zpool import -o readonly=on disk1 now start the array in normal mode, disk1 will still show unmountable but the data will be under /mnt/disk1, you can then copy to another array disk.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.