I have 8 drives in my array (2 parity + 6 data disks). They're connected to a 5 bay and 3 bay Kingwin drive enclosure, which in turn is connected to an HBA (LSI 9500-8I). The 5 bay is filled with 12TB drives where two of which are parity for the array, and the 3 bay is filled with 8 TB drives.

 

Tuesday night, Unraid started doing it's scheduled parity check. During this process, Parity2 and disk1 appear to have disconnected themselves from Unraid and remounted themselves under a different mountpoint. This caused a few error, Unraid disabled Parity2 and Disk1 and paused the parity check. Weird, no biggie, I'll work on rebuilding the array over the weekend (stupid me).

 

I log back in to Unraid yesterday and notice while the parity check was paused, Mover decided to run. Disk4 shows connected but spewing millions of errors. I panic, download diags (attached), attempt to reboot the server, but it doesn't reboot. So I force it (stupid me). Server boots back up. I start the array. Parity2 and Disk1 show disabled, and no errors on the other drives. I do a quick SMART test on Parity2, Disk1, and Disk4, and it comes back with no problems. Ok... let's rebuild. I stop the array, remove disk1, start the array, stop the array, add disk1 back, and the array starts rebuilding. A few minutes later Disk4 is spewing millions of errors again, and Parity1 now shows disabled. I shut down the server, reseat the cables, reseat the HDDs in the bays, boot the server back up, and kick off extended SMART tests.

 

The extended SMART tests just completed (attached) but they reported no errors. With parity1 and parity2 disabled, I'm going to assume a rebuild is not possible. I have no idea what to do next here. I'm hoping there's a way to come out of this with a few broken bones, but not completely dead. TIA

before-initial-reboot_unraidserver-diagnostics-20250109-1813.zip after-smart-tests_unraidserver-diagnostics-20250110-1518.zip

Multiple disks dropping offline at the same time, looks more like a power/connection issue, any splitter is use? It could also be the miniSAS cables or a weak/failing PSU

Parity1, Parity2 and Disk 1 are connected with the same 1 to 4 minisas cable, but disk4 is on the other 1 to 4 minisas cable.

 

The power is an interesting thought, thank you. Each drive enclosure acts as a power splitter. I can connect the enclosure that my 3 8tb drives are in to an external power source to shed usage from the PSU.

 

After doing this, would starting the array still actually try to rebuild or what would need to be done?

16 hours ago, gdesoto said:

After doing this, would starting the array still actually try to rebuild or what would need to be done?

Since there are 3 invalid disks, you will need to do a New Config to re-enable disk1 and re-sync parity.

So I went out and got a new power supply, and the disks are no longer disconnecting! Thanks for that!

 

Now I'm trying to restart the array. I have corruption on disk1 (zfs) and disk4 (xfs) of the array. When I went to start the array, it hung mounting disk1.

 

Jan 12 16:25:40 UnraidServer emhttpd: Mounting disks...
Jan 12 16:25:40 UnraidServer emhttpd: mounting /mnt/disk1
Jan 12 16:25:40 UnraidServer emhttpd: shcmd (152): mkdir -p /mnt/disk1
Jan 12 16:25:40 UnraidServer emhttpd: shcmd (153): mount -t xfs -o noatime,nouuid /dev/md1p1 /mnt/disk1
Jan 12 16:25:40 UnraidServer kernel: SGI XFS with ACLs, security attributes, no debug enabled
Jan 12 16:25:40 UnraidServer kernel: XFS (md1p1): Invalid superblock magic number
Jan 12 16:25:40 UnraidServer root: mount: /mnt/disk1: wrong fs type, bad option, bad superblock on /dev/md1p1, missing codepage or helper program, or other error.
Jan 12 16:25:40 UnraidServer root:        dmesg(1) may have more information after failed mount system call.
Jan 12 16:25:40 UnraidServer emhttpd: shcmd (153): exit status: 32
Jan 12 16:25:40 UnraidServer emhttpd: /mnt/disk1: no btrfs or device /dev/md1p1 is not single
Jan 12 16:25:40 UnraidServer emhttpd: /usr/sbin/zpool import -f -d /dev/md1p1 2>&1
Jan 12 16:25:44 UnraidServer emhttpd:    pool: disk1
Jan 12 16:25:44 UnraidServer emhttpd:      id: 9186586923919620123
Jan 12 16:25:44 UnraidServer emhttpd: shcmd (154): /usr/sbin/zpool import -f -N -o autoexpand=on  -d /dev/md1p1 9186586923919620123 disk1
Jan 12 16:25:50 UnraidServer emhttpd: shcmd (155): /usr/sbin/zpool online -e disk1 /dev/md1p1
Jan 12 16:25:50 UnraidServer kernel: WARNING: Pool 'disk1' has encountered an uncorrectable I/O failure and has been suspended.

 

I ran "zpool status" and got the following:

  pool: disk1
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.

 

I ran "zpool clear disk1" and it suspended again with the same "uncorrectable I/O" error.

 

I restarted the server and started the array in maintenance mode. I was able to kick off a parity sync, and all the drives were active and not disconnecting. I've read I may be able to use the "zpool scrub" command to find which files are corrupted, and delete those. When I tried, I got a message that the pool was suspended.

 

How can I get disk1 mounted to try the zpool scrub command, or should I be trying something else?

Post new diags please.

Here you go. The array is started, but still in maintenance mode. Parity sync is complete, but I haven't tried starting it while not in maintenance mode.

unraidserver-diagnostics-20250113-1328.zip

So I was able to figure out how to mount disk1 in readonly mode (zpool import -f -o readonly=on -F disk1). That allowed me to mount the drive and pull some data off of it.

 

When I ran zpool status disk1 I got the following:

  pool: disk1
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
config:

        NAME        STATE     READ WRITE CKSUM
        disk1       ONLINE       0     0     0
          md1p1     ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        disk1/Media:<0x0>

 

disk1/Media is one of the root folders that had a majority of the data. I was able to successfully get about 10% of the data on the disk. Luckily, the more important disk1/Backups directory appeared to be intact.

 

So now the question is, is there any path to repair from here? I'd like to be able to squeeze as much from what remains as possible. If not, then it is what it is. How would I clear the error to be able to use the disk again while not in readonly mode? Updated diags attached. Thanks!

unraidserver-diagnostics-20250113-2306.zip

3 hours ago, gdesoto said:

disk1/Media:<0x0>

This means metadata corruption, you should backup what you can and then reformat that disk.

I stopped the array. Set the format of disk1 to xfs. Started the array. All drives except disk1 mounted pending a format. I formatted disk1. Format completed showing the 12 TB free. Stopped the array. Switched it back to zfs. Tried to start the array and it hung on disk1 again.

Jan 14 09:12:43 UnraidServer kernel: WARNING: Pool 'disk1' has encountered an uncorrectable I/O failure and has been suspended.

 

I'm currently running "zpool clear disk1", but it's taking a while (going on 30+ minutes so far) and I'm not sure if it's doing anything. Prior to the format zpool clear would almost immediately throw another uncoorectable I/O failure error. So the fact that didn't reappear I assume means the format helped.

 

I'd attach diags, but the last time I tried while disk1 was suspended, diags also hung. Should I have done something else besides zpool clear?

Post new diags after that error.

Here's the syslog. Trying to get diags now hangs.

unraidserver-syslog-20250114-1701.zip

Strange because I don't see any i/o errors before the pool was suspended, but when a zfs pool is suspended, you need to force a restart, when that happens, zfs prevents any Linux OS from doing a clean shutdown

So how do I get unraid to recreate the disk1 pool so it knows it's been formatted?

Try wiping it before array start (all data will be lost, but it was already formatted anyway):

 

wipefs -a /dev/sdX1
then
wipefs -a /dev/sdX

 

Then start the array

Here's the results:

root@UnraidServer:~# wipefs -a /dev/sdg1
root@UnraidServer:~# wipefs -a /dev/sdg
/dev/sdg: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sdg: 8 bytes were erased at offset 0xae9fffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sdg: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdg: calling ioctl to re-read partition table: Success

 

I tried to start the array and disk1 hung everything up again.

 

Attached diags are from after the wipefs commands, but before starting the array. Attached syslogs is after starting the array.

unraidserver-syslog-20250114-2329.zip unraidserver-diagnostics-20250114-1727.zip

I think I got it! Unraid must be caching the zfs pool information somewhere.

 

I started the array in maintenance mode, and was still able to import the pool manually with readonly=on mode using:

zpool import -f -o readonly=on  -d /dev/md1p1 9186586923919620123 disk1

 

I checked the status and the same error info was still there from before. I then had to run the command:

zpool destroy disk1

 

Once I did that, I restarted the array not in maintenance mode, and Unraid finally let me format the disk back to zfs. I'm copying the data back now.