This morning I was informed of an unclean shutdown, I looked at the array (all zfs formatted) and all seemed fine, so i started the array. I went to get coffee an came back to hundreds of thousands of errors corrected during parity sync. One of my drives showed as "Device Disabled, contents emulated". I examined the zpool status and (my mistake) issued a scrub cause I thought the drive was online but just needed a scrub to clear errors. a few minutes later I figured out it was using the entire array emulation to 'scrub' the disk which didnt make sense to me, and cancelled that...

At this point I decided to stop and seek advice. Specific steps to restore the array with as little loss as possible. I have a new hard drive arriving tommorow. But I am curious the 'order of operations' of steps I should be taking to get back on track. 

Edited January 27, 20251 yr by Cyong

My current thoughts are to replace the drive, let the rebuild occur, then perform a scrub, and use whatever the output is of zpool status -v to identify specific files to delete/restore from backup

 

Please post the diagnostics.
 

unraid-diagnostics-20250127-1443.zip

SMART for the disable disk looks fine, but there are HBA related issues logged:

 

Jan 27 02:28:31 UnRaid kernel: mpt3sas_cm0 fault info from func: mpt3sas_base_make_ioc_ready

 

Make sure it's well seated and sufficiently cooled

Also, the emulated disk is mounting, but it's mostly empty, I assume it's expected?

Ok, I have ensured the cable are seated. Its a 20tb drive but should have about 17.9 TB of data (according to the Unraid UI)

What steps should I take to repair the zpool status and restore the array health? 

 

run parity check then zfs clear/scrub or reverse?

Edited January 27, 20251 yr by Cyong

11 hours ago, Cyong said:

Ok, I have ensured the cable are seated.

Not the cable, the HBA.

 

11 hours ago, Cyong said:

Its a 20tb drive but should have about 17.9 TB of data (according to the Unraid UI)

 

Emulated disk11 is 2TB and is basically empty.

Filesystem                                                                     Size  Used Avail Use% Mounted on
disk11                                                                         1.9T   35G  1.8T   2% /mnt/disk11

 

Ok, I have ensured the HBA is seated fully. (Didnt feel like it moved)

 

Could that be due to the zfs scrub i cancelled when i was confused why it was using the entire array? 

54 minutes ago, Cyong said:

Could that be due to the zfs scrub i cancelled when i was confused why it was using the entire array?

Since the disk is emulated, any i/o will involve all other disks, but that has nothing to do with capacity or used space.

hmmm, ok... So what is the path forward then for a disabled disk in the array and the zfs errors on the disk?

 

 pool: disk11
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub canceled on Mon Jan 27 08:50:20 2025
config:

	NAME                  STATE     READ WRITE CKSUM
	disk11                ONLINE       0     0     0
	  /dev/mapper/md11p1  ONLINE       0     0     0

errors: 1 data errors, use '-v' for a list

 

Edited January 28, 20251 yr by Cyong

Check the output with -v to see the list of corrupt files.

Ok, its a replaceable file from backup. What next? 

If the emulated disk contents look correct, you can rebuild on top:

 

https://docs.unraid.net/unraid-os/manual/storage-management#rebuilding-a-drive-onto-itself

Small update. I followed the instructions to remove the disk from the array, and then start, stop, re-add, let the rebuild occur. However a few hours later the rebuild 'paused' around 12% (interestingly, that would be about 1.8TB...) and the disk was disabled again. I have installed a new hard drive yesterday and it has been building into the array very somethly and should complete sometime later today.

 

I am guessing after it completes I should perform a zfs scrub on that disk, make note of any uncorrectable files, delete them and restore from backup?

Yep