Unraid 6.10.3

zfs-2.1.5-1

zfs-kmod-2.1.5-1

 

If you need my diagnostics file, please let me know so I can send it to you via DM. I don't wish to post it publicly.

 

A few months ago I added a ZFS pool to my system (before Unraid 6.12 was released). It included (3) 3TB drives in what I believe was raidz. Well, I had not rebooted my system after it was all working until a few days ago and now the ZFS pool is gone.
 

zpool status
zpool import
zpool import -a
zpool import -D -f [poolname]
zfs list

all come back saying there is no pool available to import or no datasets available.

 

Using

ls -l /dev/disk/by-id/

I get the below for the 3 disks in question:

lrwxrwxrwx 1 root root  9 Jul  3 18:01 ata-Hitachi_HDS5C3030ALA630_MJ1311YNG39DAA -> ../../sdg
lrwxrwxrwx 1 root root  9 Jul  3 18:01 ata-Hitachi_HDS5C3030ALA630_MJ1313YNG1PTSC -> ../../sdh
lrwxrwxrwx 1 root root  9 Jul  3 18:01 ata-Hitachi_HDS5C3030ALA630_MJ1313YNG244VC -> ../../sdf

 

fdisk -l

Returns:

Disk /dev/sdf: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: Hitachi HDS5C303
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdg: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: Hitachi HDS5C303
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdh: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: Hitachi HDS5C303
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

 

I also cannot seem to find any zpool.cache files anywhere in /etc or /usr.

 

I presume I need to first recover the partitions on each of the 3 drives before I can even attempt to get them re-imported, but I am not sure how to do that. I've seen other posts talk about using sgdisk and specifying the partition start and end points, but I don't know what those values would be on my drives or where to find that info. Many other posts talk about using tools that I can't seem to find for Unraid or paid for tools that are simply out of the question here.

 

Can anyone provide me some guidance on how to recover the partitions and how to prevent this from happening in the future? I'll likely upgrade the OS to 6.12 after I can get the pool working again (even if I have to blow the whole thing out and lose the data).

 

EDIT:

gdisk -l /dev/sdg

Returns:

GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries in memory.
Disk /dev/sdg: 5860533168 sectors, 2.7 TiB
Model: Hitachi HDS5C303
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 7E12C536-4BE0-497C-9432-B1E75D2C4AB5
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 2048-sector boundaries
Total free space is 5860533101 sectors (2.7 TiB)

Number  Start (sector)    End (sector)  Size       Code  Name

 

dd if=/dev/sdg bs=512 count=2048 2>/dev/null | hexdump -C | grep EFI.PART

Returns nothing at all.

 

Considering I do have a spare one of those 3TB Hitachi drives, I could create a new ZFS pool on it, check the partition table positions, and then use sgdisk to reconstruct the partition table on the 3 other drives, correct? I'll probably use clonezilla to image the drives before doing anything just in case.

Edited July 5, 20233 yr by MrFrizzy

First thing would be to try and find out what happened to the partitions, are the disks connected to SATA/HBA or RAID controller?

 

As for the recovery, if you know how you created the original pool create one the same way using a spare 3TB disk then export and import that MBR, see here how, to play it safer you can export the MBR from the spare, clone one of the original disks to the spare then restore the MBR there to see if it works.

 

 

8 hours ago, JorgeB said:

...are the disks connected to SATA/HBA or RAID controller?

They are connected directly to a SAS2008 HBA via a HD mini-SAS to (4) SATA cable along with the (4) drives in my main array on a second HD mini-SAS to (4) SATA cable. The main array shows no signs of anything wrong. My other drives are plugged into the motherboard directly: (2) 500GB SATA drives for cache, (1) 250GB SATA via Unassigned Devices, and a 1TB NVME drive as a second BTRFS pool device. No issues to report with those either. 

 

8 hours ago, JorgeB said:

...if you know how you created the original pool...

I don't recall exactly, but I do know that it was RAIDZ and what name I used for the pool. Once I have all 3 drives imaged, I'll shut everything down, add my spare 3TB drive, and start testing the procedure with creating and copying the partition tables over.

 

The data on the ZFS pool isn't critical, I don't bother backing it up, but it would be nice to also not lose 5TB of data just because I restarted my machine.

Edited July 6, 20232 yr by MrFrizzy

33 minutes ago, MrFrizzy said:

but I do know that it was RAIDZ and what name I used for the pool.

That's not really important, the main thing is that you create the new pool the same way, e.g., if it was created using the CLI like this:

zpool create tank raidz1 /dev/sdx /dev/sdy /dev/sdz

just do the same with a single device

zpool create tank /dev/sdx

then backup and restore that MBR.

 

 

1 hour ago, JorgeB said:

the main thing is that you create the new pool the same way

Ah, the command line is the way I did it before and the only way I know how to. Should be pretty straightforward to create a new zpool on the spare and copy the partition info over. We'll see how it goes later today, time permitting.

 

Any idea how this could have happened in the first place? It would be one thing to have the pool not mount upon boot, or have an issue with a drive or two, but having the partition info randomly deleted on all 3 drives is not something I can explain. That is unless all of that information was never written to the drives or otherwise saved anywhere besides memory? Would that be possible?

13 hours ago, MrFrizzy said:

Any idea how this could have happened in the first place?

Not really, assuming the HBA is in IT mode, especially because it happened to all the pool disks, but only those disks, very strange. 

Update time!

 

I added the spare 3TB drive to my system and created a new zpool on it using:

zpool create pool /dev/sdl

Then ran fdisk to get the partition info:

fdisk -l /dev/sdl
Disk /dev/sdl: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: Hitachi HDS5C303
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 018D37A8-E21E-DC4B-AC14-7D3B1CB9CBFA

Device          Start        End    Sectors  Size Type
/dev/sdl1        2048 5860515839 5860513792  2.7T Solaris /usr & Apple ZFS
/dev/sdl9  5860515840 5860532223      16384    8M Solaris reserved 1

Then ran sgdisk to copy that info over to the 3 other drives:

sgdisk -n1:2048:5860515839 -t1:BF01 -n9:5860515840:5860532223 -t9:BF07 /dev/sdi
sgdisk -n1:2048:5860515839 -t1:BF01 -n9:5860515840:5860532223 -t9:BF07 /dev/sdj
sgdisk -n1:2048:5860515839 -t1:BF01 -n9:5860515840:5860532223 -t9:BF07 /dev/sdk

EDIT:

Forgot to include that I had to re-import the pool first before the scrub:

zpool import -a

Then started a scrub. The data seems to be intact so far, but the scrub is going to take more than 12 hours looking at the average disk speeds.

Edited July 7, 20232 yr by MrFrizzy

If the pool is back data should be intact.

I agree. I don't expect the scrub to come back with any errors but it's always good to be certain.

 

After 2 days of researching this issue, it seems that the disk partitions created by ZFS are deterministic. Do you have any insight on that?

 

At least for drives that have 512-byte logical sector sizes (even if the physical sector size is 4096), the sources I can find within the last few years all seem to show that ZFS will align the first partition to sector 2048 and the last partition to exactly 8MiB before the last full 2048 sector chunk of the disk. I didn't really find any info on drives with 4096-byte logical sectors (4K Advanced Format) so I would imagine the behavior there is different. I don't have any 4096-byte logical sector drives to test with and even if I did, my SAS2008 HBA doesn't support 4K logical sectors, only 512 physical and emulated.

 

By my math that means the partition start and end points can be calculated based purely on the total sector count of a given drive. That could save someone who didn't have a spare identical drive or a working identical drive still in the pool. It would also mean no need to create a sparse qemu-img somewhere of the same sector count as the drive in question just to create a new zpool on it in order to grab the partition info (which didn't work for me).

For 512-byte logical sector drives, calculate the end of partition 9:
1. Total sector count of drive / 2048
2. Get rid of everything after the decimal point
3. Multiply by 2048
4. Subtract 1

For example:
1. 5860533168 / 2048 = 2861588.46094
2. 2861588
3. 2861588 * 2048 = 5860532224
4. 5860532224 - 1 = 5860532223 = part9 end sector

Then get the partition 9 start sector:
1. Take the partition 9 end sector and subtract 16383 (in this example: 5860532223 - 16383 = 5860515840)

Lastly, get the partition 1 end sector by subtracting 1 from the partition 9 start sector (in this example: 5860515840 - 1 = 5860515839)

 

Thoughts?

7 minutes ago, MrFrizzy said:

By my math that means the partition start and end points can be calculated based purely on the total sector count of a given drive.

Probably yes for zpool create, but depends on how they were created, if using Unraid v6.12 for example Unraid will create the partition and zfs just formats it, Truenas creates a 2GB swap partition and puts zfs on partition #2, hence why I mentioned that the important part was creating the pool using the same way as before.

8 minutes ago, JorgeB said:

depends on how they were created

Thanks for the examples, that makes a lot of sense! It seems that every distro uses ZFS a little differently and there is no one way to go about things.