Hi,

 

Since a week or so, I've been having problems with my server, disk 17 and 19 were reporting errors.

 

I wanted to replace disk 17 with a new disk so shutdown the server and changed disks.

Upon reboot there were more errors on other disks, all using the same icybox bay (5xhdds) and breakout cable, so I changed cables and no change, thinking it was maybe the card I plugged in an old Adaptec ASR-5405 I had lying around and connected the bay to this card and restarted, I then had the error "mount error: Unsupported partition layout" on 4 of the disks.

 

Thinking that maybe the card wasn't compatible with UNRAID (yes I should have checked first), I disconnected the card and reconnected as before but the error remains on all these drives.

 

I tried a xfs_repair (first with -n then without) and no change.

 

I searched on the forum and found this thread that mentions the Adaptec ASR-5405 and mount error: Unsupported partition layout errors, so I tried connecting one of the drives directly to the motherboard through SATA and still no change.

I also wanted to try mounting the drive using Unassigned devices but instead of "mount" it wants a format, so obviously I stopped there.

 

Can anyone please help me recover these drives please?

 

 

msstorage-diagnostics-20210806-1936.zip

Edited August 8, 20214 yr by footballmad

I ran this sgdisk command and these are the results.

 

root@MSSTORAGE:~# sgdisk -v /dev/sdc
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!
Main header: OK
Backup header: ERROR
Main partition table: OK
Backup partition table: ERROR

Invalid partition data!
Verification may miss some problems or report too many!

Caution: The CRC for the backup partition table is invalid. This table may
be corrupt. This program will automatically create a new backup partition
table when you save your partitions.

Warning! Mismatched GPT and MBR partition! MBR partition 1, of type 0x83,
has no corresponding GPT partition! You may continue, but this condition
might cause data loss in the future!

Identified 2 problems!

 

Does this information help anyone that knows more than me about this?

  

10 hours ago, footballmad said:

"mount error: Unsupported partition layout" on 4 of the disks.

Which 4? Was it 17,18,20,21?

 

10 hours ago, footballmad said:

xfs_repair

 

Did you do this from the webUI or from the command line?

10 hours ago, trurl said:

  

Which 4? Was it 17,18,20,21?

 

 

Did you do this from the webUI or from the command line?

I'm not home right now but I believe it's the 4 disks you mention.

 

I did this from the webUI, I forgot to mention in my previous posts, these disks are encrypted.

8 hours ago, footballmad said:

these disks are encrypted

Could see that from diagnostics. Might make recovery difficult.

 

On 8/8/2021 at 5:16 AM, footballmad said:

tried a xfs_repair

The usual and documented method.

 

On 8/8/2021 at 10:34 AM, footballmad said:

sgdisk command

Where did you get this idea from?

4 hours ago, trurl said:

Could see that from diagnostics. Might make recovery difficult.

Oh no, not what I wanted to hear.

 

4 hours ago, trurl said:

The usual and documented method.

Would you recommend using the -L flag?

 

4 hours ago, trurl said:

Where did you get this idea from?

From another post here, what's your opinion on this?

 

Do you think the fact that I tried a different expansion card could've caused this?

 

Any ideas on how to attempt recovery?

43 minutes ago, footballmad said:

Do you think the fact that I tried a different expansion card could've caused this?

Could be, do you know what it was, do you still have the other card?

9 hours ago, trurl said:

Could be, do you know what it was, do you still have the other card?

Yes and all these disks are connected tot the old card as before, when I did that I was hoping that UNRAID would see them as before and that it was possibly a "bug" with the Adaptec ASR-5405 card.

I'd love people's ideas on if there's a possible recovery and how to proceed.

4 hours ago, footballmad said:

I'd love people's ideas on if there's a possible recovery and how to proceed.

@JorgeB

Very likely the Adaptec controller changed the MBR on the disks, don't use RAID controllers with Unraid, please post diags after array start.

Thanks for taking a look Jorge

msstorage-diagnostics-20210812-2015.zip

Not related, but you seem to be installing every package from NerdPack. Do you know what any of that is for? I recommend only installing those that you use regularly.

14 minutes ago, trurl said:

Not related, but you seem to be installing every package from NerdPack. Do you know what any of that is for? I recommend only installing those that you use regularly.

I know what a lot of it is for, I guess the only reason is being a wannabe nerd, trying to learn as much as I can.

 

I appreciate your time too trurl

Edited August 12, 20214 yr by footballmad
Thanks

atop is notorious for filling log space

8 minutes ago, trurl said:

atop is notorious for filling log space

It's not something I really use, so I guess it's best to uninstall?

Why did you replace disk 17 now? Log is a mess, there are hardware errors for a disk and a strange errors with disk19, you didn't start the array in normal mode so disks weren't mounted, but for now stop the rebuild and post the output of:

 

fdisk -l /dev/sdX

 

for all the invalid partition disks.

6 hours ago, JorgeB said:

Why did you replace disk 17 now? Log is a mess, there are hardware errors for a disk and a strange errors with disk19, you didn't start the array in normal mode so disks weren't mounted, but for now stop the rebuild and post the output of:

 

fdisk -l /dev/sdX

 

for all the invalid partition disks.

Because the disk was failing.

 

I'll post the results once home.

31 minutes ago, footballmad said:

Because the disk was failing.

You should first attempt to solve the other issues, or it might make things worse.

The weird errors on disk19 appear to be because you assigned a smaller disk than it was there before, though not sure how you did that:

 

Aug 12 20:12:41 MSSTORAGE kernel: md: disk19 write error, sector=7814036864
Aug 12 20:12:42 MSSTORAGE kernel: attempt to access beyond end of device

 

Current:

 

[id] => ST3000DM001-1CH166_W1F2G6PP
            [size] => 2930266532

 

Previous:

[idSb] => ST4000VN008-2DR166_ZGY92QQX
            [sizeSb] => 3907018532

 

 

41 minutes ago, JorgeB said:

though not sure how you did that

IIRC there's an old bug that if you cancel a rebuild, and while the disk status is "invalid" as opposed to "disable", you can assign a smaller disk.

But in case it wasn't clear you now need to assign a 4TB (or larger) disk there.

This is the result for disk21

 

root@MSSTORAGE:~# fdisk -l /dev/sdac
The primary GPT table is corrupt, but the backup appears OK, so that will be used.
Disk /dev/sdac: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST3000DM001-1ER1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 205CED32-7A0F-4E32-81F6-36E483C5065F

Device     Start        End    Sectors  Size Type
/dev/sdac1    64 5860533134 5860533071  2.7T Linux filesystem

 

Then I started the array and the error remains.

1 minute ago, JorgeB said:

But in case it wasn't clear you now need to assign a 4TB (or larger) disk there.

OK, I'll look into that if we can manage to recover the others.

 

At the time of starting the array, I was sitting in front of the console and this appeared.

  

Don't know if that can help in any way.