All Array Disks show unmountable: Unsupported partition layout and Cachepool btrfs shows unsupported or no file system after reboot

[JorgeB]

See my edit, it changed:

 

[Schulmeister]

It is checking - many messages like: imap claims in-use inode 2565432 is free, would correct imap.

I am not very hopeful.

Let's wait and see

[Schulmeister]

I am still absolutely baffled how that could happen ! I have no idea.

[JorgeB]

Once done also check the emulated disk to se if it's any better, UD disk needs to be unmounted or the emulated disk will mount at the same time.

[Schulmeister]

3 hours ago, JorgeB said:

Once done also check the emulated disk to se if it's any better, UD disk needs to be unmounted or the emulated disk will mount at the same time.

 

Should I press Run or Done ?

[JorgeB]

Run

[Schulmeister]

it worked - So I do this for all the Seagate HDDs of the same size. How about the Toshiba ? Rebuild like the 18 TB HDD ?

[JorgeB]

12 hours ago, Schulmeister said:

How about the Toshiba ? Rebuild like the 18 TB HDD ?

That's what I would recommend.

[Schulmeister]

Good morning Jorge,

sadly the trick with the part_table copying worked on just 2 HDDs - with the others UD shows a partition but is not able to check them - very strange.

 

Any more suggestions on the BTRFS Cachedrive ?

Any ideas how the partition tables on all Array-Drives incl. Cache and excluding Parity (lucky me) could have happened ? I am kinda spooked.

[JorgeB]

1 hour ago, Schulmeister said:

Any more suggestions on the BTRFS Cachedrive ?

If btrfs restore doesn't work not much more it can be done, that error means pasts of the filesystem were also deleted, not just the partition layout.

 

1 hour ago, Schulmeister said:

Any ideas how the partition tables on all Array-Drives incl. Cache and excluding Parity (lucky me) could have happened ? I am kinda spooked.

Parity doesn't care about the partition layout, not sure what could have happened to destroy the partitions of so many devices, including NVMe, would be worried about someone getting access to the server and using for example dd to overwrite the beginning of every device. Is your server exposed to the internet?

 

[Schulmeister]

22 minutes ago, JorgeB said:

If btrfs restore doesn't work not much more it can be done, that error means pasts of the filesystem were also deleted, not just the partition layout.

 

Parity doesn't care about the partition layout, not sure what could have happened to destroy the partitions of so many devices, including NVMe, would be worried about someone getting access to the server and using for example dd to overwrite the beginning of every device. Is your server exposed to the internet?

 

Yes it is, but there is a firewall and access is only via vpn 

[Schulmeister]

Hi Jorge,

the trick with unassigned devices only worked on the first HDD (disk2) every other try failed.

I unassigned disk4 (my VM-backups are on 4+5) to start the recovery process and waited another day.

After the Data-Rebuild was done, I restarted the server and tried to Start the Array but the Toshiba was still shown as non-mountable.

Why does this happen after a complete rebuild

I don't know if I have to format the disks before rebuild ? But that was never mentioned - maybe the overwriting of the partition table wasn't such a good idea ?

 

The I tried disk5 - I hope I did'nt break things - it looks slightly different then with the first try.

Edited August 9, 2023 by Schulmeister

[JorgeB]

Post new diags

[Schulmeister]

Thanks and good morning

hsunraid-diagnostics-20230810-1146.zip

Edited August 10, 2023 by Schulmeister

[JorgeB]

Check filesystem on disks 3 through 6, disk6 at least is showing an xfs filesystem, others are not, but maybe it can be recovered from a backup superblock.

[Schulmeister]

as I mentioned - I tried to copy the table - like you told me on every disk from 3 to 6, with the UD nothing was recoveryble (after hours and hours of trying)
Then I went back to recovering the Disks vie emulated via parity - weirdly it seems the Unraid is mistaken these disks (3-6) for disk1 or disk2.

Maybe that has s.th. to do with the copied table ??

 

I have pause the data rebuild on disk5 - because it shows nothing new.

Can I resart and try disk6 or is that too risky ?

Edited August 10, 2023 by Schulmeister

[JorgeB]

No, that's just the partition info and it's outside parity, check filesystem on all.

[Schulmeister]

Can I take the array offline during a rebuild process ? Should I cancel the rebuild or let it run another 18 hrs ?

[itimpi]

1 minute ago, Schulmeister said:

Can I take the array offline during a rebuild process

Not completely, but you can start the array in Mainenace mode rather than normal mode.

[Schulmeister]

Sorry, english is not my first language.

From the beginnig:

1. I tryied to rebuild disk1 from parity - worked !

2. I tryied to rebuild disk2 vie copying the part-table from disk1 - worked !

-----

Jorge wrote this:

With array stopped:

sfdisk -d /dev/sdX > part_table

replace X with disk1

sfdisk /dev/sdY < part_table

replace Y with other disk.

 

Then reboot and see if it's back, if yes, do it for the others, except disk7, if not do the rebuild thing.

-----

3. I tryied the same with disks 3-6 ... that did not work.

 

4. I went back to rebuilding from parity and started with disk 4 (Toshiba) and after 20 hrs it told me rebuild is ready but the disk was still not mountable.

 

So my question is: maybe I should do a clean format of the hdd before rebuilding the disks ?

 

I am very sorry for being such an annoyance but I am desperate and very nervous

Edited August 10, 2023 by Schulmeister

[JorgeB]

4 minutes ago, Schulmeister said:

So my question is: maybe I should do a clean format of the hdd before rebuilding the disks ?

That won't make any difference, Unraid still recreates the partition (overwriting the other one) on a rebuild, also you can just check by starting the array without any disk assigned, whatever shows on the emulated disk is what's going to show on the rebuilt disk.

[Schulmeister]

5 hours ago, JorgeB said:

That won't make any difference, Unraid still recreates the partition (overwriting the other one) on a rebuild, also you can just check by starting the array without any disk assigned, whatever shows on the emulated disk is what's going to show on the rebuilt disk.

Can I try this with multiple disks at once (eg. disk3, 4, 5) or would you suggest to rebuild one disk at a time ?

 

I tried it with disk 4 (Toshiba) but the emulated disk did not show any data. Now I am trying disk 5 - sadly the same emptyness.

Edited August 10, 2023 by Schulmeister

[JorgeB]

With single parity you can only emulate one disk at a time, if the emulated disk doesn't mount check filesystem on it and post the output.

 

 

[Schulmeister]

At the moment I am trying disk 5 - it shows that the disk is emulated but that's it - no disk, files or folders.

 

[JorgeB]

6 minutes ago, JorgeB said:

check filesystem on it and post the output.