Hello,

 

I recently got two drives to add to my system, which I first precleared.  For the first drive, I replaced an existing drive in the system (Disk 1).  I let data rebuild complete on the drive.  All looked good and rebuild completed successfully.

 

For the second drive, I stopped the array, added it as a new drive in an empty slot (Disk 2), and again started the array.  Both the recently-rebuilt, and the newly added drive, now say "Unmountable: Unsupported partition layout".  Disk 1 shows xfs (correct, that's what the prior drive was) and Disk 2 shows auto.  But both show this unmountable error and a request to reformat the drives.

 

I stopped the array without formatting.  I'm not sure how to proceed.  Formatting will delete all data on the rebuilt drive, but I do need to format the new drive.  What do I do?

 

Screenshot and diagnostic log attached.

 

Here are the relevant fdisk entries:

 

sdc is Disk 1, which was recently rebuilt.  It shows a Linux Filesystem

Disk /dev/sdc: 14.55 TiB, 16000900661248 bytes, 31251759104 sectors
Disk model: WDC WD161KFGX-68
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8249EC25-BC98-446C-BACF-B85E6F06A273

Device     Start         End     Sectors  Size Type
/dev/sdc1     64 31251759070 31251759007 14.6T Linux filesystem

 

sdp is Disk 2, the one I just added to an empty slot as a new drive.  It shows unformatted.

Disk /dev/sdp: 14.55 TiB, 16000900661248 bytes, 31251759104 sectors
Disk model: WDC WD161KFGX-68
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start        End    Sectors Size Id Type
/dev/sdp1           1 4294967295 4294967295   2T  0 Empty

Partition 1 does not start on physical sector boundary.

 

Perhaps worth mentioning that sdk is a precleared warm spare, so is not expected to have a filesystem.

 

I saw a vaguely similar post where the output of lsblk was requested, so here that is (run without the array started).  Seems like sdc1 being 2T is indicative of the problem here.  Presumably that should be 14.6T.  This conflicts with what fdisk says above for the same drive, though, so I am confused.

root@Hyperion:~# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0  63.5M  1 loop /lib
loop1    7:1    0 348.5M  1 loop /usr
sda      8:0    1   3.7G  0 disk
└─sda1   8:1    1   3.7G  0 part /boot
sdb      8:16   0   3.6T  0 disk
└─sdb1   8:17   0   3.6T  0 part
sdc      8:32   0  14.6T  0 disk
└─sdc1   8:33   0     2T  0 part
sdd      8:48   0 931.5G  0 disk
└─sdd1   8:49   0 931.5G  0 part
sde      8:64   0  14.6T  0 disk
└─sde1   8:65   0  14.6T  0 part
sdf      8:80   0   3.6T  0 disk
└─sdf1   8:81   0   3.6T  0 part
sdg      8:96   0   3.6T  0 disk
└─sdg1   8:97   0   3.6T  0 part
sdh      8:112  0  14.6T  0 disk
└─sdh1   8:113  0  14.6T  0 part
sdi      8:128  0  14.6T  0 disk
└─sdi1   8:129  0  14.6T  0 part
sdj      8:144  0 931.5G  0 disk
└─sdj1   8:145  0 931.5G  0 part
sdk      8:160  0  14.6T  0 disk
└─sdk1   8:161  0     2T  0 part
sdl      8:176  0  14.6T  0 disk
└─sdl1   8:177  0  14.6T  0 part
sdm      8:192  0   3.6T  0 disk
└─sdm1   8:193  0   3.6T  0 part
sdn      8:208  0   3.6T  0 disk
└─sdn1   8:209  0   3.6T  0 part
sdo      8:224  0   3.6T  0 disk
└─sdo1   8:225  0   3.6T  0 part
sdp      8:240  0  14.6T  0 disk
└─sdp1   8:241  0     2T  0 part
sr0     11:0    1  1024M  0 rom

 

hyperion-diagnostics-20241205-1842.zip

Edited December 6, 20241 yr by Gizmotoy

You should try running a check filesystem on disk1.

 

Do NOT format anything at this stage if you do not want data loss.   You WILL have to format the newly added drive before you can use it but do not do it until disk1 is sorted out.

Invalid partition layout means the partition or MBR signature does not confirm to what Unraid expects, for diosk2 it's obvious why, not so clear for disk1, but if you disable the disk, Unraid should recreate the partition correctly:

 

Stop the array, unassign disk2 first, start the array, post new diags.

6 hours ago, JorgeB said:

Invalid partition layout means the partition or MBR signature does not confirm to what Unraid expects, for diosk2 it's obvious why, not so clear for disk1, but if you disable the disk, Unraid should recreate the partition correctly:

 

Stop the array, unassign disk2 first, start the array, post new diags.

 

I see.  Ok, I unassigned disk2, then started the array with disk2 missing.  It did not change the error associated with disk1.  Attached are the new diagnostics.

 

I did not run the check filesystem step on disk1 yet.  The link is not clear if it should be run in standard mode or in maintenance mode.  Since the repair could change the contents (I think? Or maybe invalidate current parity?), I wanted to be really clear how to run it.  So to clarify: start array in standard mode, run check filesystem from the webgui?

hyperion-diagnostics-20241206-0658.zip

Edited December 6, 20241 yr by Gizmotoy

1 hour ago, Gizmotoy said:

 It did not change the error associated with disk1.

That's expected, but sorry because I missed that disk2 was new and not formatted, rebuild disk2 first, don't format it, or it would also need to format disk1 together, and once the rebuild is done, unassign disk1, start array and post new diags.

20 minutes ago, JorgeB said:

That's expected, but sorry because I missed that disk2 was new and not formatted, rebuild disk2 first, don't format it, or it would also need to format disk1 together, and once the rebuild is done, unassign disk1, start array and post new diags.

Sorry, I'm a little slow here and want to move cautiously.  Just to confirm:

Disk1 - Recently-rebuilt disk.  Should have data on it, but has some kind of MBR issue and won't mount.

Disk2 - Newly added disk (precleared, but unformatted) in a brand new, previously-unused slot.  Was not a previous disk in the system, so would have no data and does not need to be rebuilt.

 

Unfortunately I found out about the problem with Disk1 when I added Disk2 to the system and started the array, complicating the problem.  Unfortunately it seems like I can't just go back to the state without Disk2 to deal with the problem with Disk1 on its own (if I remove the Disk2 assignment, Unraid complains it is missing as the most recent diagnostic logs show).

 

I assume we're trying to get to a place where disk1 gets rebuilt again?  I'm not sure how to do that with disk2 in its current state.

Edited December 6, 20241 yr by Gizmotoy

55 minutes ago, Gizmotoy said:

I assume we're trying to get to a place where disk1 gets rebuilt again?

Correct, first we want to emulate it, but for that disk2 must be re-enabled, to do that:

 

stop the array

reassign disk2

start the array

 

disk2 will also still show unmountable during the rebuild, just let it rebuild that way, don't format anything, once the rebuild is done, stop the array, unassign disk1, start the array, post new diags.

6 minutes ago, JorgeB said:

Correct, first we want to emulate it, but for that disk2 must be re-enabled, to do that:

 

stop the array

reassign disk2

start the array

 

disk2 will also still show unmountable during the rebuild, just let it rebuild that way, don't format anything, once the rebuild is done, stop the array, unassign disk1, start the array, post new diags.

How do I actually trigger the rebuild of disk1? When I reassign disk2 I’m back at the initial state from my first post.

 

Are the steps then actually:

1) Stop the array

2) reassign disk 2

3) start the array

 

Now disk1 is still invalid/corrupted, and disk2 is back in unformatted state. 

 

4) stop the array

5) unassign disk 1

6) start the array

 

disk1 is now in emulated state and disk2 is unformatted

 

7) stop the array

8 reassign disk1

9) start the array

 

disk1 is now emulated and rebuilding, disk2 is unformatted

 

is that about right?

Edited December 6, 20241 yr by Gizmotoy

Didn't notice that you have dual parity, must be needing some coffee, post current diags.

6 minutes ago, JorgeB said:

Didn't notice that you have dual parity, must be needing some coffee, post current diags.

 

Sure, no worries.  Diag attached.

 

Current state:

disk1 - assigned, but somehow corrupted

disk2 - unassigned (and "missing")

 

hyperion-diagnostics-20241206-1025.zip

Unassign disk1, leave disk2 unassigned, start the array and don't do anything else for now, just post new diags.

5 minutes ago, JorgeB said:

Unassign disk1, leave disk2 unassigned, start the array and don't do anything else for now, just post new diags.

Done.

 

disk1 emulated, but back to good state (mounted correctly).  disk2 emulated and unformatted.

 

No action taken beyond starting the array in this state and producing the diags.

hyperion-diagnostics-20241206-1048.zip

Edited December 6, 20241 yr by Gizmotoy

Now stop the array

re-assign disk1

re-assign disk2

start the array to begin rebuilding both

 

If disk2 is still the only unmountable disk, and it should be, you can format it during the rebuild.

12 minutes ago, JorgeB said:

Now stop the array

re-assign disk1

re-assign disk2

start the array to begin rebuilding both

 

If disk2 is still the only unmountable disk, and it should be, you can format it during the rebuild.

 

Ok, done, and disks are rebuilding.  disk2 is the only unmountable disk.  Before I format, figured I'd post one last diags

 

Also, to be clear, I should probably be gentle with the server right now because I'm without parity protection, correct?  Since 2x drives are rebuilding?

 

hyperion-diagnostics-20241206-1101.zip

Edited December 6, 20241 yr by Gizmotoy

Just now, Gizmotoy said:

because I'm without parity protection, correct?  Since 2x drives are rebuilding?

Correct, avoid high i/o with the array, or it can slow down the rebuild.

Rebuild finished successfully.  I then formatted the drive I added.

 

Then I stopped and started the array a few times to check that Disk1 mounts properly, and it did.

 

I re-enabled Docker to get the server back to fully-operational, and kicked off a parity check just as final confirmation everything is good again (running without issue so far).  I appreciate the help getting things back up and running!