[SOLVED] EMERGENCY: Rebuilt drive, then formatted drive. Need to rebuild formatted drive again.

[David Spivey]

This is a total horror story. I had 2 drives go down. I have dual parity. I followed the procedure to begin replacement of the first drive that failed. I unassigned both drives, stopped the array, and added one (new) drive to the empty slot. While the drive was in the process of rebuilding, a power issue happened, and the system got powered off. When the system came back on, it reported that the rebuild was still in process. However, it was asking me to format the drive and the unassigned device. I left that request alone until the rebuild was completed on that drive, assuming it would go away after the rebuild. After the rebuild finished on the first drive, I rebooted. The message was still there, and the drives said they were unmoutable, so I made the mistake of using that format option. I now have an unassigned drive and the newly rebuilt drive that are both EMPTY. I am horrified. I need to know how to get this system back to normal.

Edited August 24, 2021 by David Spivey
solved

[JorgeB]

Formatting is never part of a rebuild, there's a big warning about it, not clear what you mean about the unassigned device, it would not be formatted by the GUI, if you mount it is it also empty?

[trurl]

You formatted both the emulated (missing) disk1 and the rebuilt disk7. So now not only is the rebuilt disk7 formatted, but anything you put in the disk1 slot for rebuild will rebuild a formatted disk1.

 

You knew format was a bad idea, yet you did it anyway before asking for help?

 

Possibly there was nothing really wrong with either disk and if you had posted Diagnostics and asked for advice before doing anything at all we could have told you what happened and how to proceed. Bad connections are much more common than bad disks.

 

Is the Unassigned Device the original disk1? Or is it the original disk7? Or what?

 

 

[trurl]

Since you apparently wound up with unmountable emulated disks I have to wonder about the cause. Probably bad connections but possibly problems with other disks. All other disks must be reliably read to reliably rebuild a disk.

 

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

 

If you have any Diagnostics or even just syslog from before rebooting attach that also.

[David Spivey]

@JorgeB The "not installed" disk from my screenshot has been formatted. I had no idea that unraid could or would even format an emulated drive.
@trurl Both disk1 and disk7 were almost fully filled disks in the array. The current disk7 is the replacement drive for that slot. My diags are attached.

 

diagnostics-20210816-1520.zip

[JorgeB]

8 minutes ago, David Spivey said:

The "not installed" disk from my screenshot has been formatted. I had no idea that unraid could or would even format an emulated drive.

I was asking about the unassigned disk:

 

[David Spivey]

2 minutes ago, JorgeB said:

I was asking about the unassigned disk:

 

This is the disk from disk1's slot. It is mountable and readable. The other drive that was unassigned from disk7's slot is outside of the system.

[JorgeB]

52 minutes ago, David Spivey said:

It is mountable and readable.

If it has the old data you can copy the data to the array, or do a new config assign it to the array and re-sync parity, same for disk7.

[David Spivey]

10 minutes ago, JorgeB said:

If it has the old data you can copy the data to the array, or do a new config assign it to the array and re-sync parity, same for disk7.

As far as assigning the disks and rebuilding parity, wouldn't that be assuming that the drives unraid previously said were bad were actually good?
If the drives are actually going bad won't parity sync fail?

It looks like I need to do some tests on these drives to see if they're bad first.

[JorgeB]

8 minutes ago, David Spivey said:

wouldn't that be assuming that the drives unraid previously said were bad were actually good?

Two drives going down at the same time is usually not a disk problem, the unassigned disk looks healthy, you'd need to connect the other one to check, but probably it's also good.

[David Spivey]

On 8/16/2021 at 12:31 PM, JorgeB said:

If it has the old data you can copy the data to the array, or do a new config assign it to the array and re-sync parity, same for disk7.

@JorgeB I took the original drives and tested them in another system. After diags, they're fine. If I understand correctly, I should put the original drives back in and remove the formatted ones, then create a new config. If I create a new config, and make sure the parity drives are parity drives and the data drives are data drives, does it matter the order of the slots the drives are in?

[JorgeB]

8 minutes ago, David Spivey said:

If I create a new config, and make sure the parity drives are parity drives and the data drives are data drives, does it matter the order of the slots the drives are in?

It doesn't, since parity will be be re-synced, just make sure you don't assign a previous data drive to a parity slot.

[JonathanM]

1 hour ago, David Spivey said:

If I understand correctly, I should put the original drives back in and remove the formatted ones, then create a new config.

Maybe safer to do the new config before putting the original drives in, that way Unraid can't try to write the corrupt data to those disks as well. Just be sure you have an accurate list of which drives are supposed to be in which slots and don't allow the array to start until everything is assigned correctly. Personally I would disable array autostart first thing.

[trurl]

1 hour ago, JonathanM said:

Maybe safer to do the new config before putting the original drives in

But you can't just add those original drives back into an already established array or they will be cleared.

[JonathanM]

24 minutes ago, trurl said:

But you can't just add those original drives back into an already established array or they will be cleared.

Yes. I'm talking about physically changing the drives after setting the new config to clear the assignments, but not starting the array until all the drives are correctly in place. I'm afraid since he has valid dual parity that after replacing the drives he will trigger a rebuild on those drives, overwriting the good info instead of rebuilding parity with the old good drives.

 

Perhaps I should be clearer with the physical vs. logical assignments, I read this

3 hours ago, David Spivey said:

If I understand correctly, I should put the original drives back in and remove the formatted ones, then create a new config.

and was afraid he would put the original drives in and immediately assign them to their intended slots, not realizing the need to create the new config before assigning the drives or starting the array.

[David Spivey]

5 minutes ago, JonathanM said:

Yes. I'm talking about physically changing the drives after setting the new config to clear the assignments, but not starting the array until all the drives are correctly in place. I'm afraid since he has valid dual parity that after replacing the drives he will trigger a rebuild on those drives, overwriting the good info instead of rebuilding parity with the old good drives.

 

Perhaps I should be clearer with the physical vs. logical assignments, I read this

and was afraid he would put the original drives in and immediately assign them to their intended slots, not realizing the need to create the new config before assigning the drives or starting the array.

Thanks for the concern. That was a reasonable concern considering how I messed things up already. Fortunately I had already been careful enough to go into disk settings and disable auto start before I rebooted to put the drives in.

[trurl]

2 minutes ago, David Spivey said:

Fortunately I had already been careful enough to go into disk settings and disable auto start before I rebooted to put the drives in.

Rebooted... So does that mean you already put the drives in and booted up?

[David Spivey]

7 minutes ago, trurl said:

Rebooted... So does that mean you already put the drives in and booted up?

Yep. New config already done. All drives carefully assigned to their proper places, and it's rebuilding parity now.

[trurl]

Post a screenshot and new diagnostics

[David Spivey]

8 hours ago, trurl said:

Post a screenshot and new diagnostics

 

dstorage-diagnostics-20210819-2345.zip

[trurl]

That looks good, data on disks 1,7, zeros in the Errors column with parity resyncing, nothing bad in syslog.