Force Read Only Array Mount with Old Parity Drive

[Kevin T]

I have a 1 parity unRAID system, I was preparing to upgrade multiple drives to larger sizes. 

 

First I did the parity drive (Drive 1) upgrade which was okay. 

After that completed I tried upgrading another drive (Drive 5) and during the rebuild it completely failed and isn't even recognized after rebooting.  While that drive was bad, I just tried to copy a small file from another drive (Drive 2) and the server locked up.  Then that Drive 2 is now not being recognized by the BIOS and doesn't seem to be spinning up.

 

I was wondering if there is a way I can read-only mount the array using my original parity drive (Drive 1) and my original Drive 5 without the Drive 2 being present where I can emulate the data on it and copy it off to another drive?

 

The system was somewhat unchanged during this process but I am not sure if my new parity drive had more data written to it than my original parity drive so I figured that might be the best option.  I understand there may be a few files where the parity may fail.

 

The GUI now thinks all the swapping attempts are new drives so it doesn't see the Drives 1 and 5 now being part of the array, so I'm not sure if it can be done in a commandline or if I can modify some config files?

Thank you,
Kevin

 

 

[JorgeB]

You can do a new config to emulate drive2 (I assume that is disk2, but since you called parity drive1 adjust as necessary), but you need a spare disk of the same size or larger as the old one, and since parity won't be 100% valid there might be some corruption, I'll leave the instructions if you want to try that, also note that there' no way of starting the array read only, but as long as there are no writes it should not be a problem.

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including the new disk you want to emulate, replacement disk should be same size or larger than the old one
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign the disk you want to emulate
-Start array (in normal mode now), ideally the emulated disk will now mount and contents look correct, if it doesn't you should run a filesystem check on the emulated disk
-If the emulated disk mounts and contents look correct you can access it or even rebuild, to rebuild stop the array
-Re-assign the disk to rebuild and start array to begin.

[trurl]

Please don't call parity, Drive 1. It makes things very confusing. You can call it Drive 0 if you want.

 

You seem to be calling Drive 2 and Drive 5 by their assignments in the array but that might be assuming too much. What would you call Drive 1 if we needed to refer to it? 

 

No real evidence that any disk is bad, connection problems are much more common. 

 

See if you can get the disks reconnected and 

 

Attach Diagnostics to your NEXT post in this thread 

[Kevin T]

10 hours ago, JorgeB said:

You can do a new config to emulate drive2 (I assume that is disk2, but since you called parity drive1 adjust as necessary), but you need a spare disk of the same size or larger as the old one, and since parity won't be 100% valid there might be some corruption, I'll leave the instructions if you want to try that, also note that there' no way of starting the array read only, but as long as there are no writes it should not be a problem.

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including the new disk you want to emulate, replacement disk should be same size or larger than the old one
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign the disk you want to emulate
-Start array (in normal mode now), ideally the emulated disk will now mount and contents look correct, if it doesn't you should run a filesystem check on the emulated disk
-If the emulated disk mounts and contents look correct you can access it or even rebuild, to rebuild stop the array
-Re-assign the disk to rebuild and start array to begin.

 

That is very helpful thank you, I am ordering another drive and will try what you suggest after I receive it, I just have a few questions:

 

1. Does a New Config only overwrite the disk identification related settings or everything like shares, etc.?
2. Would you recommend using the older pre-issue parity or the most recent parity when the issues occurred?  They are from the same day just hours apart and since the array won't mount, the server has been offline since.
3. When rebuilding the drive, if there is some corruption, that will only affect the drive that is rebuilding and not the other drives that are okay, correct? 
4. Is there a way to scan for / detect any corruption while the drive is being emulated that I can maybe try to fix?

 

Thank you,

Kevin

 

[trurl]

1. New Config allows you to change drive assignments, and optionally (by default) rebuild parity. In your case you don't want to rebuild parity. New Config doesn't affect anything else.
2. Current parity is the one you should use.
3. Rebuild has no effect on other disks.
4. The emulated disk can be accessed just as if it is really there. Its contents are emulated from the parity calculation.

I still recommend posting diagnostics, and working with us instead of trying to follow those instructions by yourself.

 

[Kevin T]

The drive that stopped working sounds like it keeps trying to spin up, but I have tried it in 2 different systems and the BIOS does not recognize it at all.  When I swap for a different drive it recognized the new one in the same position immediately and they are hot swap bays so it's not a connection issue.

 

I can turn it on and send a diagnostics, I just like to keep it turned off right now since I have no protection.
 

Edited October 1, 2022 by Kevin T

[trurl]

2 hours ago, Kevin T said:

ordering another drive

Let us know when you get it, we can get diagnostics when you are ready to work through this

[Kevin T]

On 10/1/2022 at 5:31 PM, trurl said:

Let us know when you get it, we can get diagnostics when you are ready to work through this

 

My new drive is arriving today so I wanted to attempt this tonight, here is my diagnostics.

 

Thank you,
Kevin

unraid-diagnostics-20221001-1815.zip

[trurl]

Oct  1 18:11:19 unRAID emhttp: unRAID System Management Utility version 6.1.9

Even though you just joined the forum, you are running an extremely old version of Unraid. Why?

 

This is a tricky situation you are in currently, requiring several steps to get it right. Nobody has that old version anymore so any advice we could give about this situation may not accurately reflect how you must work through this.

 

I doubt the instructions you have already been given will work exactly as described. Possibly have to go to the command line with the old 'invalidslot' method.

 

Maybe a screenshot will help refresh some memories.

 

@JorgeB may not rejoin this thread until tomorrow since he is in Europe

 

 

[trurl]

On 10/1/2022 at 6:43 AM, JorgeB said:

-Unassign the disk you want to emulate
-Start array (in normal mode now), ideally the emulated disk will now mount and contents look correct, if it doesn't you should run a filesystem check on the emulated disk

Is this even possible with invalidslot? Since rebuild will be to a new disk I guess it doesn't matter if filesystem repair has to happen after rebuild instead of before.

[trurl]

One thing I will warn you about. DO NOT format any array slot, whether a disk is assigned to it or not.

[Kevin T]

I tried automatically upgrading 2 times previously from 6.1.9 and had issues so I had to revert back (I believe the issue was it wouldn't boot but I never tried to do a manual upgrade), plus this server has been offline since November of last year.  I was planning on trying to upgrade again once I got this straightened out, I just needed to figure out if I need to do an intermediary version first before upgrading to the current version and if so, which version would be safest.  I'm not sure if that's something I could do with the array being down.

 

I attached a couple of pictures I took, before and after the issues, the first picture was taken before I started trying to upgrade the drives in November when everything was fine.  Disk 1 is the only that completely failed and is not recognized at all even though it is in place, I still have the original Parity 5TB and Disk 4 4TB which I put the Disk 4 back in since the one I did the upgrade also failed.  The newer Parity 8TB seemed to upgrade okay so I put that in as you suggested.

 

My drive arrived that is being used for Disk 1 so I am ready to start on this, I also bought another 8TB that I will use as the new Parity if I am able to rebuild on the new Disk 1 so I still have my 2 original Parity drives as backup if I need to try over.

 

So do I need to proceed differently from the original instructions?  I'm not sure what invalidslot is as I have never had to do anything commandline in unRAID before other than a file system check.  It's always just run with minimal issues.

 

 

Edited October 4, 2022 by Kevin T
Images wrong

[trurl]

Why photos instead of screenshots? We may need to see more screenshots along the way as we proceed since what you are looking at is something we haven't seen in a few years.

[Kevin T]

That's all I had available, they were taken previously and they were for my own quick reference and I planned on deleting them afterwards, I had no idea I would have issues with it at the time.

 

Everything's shut off right now, if you want screenshots let me know what you want them of and I can turn it on , I just want to keep it to a minimum since I had 2 unexpected drive issues within a day's time and don't want another one.  I'm ready to do something on it now, I just need to know how to proceed.

 

I have the new drive installed to replace Disk 1 (note it is now 6TB but was originally 5TB), screenshot shows all the correct drives (Disk 4 shows as new since the new one failed so I had to revert to the original one).  I also included a screenshot under New Config, I'm not sure if what is says is normal.

Edited October 5, 2022 by Kevin T

[JorgeB]

The instructions should still work fine with v6.1.9.

[Kevin T]

So I don't need to worry about the invalidslot command that was mentioned?

 

What about the message on the New Config page in the screenshot above that says "Do Not Use this Utility thinking it will rebuild a failed drive", it is safe to check the box and proceed?

 

How does it know what disk to rebuild (Disk 1 according to unRAID)?

 

Edited October 5, 2022 by Kevin T

[JorgeB]

It will rebuild the disk you unassign in this part:

 

On 10/1/2022 at 11:43 AM, JorgeB said:

-Stop array
-Unassign the disk you want to emulate

 

[JorgeB]

The instructions should still work fine with v6.1.9.

Now that I think more about I'm not sure that the "parity is already valid" option worked with v6.1.x, I believe there was a bug with that, @Kevin Tmake sure you update first.

[Kevin T]

3 minutes ago, JorgeB said:

Now that I think more about I'm not sure that the "parity is already valid" option worked with v6.1.x, I believe there was a bug with that, @Kevin Tmake sure you update first.

 

 

Update?  To what?

 

If you mean a software update, I had issues in the past attempting to upgrade but I believe it was using the automatic method, so I will have to do the manual method.  I just need to know which version is recommended for now even if it's not the most current (can do that later).  Would it be okay with the array not working?

[JorgeB]

Update to v6.10.3 or v6.11.0, you can update manually, the array not working is not a problem.

[Kevin T]

On 10/1/2022 at 3:43 AM, JorgeB said:

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, including the new disk you want to emulate, replacement disk should be same size or larger than the old one
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign the disk you want to emulate
-Start array (in normal mode now), ideally the emulated disk will now mount and contents look correct, if it doesn't you should run a filesystem check on the emulated disk
-If the emulated disk mounts and contents look correct you can access it or even rebuild, to rebuild stop the array
-Re-assign the disk to rebuild and start array to begin.

 

I upgraded to 6.11.0 and followed the steps above, but after I unassigned Disk 1 and restarted the array normally, below is what I got.  I don't think this is right so I just stopped the array and not sure what to do.  When browsing it appears Disk 1 is missing.

I do have another previous Parity drive prior to me trying to upgrade drives before everything went screwy if it helps at all.  The current Parity is throwing a SMART error.

 

I also added new Diagnostics if needed.

 

unraid-diagnostics-20221005-1956.zip

Edited October 6, 2022 by Kevin T

[JorgeB]

No valid filesystem in being detected on the emulated disk1, that suggests parity is not valid, are you sing new or old parity?

[Kevin T]

I was using the new Parity, but it is the one after I started having issues. 

 

I was originally upgrading to larger drives, so I did the Parity first which was a 5TB to an 8TB, which was successful.  Then I did Disk 4 which the drive failed during the rebuild process and it won't recognize in BIOS (but I still have the original Disk 4 which I put back in and is fine).  While that one was failed and before I could do anything else on it, I just accessed Disk 1 to copy a small file locally and the whole server locked up.  Then Disk 1 wouldn't recognize either in BIOS.

 

So I'm not sure if those events affected the new Parity drive, which is the same one I tried (the 8TB).  I still have the original 5TB Parity untouched.  And again, both these Parity drives were within the same day so it's not like the old one was sitting around for months.

 

Edited October 6, 2022 by Kevin T

[JorgeB]

By your description parity should still be mostly valid, but at leas with the new one it's not, you can try repeating the procedure with the old parity.

[Kevin T]

Ugh, with the old Parity drive it's giving me a "Disk in parity is not biggest" message, I guess because my new replacement Disk 1 is 6TB even though I wasn't using all of it and I don't have any more 5TB drives that work.  It also says "If you are adding a new disk or replacing a disabled disk, try Parity-Swap", not sure what parity swap is.

 

I'm trying to do the Disk 1 assignment then unassignment and I have one unassigned external 4TB drive in the system but it has data on it (it says "All existing data on this device will be OVERWRITTEN when array is Started" next to the Parity drive when I select this drive for Disk 1), is there anyway to use that (or another drive) as a placeholder?  I wasn't sure in that process if the same or larger sizing restriction applies.

 

Or I do have a new 8TB I was going to use as the future Parity drive, is there a way I can clone the 5TB Parity to my new 8TB so I could use it instead so I don't have a size issue?

 

Or any other ideas?  This is more frustrating than I thought it would be.

 

Edited October 6, 2022 by Kevin T