parity sync errors on new parity drive

[Lassley7]

i decided too upgrade my parity disk too a larger one so i can have larger disks in the array, i ran a non correcting check and im getting read errors on disk 5. can someone look into the diag and tell me if they think a correcting pass should be done, or what i should do next? 

poseidon-diagnostics-20230830-1754.zip

Edited August 30, 2023 by Lassley7

[itimpi]

You do NOT want to run a correcting check with a disk reporting read errors as this is just likely to end up corrupting parity.   This is one of the reasons we recommend that scheduled checks are set non-correcting so you only run correcting checks when you do not have any hardware errors.  You might as well cancel the current check if it is still running.

 

The SMART information for disk5 looks fine, but the syslog shows the drive getting continual resets that look like connection issues prior to starting reporting read errors.  I suggest carefully reseating the cabling to the drive (both power and SATA) and then trying the check again (non-correcting).

[Lassley7]

1 hour ago, itimpi said:

You do NOT want to run a correcting check with a disk reporting read errors as this is just likely to end up corrupting parity.   This is one of the reasons we recommend that scheduled checks are set non-correcting so you only run correcting checks when you do not have any hardware errors.  You might as well cancel the current check if it is still running.

 

The SMART information for disk5 looks fine, but the syslog shows the drive getting continual resets that look like connection issues prior to starting reporting read errors.  I suggest carefully reseating the cabling to the drive (both power and SATA) and then trying the check again (non-correcting).

Reseated my breakout cables. Also cleaned some dust out of the machine. I don’t see anything obvious on the cables. Time too try a new non correction parity? Should I cancel the zeroing pass on the new drive I’m adding and prioritize parity? 

[itimpi]

42 minutes ago, Lassley7 said:

I don’t see anything obvious on the cables.

 

SATA connection are notoriously prone to disturbance.   Interestingly on my systems I get more reliable connections with non-locking SATA cables which seems counter-intuitive.

 

You would need another non-correcting check to see if this has resolved the issue.   Up to you which you treat as higher priority as the whole point of a zeroing task is that it leaves parity unchecked.

[Lassley7]

4 minutes ago, itimpi said:

 

SATA connection are notoriously prone to disturbance.   Interestingly on my systems I get more reliable connections with non-locking SATA cables which seems counter-intuitive.

 

You would need another non-correcting check to see if this has resolved the issue.   Up to you which you treat as higher priority as the whole point of a zeroing task is that it leaves parity unchecked.

So adding new disk won’t add to the party problem. That’s good to know yeah I’m not a fan of sas backplanes too breakout cables.. unfortunately that’s all I had to work with and up until now I’ve seen no issues. Are u suggesting this is NOT a bad drive and more 99% a bad connection? If party has errors again would you try a new slot or a new breakout cable 

[itimpi]

1 hour ago, Lassley7 said:

Are u suggesting this is NOT a bad drive and more 99% a bad connection? If party has errors again would you try a new slot or a new breakout cable 

It is nearly always the cabling (although very occasionally the drive).    Note it can also be the power cabling if a momentary disconnect is possible (particularly when using power splitters).

[Lassley7]

5 minutes ago, itimpi said:

It is nearly always the cabling (although very occasionally the drive).    Note it can also be the power cabling if a momentary disconnect is possible (particularly when using power splitters).

If the next parity sync is fine, would u then run a parity correction pass? Or would you run more than one non correction sync?

Edited August 31, 2023 by Lassley7

[itimpi]

Only run a correcting check if you have had errors reported a non-correcting check and you are reasonably confident there is no current hardware problem.  There is a good chance I would think that the non-correcting check may not report any errors the next time it is run.

[Lassley7]

2 minutes ago, itimpi said:

Only run a correcting check if you have had errors reported a non-correcting check and you are reasonably confident there is no current hardware problem.  There is a good chance I would think that the non-correcting check may not report any errors the next time it is run.

Correct so if the non correcting check has 0 errors I then need to run a correcting check? Or am I good with just the non correcting check with 0 errors

[itimpi]

Just now, Lassley7 said:

Correct so if the non correcting check has 0 errors I then need to run a correcting check? Or am I good with just the non correcting check with 0 errors

No correcting check required if 0 errors .

[Lassley7]

14 hours ago, itimpi said:

No correcting check required if 0 errors .

well we came back with errors again.. same disk.. going too try swapping too a different tray if that doesn't do it im assuming bad drive? heres diag just incase but appears too be the same errors 

poseidon-diagnostics-20230831-2034.zip

Edited September 1, 2023 by Lassley7

[JorgeB]

SMART doesn't look good for that disk, you can also run an extended SMART test to try and confirm if it's really bad.

[Lassley7]

13 minutes ago, JorgeB said:

SMART doesn't look good for that disk, you can also run an extended SMART test to try and confirm if it's really bad.

Well I swapped it with a different tray location and started sync again just woke up to that same disk showing errors. Sounds to me like it’s time too replace that disk all 5 syncs I’ve tried have all been errors on that single disk..

 

am I safe too go ahead and take that drive out and assign a new blank one in it’s place and rebuild? Do I do a non correction sync?

Edited September 1, 2023 by Lassley7

[JorgeB]

Assuming parity is valid it should be fine to just replace and rebuild.

[Lassley7]

2 minutes ago, JorgeB said:

Assuming parity is valid it should be fine to just replace and rebuild.

Well when running a non correcting sync is when the errors happen so how does one assume it’s good?

[itimpi]

22 minutes ago, Lassley7 said:

Well when running a non correcting sync is when the errors happen so how does one assume it’s good?

Chances are good that if you were using non-correcting checks then parity is likely to be valid.   

 

Only way at this point I can think of to check is to:

• Stop array
• Unassign problem disk
• Restart array
• The missing drive should now be emulated.   See if the emulated disk mounts and has the expected contents.  

 

If it does then you are good to go with a replacement disk.  If it does not then you might have to run a check filesystem on the emulated drive. Keep the removed drive intact until you are back up and running if possible to keep some other recovery options open.

[Lassley7]

14 minutes ago, itimpi said:

Chances are good that if you were using non-correcting checks then parity is likely to be valid.   

 

Only way at this point I can think of to check is to:

• Stop array
• Unassign problem disk
• Restart array
• The missing drive should now be emulated.   See if the emulated disk mounts and has the expected contents.  

 

If it does then you are good to go with a replacement disk.  If it does not then you might have to run a check filesystem on the emulated drive. Keep the removed drive intact until you are back up and running if possible to keep some other recovery options open.

How can I replace that failed disk with an empty one that’s already part of the array. I just added it there’s no data on it but it is in fact part of the array now 

 

assuming I have to do a new config at this point? Since I stupidly added my only new drive already

Edited September 1, 2023 by Lassley7

[itimpi]

15 minutes ago, Lassley7 said:

How can I replace that failed disk with an empty one that’s already part of the array. I just added it there’s no data on it but it is in fact part of the array now

 

Have you formatted the new drive yet  (a formatted drive is not 'empty' as far as parity is concerned)?    If so you will not be able to use it to replace the failed drive without data loss.

 

Regardless the steps I outlined will at least tell you if parity is valid so I would try those anyway before doing anything more drastic.

[Lassley7]

Just now, itimpi said:

 

Have you formatted the new drive yet  (a formatted drive is not 'empty' as far as parity is concerned)?    If so you will not be able to use it to replace the failed drive without data loss.

 

Regardless the steps I outlined will at least tell you if parity is valid so I would try those anyway before doing anything more drastic.

the drive has sadly been formatted.. what i mean by empty is theres no important data on it as of now so how can one replace the failed one with one that is already added to the array... the failed drive did populate as emulated and from what i could tell all data was there.. the problem is i dont have the funds for another disk atm i just added this new one im reffering too, can a new config accomplish this?

[itimpi]

5 minutes ago, Lassley7 said:

the drive has sadly been formatted.. what i mean by empty is theres no important data on it as of now so how can one replace the failed one with one that is already added to the array... the failed drive did populate as emulated and from what i could tell all data was there.. the problem is i dont have the funds for another disk atm i just added this new one im reffering too, can a new config accomplish this?

Cannot use New Config tool at this point in a simple way without losing data on the failed/emulated drive unfortunately.

 

If the emulated (failed) drive is mounting and its data appears intact then you could copy its data across to the (currently empty) new drive.   I suggest copy rather than move as it puts less strain on the system. When that completes you could then use the New Config tool to remove the failed drive; reorganise the order of the remaining drives if so desired; and build new parity based on the remaining good drives

[Lassley7]

3 minutes ago, itimpi said:

Cannot use New Config tool at this point in a simple way without losing data on the failed/emulated drive unfortunately.

 

If the emulated (failed) drive is mounting and its data appears intact then you could copy its data across to the (currently empty) new drive.   I suggest copy rather than move as it puts less strain on the system. When that completes you could then use the New Config tool to remove the failed drive; reorganise the order of the remaining drives if so desired; and build new parity based on the remaining good drives

what if one already decided too remove the config for the new drive and now its showing as a "new device" and start is saying 

`Start will start Parity-Sync and/or Data-Rebuild.`

[itimpi]

If you continue then the contents of the failed/emulated drive will be lost (so only start the array if you are happy to lose its contents).

 

I would suggest that at this point you post your system's diagnostics zip file before doing anything else so we can see exactly what state the array is currently in.   That will allow us to give you the best way to proceed without data loss.   It should be possible but the steps will have to be carefully done as you will be bypassing Unraid's normal recovery process.

[Lassley7]

Just now, itimpi said:

If you continue then the contents of the failed/emulated drive will be lost (so only start the array if you are happy to lose its contents).

 

I would suggest that at this point you post your system's diagnostics zip file before doing anything else so we can see exactly what state the array is currently in.   That will allow us to give you the best way to proceed without data loss.   It should be possible but the steps will have to be carefully done as you will be bypassing Unraid's normal recovery process.

 

poseidon-diagnostics-20230901-0543.zip

[JorgeB]

You can try this assuming parity is still valid:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments are correct including the old disk5 and new disk11, they should all be assigned to their original positions
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk5
-Start array (in normal mode now) and post new diags.

[Lassley7]

33 minutes ago, JorgeB said:

You can try this assuming parity is still valid:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments are correct including the old disk5 and new disk11, they should all be assigned to their original positions
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk5
-Start array (in normal mode now) and post new diags.

I followed The "Remove Drives Then Rebuild Parity" Method as a bunch suggested on discord… they said to then mount the disk with issues using UD and copy it over… I hope this wasn’t a mistake