Failed Parity and Data disk at the same time.

[Gsusking2]

Hello,

I checked on my server today and had a red x on a data disk and on a Parity drive.

They both showed the same error count, so i assume some CRC power error threw them both out of wack.

My question is how do I proceed, Do i replace the data disk and start rebuild? 

do i repair the disk (XFS) and then replace the parity?

Is it possible to just reseat the 2 disks and see if they throw errors?

I had to reboot before i could get the diags,

here are the diags after a restart.

tower-diagnostics-20240617-1617.zip

[JorgeB]

Diags are after rebooting so we can't see what happened.

 

There a device (sdc) causing constant errors, see if you can disconnect that, then post new diags after array start.

[Gsusking2]

Thank you for the reply Jorge,

when i remove the disk (sdc), i am unable to start my array.

I have posted the Diag after boot with removed (sdc).

Let me know if you need any other info.

 

tower-diagnostics-20240618-0953.zip

[JorgeB]

sdc wasn't assigned to the array in the previous diags, but those IDs can change, I'm not seeing any ATA errors so far, so reconnect disk1 and post new diags after array start.

[Gsusking2]

Hi Jorge,

After starting the array i now have another disk that threw an error and is showing a non supported file system 
Please find attached diags,

tower-diagnostics-20240618-1102.zip

[JorgeB]

Disk6 appears to be failing, run an extended SMART test, don't write anything to the array for now, in case you need to force enable a disable disk to rebuild disk6.

[Gsusking2]

Thank you, will update after the extended test.

 

[Gsusking2]

please see result of extended test of disk 6

tower-smart-20240618-1137.zip

[itimpi]

That result shows that the disk is starting to fail as it did not complete that test error free.

[JorgeB]

You can try force-enabling the other disks, since they look healthy, to rebuild disk6, but before that, disk22 is unmountable, looks like it was never formatted, is that correct?

[Gsusking2]

17 minutes ago, JorgeB said:

You can try force-enabling the other disks, since they look healthy, to rebuild disk6, but before that, disk22 is unmountable, looks like it was never formatted, is that correct?

Ok, lets give it a shot.  If you could give me a quick guide on what to do i have done stuff like this before with my server (see post history) and im very capable of following instructions.

 

Background:

disk 22 was being added to the array and had just finished its preclear.  When i went to format it via the GUI, it was hanging and not responding,  i then initialized a restart of the server. as i clicked the button i noticed the red x on drive 2 and parity. hence why i dont have the original diags

 

[JorgeB]

OK, so try this:

 

-Tools -> New Config -> Retain current configuration: All -> Apply
-Check all assignments and assign any missing disk(s) if needed, for now leave old disk6 assigned to its slot, as a place holder
-IMPORTANT - Check both "parity is already valid" and "maintenance mode" and start the array (note that the GUI will still show that data on parity disk(s) will be overwritten, this is normal as it doesn't account for the checkbox, but it won't be as long as it's checked)
-Stop array
-Unassign disk6
-Start array (in normal mode now) and post new diags.

 

[Gsusking2]

Hello,

I followed the steps above and this is the diag output

tower-diagnostics-20240618-1326.zip

[JorgeB]

The disabled disk is mounting, check contents but they should be OK, for the emulated disk6, stop the array, click on disk6, set the filesystem to xfs, then check filesystem on that disk, run it without -n.

 

Keep old disk6 intact for now.

[Gsusking2]

Thank you,

I have done above here is the output from  the file system check

Check File System Output - Disk 6.rar

[JorgeB]

File is empty, but assuming xfs_repair finished, re-start the array in normal mode, with disk6 still unassigned, and see if the emulated disk6 now mounts.

[Gsusking2]

My mistake here is the output.

I will continue with the steps described. and post again.

Check File System Output - Disk 6.rar

[Gsusking2]

49 minutes ago, JorgeB said:

File is empty, but assuming xfs_repair finished, re-start the array in normal mode, with disk6 still unassigned, and see if the emulated disk6 now mounts.

Disk 6 is still showing a red x when i start the array after the xfs file system check.

 

Edit * attached diagnostics 

tower-diagnostics-20240618-1500.zip

Edited June 18 by Gsusking2

[JorgeB]

12 hours ago, Gsusking2 said:

Disk 6 is still showing a red x when i start the array after the xfs file system check.

That's normal, but the emulated disk is mounting now, look for a lost+found folder, if all look good get a new disk and rebuild.

[Gsusking2]

Thank you Jorge!

Can i use the 10tb that is in the array? am i able to remove it from slot 22 and make it new disk 6?

Let me know if i have to reset the config or something to make it happen.

[JorgeB]

Although not impossible, I would not recommend that, if you don't want to use a new disk, format disk22, copy everything from the emulated disk there, then you can do a new config without disk6, it won't be the fastest way, but less error prone.

[Gsusking2]

Thanks,

I'm just gonna buy a new disk and hope for the best and add 22 to the array after.

Thank you again Jorge,  If you're ever in Canada let me know ill hook you up.

in the mean time ill buy you a beer through your link.

You are always so helpful, patient and kind in these situations.  You're a true gem