RAID failure on rebuild

[muntedewok]

Hi all,

 

So I'm running Ver 6.6.1 with a 3TB WD Red NAS Parity, 3x 3TB WD NAS/Seagate drives and a 2TB WD drive and a 500GB WD drive for cache (will be swapping out for an SSD soon). Yesterday noticed that Disk 2 had spat the dummy, was no longer "mountable" so I eventually managed to perform a check on it, disk was OK so only way to get it back was to rebuild it. No worries. However at the same time I noticed that Drive 4 was having SMART issues as well. The rebuild on Drive 2 appeared to work OK and then drive 4 drive appeared to fail. Further investigation on this found that the SATA connector on the SAS breakout cable from my RAID controller appears to be faulty and wasn't reading correctly and also became unmountable, during the Drive 2 rebuild the drive 4 drive racked up 700,000,000 errors. I am not kidding.

 

I ended up unplugging Drive 4 from the SAS breakout and plugged the SATA connector directly into the motherboard, no worries. However since then I've rebuild the drive and although it says it completed successfully, it has 33.6mb data on it out of 3TB. Checked drive folder view nothing on it of course. Also noticed that my shares, even though they are mountable, are empty with exception to 1 which is complaining about Some or All Files Unprotected.

 

If I browse the disks individually, all the files are there, no worries and if I check the Included Disk field for the shares it's set to All as required.

 

I've disconnected the apparently faulty disk 4 with the SMART errors and plugged in a brand new 3TB WD Red NAS drive and it's supposedly rebuilding, however it's also just sitting at 33.6mb.

 

Is it possible to recover the data on the 3x drives which are showing data when I manually browse them and recreate the shares or something? I imagine it'd be a case of recreating the RAID with the existing working disks then adding the new 3TB drive into the array as well, cutting my losses on the data that's gone missing?

 

I've obviously resigned myself to the fact I've lost a disk worth of data so I'm just in damage control/salvage mode at the moment. Good thing is it's just TV shows and movies so if worse comes to worst, I can rebuild it over time, just annoying when it's 11TB of data

 

Thanks fellas,

 

-Chris

[JorgeB]

17 minutes ago, muntedewok said:

during the Drive 2 rebuild the drive 4 drive racked up 700,000,000 errors

This means the rebuitl disk will be severely (or completely) corrupt, so this is expected:

18 minutes ago, muntedewok said:

although it says it completed successfully, it has 33.6mb data on it out of 3TB.

Though this also makes think you formatted the disk.

 

As for the rest, please post the diagnostics: Tools -> Diagnostics.

[muntedewok]

Hey Johnnie,

 

The new 3TB WD I installed was fresh out of the box, hadn't been spun up in any other machine.

 

I have uploaded the diagnostics as requested.

 

Cheers,

 

-Chris

tower-diagnostics-20190520-2134.zip

[JorgeB]

Check filesystem on disk2:

https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui

[muntedewok]

Did you want me to post the outcome of the check or run a specific command against the drive as well?

 

-Chris

[JorgeB]

No need, unless you need more help.

 

You'll also need to finish rebuilding disk4.

[muntedewok]

5 found corruptions can be fixed only when running with --rebuild-tree

 

Think it might be told to hold onto my balls!

 

 

Edited May 20, 2019 by muntedewok

[JorgeB]

It's usually fine, though there can always be some data loss.

[JorgeB]

7 minutes ago, johnnie.black said:

there can always be some data loss. 

Though in this case there will be for sure, because of the corrupt rebuild.

[muntedewok]

Probably explains what all this is then:

 

init_source_bitmap: Bitmap 14882 (of 32768 bits) is wrong - mark all blocks [487653376 - 487686144] as used init_source_bitmap: Bitmap 14883 (of 32768 bits) is wrong - mark all blocks [487686144 - 487718912] as used

 

Looks like it's going to take a while...

[JorgeB]

--rebuild-tree always takes a few hours, around 2 or 3 hours per TB.

[muntedewok]

No worries thanks for the heads up.

 

I'll check on it in the morning and kick off the rebuild of Disk 4 as well, will take a good 6-7 hours or so judging by the last effort.

 

Thanks for the assistance and hope it goes well!

 

-Chris

[muntedewok]

Hey Johhnie,

 

Disk 2 has rebuilt OK and as predicted lost a heap of data, not too fussed about that easily recoverable from elsewhere, the shares have come back up correctly and data visible in the network mapped volumes now, all good!

 

However, I've replaced the stuffed Disk 4 and it's complaining that the new disk is unmountable. I removed it and plugged it into my PC, used diskpart to clean the drive of all partitions etc and re-inserted it into the server. However when I start the volume again it's complaining the drive is unmountable even when I force a format on it.

 

I've uploaded the latest log files if required.

 

Kind regards,

 

-Chris

tower-diagnostics-20190521-0840.zip

[JorgeB]

That's strange, I see the format failing but not why it's failing, try formatting with xfs, since reiser it's no longer recommended anyway, and to see if there is more info.

[muntedewok]

Is there a way to format it while it's in the server still? I've only got remote access to my server at the moment from my office

 

-Chris

[JorgeB]

With the array stopped click on disk4 and change the filesystem to xfs, then start array and try to format again, if it still fails post new diags.

[muntedewok]

New logs uploaded, unfortunately no dice on the format!

 

-Chris

tower-diagnostics-20190521-1506.zip

[muntedewok]

Just downloaded the unassigned drives plugin, going to run a pre-clear on the drive and see what it does afterwards.

 

-Chris

[JorgeB]

Preclear won't help, it might be a bug partitioning a disabled disk, though don't remember seeing it before, wait for the rebuild to finish them format.

[muntedewok]

Managed to get it to mount this way. Started RAID with drive 4 unassigned, and performed format on it. Then dismounted RAID, then selected drive and mounted again. Now it says it's rebuilding RAID however it's back at the point where it's only showing 33.6mb on the drive.

 

New logs uploaded.

 

Thanks,

 

-Chris

tower-diagnostics-20190521-1539.zip

[JorgeB]

18 minutes ago, muntedewok said:

it's back at the point where it's only showing 33.6mb on the drive.

That's expected, it was already like that before, and it woulds always be after a format.

[muntedewok]

The interesting thing being that even though the write count is going up with the rebuild (now at 12% and 857,688 writes) the data count isn't increasing. I'm wondering if the parity that was available for the drive previously is hosed...

 

-Chris

[JorgeB]

The first diags you posted already had an empty disk4:

Filesystem      Size  Used Avail Use% Mounted on
/dev/md1        1.9T  1.7T  197G  90% /mnt/disk1
/dev/md2        2.8T  1.7T  1.1T  60% /mnt/disk2
/dev/md3        2.8T  2.5T  256G  91% /mnt/disk3
/dev/md4        2.8T   33M  2.8T   1% /mnt/disk4

No amount of rebuilding will change that, whatever the problem was it happened before that, likely you formatted an unmountable disk, that deletes all data on that disk and updates parity accordingly.

[muntedewok]

Fair call as well, hadn't factored in the fact Disk 2 had a corrupted rebuild.

 

Disk 4 has rebuilt into the RAID array OK, lost about 4TB worth of data however it was only movies and I have a log of all that was in there anyway so no big deal.

 

Shares are showing correctly as being protected with exception to the appdata and Downloads folder, which if I view the folder structure, are showing different disks being used (single or multiple disks) for each folder. I'll probably just rip these out because they're small in size, rebuild the share then move the files/folders back in.

 

Thank you very much for assistance on this Johnnie, greatly appreciated.

 

-Chris

[itimpi]

3 hours ago, muntedewok said:

Fair call as well, hadn't factored in the fact Disk 2 had a corrupted rebuild.

 

Disk 4 has rebuilt into the RAID array OK, lost about 4TB worth of data however it was only movies and I have a log of all that was in there anyway so no big deal.

 

Shares are showing correctly as being protected with exception to the appdata and Downloads folder, which if I view the folder structure, are showing different disks being used (single or multiple disks) for each folder. I'll probably just rip these out because they're small in size, rebuild the share then move the files/folders back in.

 

Thank you very much for assistance on this Johnnie, greatly appreciated.

 

-Chris

The appdata User Share is normally expected to all be on the cache drive (which is why it would show as unprotected as you only have a single drive in cache).  What setting do you have for “Use Cache” on the appdata Share?   Also what is that setting set to for the ‘Downloads’ folder?    Not sure what you mean when you say ‘rebuild’ in the context of these shares❔