So I am in the process of upgrading the remaining 2TB drives in my backup server and whilst the parity is syncing I have a failing drive, great. So far only 5 pending sectors and about 2 hours left until my parity sync is done and I can swap out the failing drive, will I lose data on the failing drive? This is a backup server so even if I lose data, I'm not really losing it as its not my primary data source. At least the failing 3TB WD RED is under warranty.

I think you are going to have to provide a more detailed description of exactly what you were(are) doing when this event occurred.  (At least, I am confused...) 

 

It would probably be a good idea to attach a diagnostics file with your next post.  And a listing of what drives are currently in your system and which ones are being replaced. 

So I have 23 data disks in my backup server, yesterday I swapped out a 2TB drive and replaced it with a 3TB drive, so the server is doing a parity sync to replace and upsize the drive. Last night I get notifications that another 3TB drive in my server has read errors, when I checked it this morning it has 5 pending sectors and many read errors.  I am just under an hour away from the parity sync finishing and when that is done, will swap out the 3TB drive that has errors. Attached is a diagnostic file.

tower-diagnostics-20160403-1005.zip

So I have 23 data disks in my backup server, yesterday I swapped out a 2TB drive and replaced it with a 3TB drive, so the server is doing a parity sync to replace and upsize the drive.

I assume you mean a rebuild to replace a data drive/  A parity sync refers to building the contents of the parity drive from the data drives.

Last night I get notifications that another 3TB drive in my server has read errors, when I checked it this morning it has 5 pending sectors and many read errors.  I am just under an hour away from the parity sync finishing and when that is done, will swap out the 3TB drive that has errors. Attached is a diagnostic file.

If  you start getting read errors while rebuilding a different drive it is highly likely that you will have corruption on the rebuilt drive (as the rebuild relies on both parity and all other data drives reading perfectly).    Do you still have the disk you replaced if you need to get data off it (or a backup of its contents).

 

I would not rush into swapping out the 3TB drive until we make sure we know the status of the disk that was being rebuilt as you do not want to compound any chances of data loss.

No I no longer have the 2TB drive that was removed and replaced by a 3TB drive. Honestly if there is any data loss its not that big a deal for me, but I would appreciate any assistance that can determine if there is data loss/corruption. The rebuild is almost done, what can I do next to determine data loss/corruption?

No I no longer have the 2TB drive that was removed and replaced by a 3TB drive. Honestly if there is any data loss its not that big a deal for me, but I would appreciate any assistance that can determine if there is data loss/corruption. The rebuild is almost done, what can I do next to determine data loss/corruption?

About the only thing you can do is put the array into Maintenance mode, click on the disk and then run a file system check to see if there is any corruption at that level.

 

For corruption of a files contents you need to either have backups to compare them against, or file checksums (here the File Integrity plugin would have been useful for both generating and checking checksums) to check against.

 

For future reference when replacing a disk it is always a good idea to keep the original available and unchanged until you know the replacement has completed without any issues arising.

The problem is now that the rebuild is at 98.8% and has slowed to a crawl of about 1MB/s which makes me wonder if its trying to read from the failing drive. Also I am getting email alerts from my Areca raid controller but about channel 14, but I didn't think this failing drive was connected to the Areca, have to look into that. Also the Unraid GUI has become unresponsive now. Is their a command I can run from the CLI that will give me the progress of the rebuild?

Not sure what to do here, I am not getting any response from the gui, I am logged in at the command prompt, should I issue a shutdown command and shut the server down? Is there a command that will tell me the status of the rebuild?

Ok so now the gui is responding, the rebuild finished and I got email alerts: So now that I have two disks failing and a rebuild that completed with 800 errors, I am facing data loss no matter what. Good thing this is a backup server.

 

Event: unRAID Data rebuild:

Subject: Notice [TOWER] - Data rebuild: finished (799 errors)

Description: Duration: unavailable (no parity-check entries logged)

Importance: warning

 

Event: unRAID Disk 12 message

Subject: Notice [TOWER] - Disk 12 returned to normal operation

Description: 2001b4d23028238c0 (sdn)

Importance: normal

 

 

Event: unRAID array errors

Subject: Warning [TOWER] - array has errors

Description: Array has 2 disks with read errors

Importance: warning

 

Disk 13 - 2001b4d23028238d0 (sdo) (errors 128)

Disk 19 - WDC_WD30EZRX-00DC0B0_WD-WCC1T1079428 (sdu) (errors 671)

I stopped the array, put it in maintenance mode, clicked on each of the affected disks and ran the xfs repair utility which did its thing but since I've never used it I am not sure what its supposed to do. I can't run a SMART test on one of the drives as its connected to my Areca (can't get into the Areca's web interface either) but I ran a SMART test on the other drive and it came back ok, still showing 5 pending sectors though.  I remounted the array, its not showing any errors now, should I run a parity check or replace the drive that is showing 5 pending sectors first?

I am posting recent diags that I just ran, maybe there is something in there that can shed some light on what is going on.

 

Thanks

tower-diagnostics-20160403-1156.zip

I stopped the array, put it in maintenance mode, clicked on each of the affected disks and ran the xfs repair utility which did its thing but since I've never used it I am not sure what its supposed to do.

It normally tells you if it is repairing anyth9ing.    You might also try looking for a lost+found folder on the disk which would indicate folders/files for which the directory entry pointing to them was lost.

I can't run a SMART test on one of the drives as its connected to my Areca (can't get into the Areca's web interface either) but I ran a SMART test on the other drive and it came back ok, still showing 5 pending sectors though.  I remounted the array, its not showing any errors now, should I run a parity check or replace the drive that is showing 5 pending sectors first?

I would say that at this point your best bet is to copy as much data as you can off the drive with pending sectors to a known good location.  With 5 pending sectors there is a chance of a small amount of corruption in some files, but it should be minimal (if any).

 

It is then time to look at the drive that may have rebuilt badly to work out the state of the data on that drive.  Others may have views on the best way to proceed with that drive.

 

EDIT:  Looking at the syslog there are lots of read errors on disk19.  Is that the one that has the pending sectors?    The number of errors suggest that the disk may have dropped offline at some point during the rebuild.

I am running a compare on my media folders right now to see if there is any discrepancy, beyond that I am not too concerned about data loss because its of really no consequence. So far I am not seeing any discrepancies that would suggest I have lost any media. I do realize that this method tells me nothing about file integrity and I could still have corruption in some of my media files. Should I proceed by replacing the drive with 5 pending sectors as my next move?

I am running a compare on my media folders right now to see if there is any discrepancy, beyond that I am not too concerned about data loss because its of really no consequence. So far I am not seeing any discrepancies that would suggest I have lost any media. I do realize that this method tells me nothing about file integrity and I could still have corruption in some of my media files. Should I proceed by replacing the drive with 5 pending sectors as my next move?

I am not sure.  Have you tried looking at the contents of the disk that was rebuilt yet?  I was wondering if anyone thinks it is worth retrying the rebuild to see if it goes better the second time around.

 

I guess just because some sectors may not have been recovered during the rebuild does not necessarily mean corruption - they could have been sectors that were empty of data.

How would I do the rebuild again?

 

I am going through some of the media files on the rebuilt drive and I have found one file so far that was corrupted, might have to do a manual check of the files, that will take awhile 

I take that back, the one file I found so far that I thought was corrupted, turns out the source file is also corrupted, so it wasn't corrupted after all. I'll have to do some more testing to see if I can find any corrupted files.

How would I do the rebuild again?

 

I am going through some of the media files on the rebuilt drive and I have found one file so far that was corrupted, might have to do a manual check of the files, that will take awhile 

To force a rebuild onto an existing drive

• stop the array and unassigned the drive
  
• start the array.  It will start with the drive 'missing'.  This step causes unRAID to 'forget' the drive serial number
  
• stop the array and reassign the drive
  
• Start the array and it will start the rebuild.  Since in theory it is writing exactly the same data as was there before it should not change anything.  However since the previous build had issues and may not have rebuilt all sectors correctly it may end up correcting some 'bad' sectors.
  

So I forced a rebuild and its around 47% now, but its crawling at under 10MB/s now, I think it reading from the drive I have 5 pending sectors on. I have attached diags, not sure what is going on.

 

tower-diagnostics-20160403-2234.zip

Since you have a drive that's experiencing significant read errors, it's not likely the 2nd rebuild will be any better than the first.    I'd be inclined to do the following:

 

(1)  Do a comparison of all of the files on the rebuilt disk with your primary server, so you'll know which files are different .. and replace any of those on the backup disk with the originals.

 

(2)  Save a complete directory of the files on the other failing disk (with the pending sector);  replaced that disk and do a rebuild onto the new disk; and then do a comparison of those files with the source.

 

These will take a long time -- as I'm not just talking about comparing the directories, but the actual binary contents.

 

Going forward, I would compute checksums for all of the files on your primary server; and after the checksum files have propogated to your backup server, run a checksum validation on the backups, to see if any of those files don't match the checksums.    Once you know you have good checksums both places, it'll be very easy in the future to identify any files that have become corrupted.

 

Thanks Gary, can you recommend any tools I can use to do the comparison?

 

I do all my maintenance from a Windows box.

 

For creating and checking checksums I use Corz Checksum.  [ http://corz.org/windows/software/checksum/ ]

 

For comparisons I use FolderMatch.  [ http://www.foldermatch.com/ ]

 

... and for completeness, I copy everything to the server using TeraCopy with the verification option.  [ http://www.codesector.com/teracopy ]  and  do all my backups using SyncBack  [ http://www.2brightsparks.com/freeware/index.html ]    (although I use SyncBack SE, the current freeware version is fine)

 

There are, of course, Linux command-line options that will do these as well, but I prefer to do it all from Windows.

 

 

 

 

I just got a weekly summary from my server doing the data rebuild (again) and it looks like it was able to recover those sync errors.

 

 

Data rebuild in progress.

Total size: 3 TB

Elapsed time: 9 hours, 45 minutes

Current position: 1.34 TB (44.7 %)

Estimated speed: 9.5 MB/sec

Estimated finish: 2 days, 19 minutes

Sync errors corrected: 799

 

 

Sync errors corrected: 799

 

This doesn't make sense, sync errors can't be corrected during a rebuild.

 

 

Then why is it saying it fixed them?

You're earlier rebuild indicated 799 errors.  I suspect parity was updated as a result of those, and the summary is simply reflecting that "correction".    Remember, however, that although the most likely fix for a parity error is to update the parity drive, that is not always the actual location of the error -- so this could instead mean you now have some corrupted data that's "protected" by parity.

 

Wtihout checksums or a "known good" set of data to compare against, there's really no way to be absolutely certain of the integrity of your files.