I have an error with (at least one) of the drives in my Unraid server (disk 5 gets read errors).

 

I looked at the syslog and found the following (see below) that is proabably the cause. I need some input on how to fix the problems.

 

Will getting a new drive solve everything or is there also an error with the filesystem? If so, how can I scan all the drives and fix possible errors? Is there a way to keep disk 5 in the array for a while? Can I scan all drives and "close off" all bad sectors?

 

Hope someone can help with this.

 

Edit: Log removed.

Edited September 4, 20178 yr by kno

Post your diagnostics: tools -> diagnostics

Sorry, I thought the syslog above might be enough as it shows the error message.

 

Here is the whole diagnostics.

 

Also, SMART report fra the drive that I think might be causing the problems, although it might be something else.

 

Edit: Diagnostics removed.

 

 

Edited September 4, 20178 yr by kno

That drive is toast. You should rebuild it to another and then you will have to repair the filesystem.

 

Unfortunately, you have another drive that's pretty bad too, and another with several pending, so a rebuild may not succeed.

 

Do you have Notifications configured? unRAID should have been Notifying you about your bad disks, and if you look at the Dashboard it is probably saying you have multiple disks with problems.

 

You absolutely must have Notifications setup so you don't allow yourself to get into a situation like this. unRAID can only rebuild a disk if all other disks are healthy.

 

Do you have backups?

28 minutes ago, trurl said:

That drive is toast. You should rebuild it to another and then you will have to repair the filesystem.

 

Unfortunately, you have another drive that's pretty bad too, and another with several pending, so a rebuild may not succeed.

 

Do you have Notifications configured? unRAID should have been Notifying you about your bad disks, and if you look at the Dashboard it is probably saying you have multiple disks with problems.

 

You absolutely must have Notifications setup so you don't allow yourself to get into a situation like this. unRAID can only rebuild a disk if all other disks are healthy.

 

Do you have backups?

 

Ok, thanks for the quick feedback.

 

• "That drive is toast. You should rebuild it to another and then you will have to repair the filesystem."

 

So that I can learn something from this: what did you look for to land on that conclusion? Is it in the SMART report or somewhere else? How do I go about to repair the filesystem?

 

• "Unfortunately, you have another drive that's pretty bad too, and another with several pending, so a rebuild may not succeed."

 

I guess I will have to try to rebuild? What are the other drives that are bad, all the 2 TB HDDs? What did you look for? How do you consider if they are failing or if they are just reporting minor issues? I exchanged one of the 2TB drives about 2 months ago, because the start-up time of the disc was very long (like 45 seconds).

 

• "Do you have Notifications configured? unRAID should have been Notifying you about your bad disks, and if you look at the Dashboard it is probably saying you have multiple disks with problems."

 

No, I did not have the notifications configured properly unfortunately. I have noticed that there were some orange markers on some of the disc. I have looked at the SMART reports but found the results difficult to interpret. One bad sector does not have to indicate a failing hdd from what I understand, so I have not taken all the SMART info to seriously. Also, the drives have little usage (about 3000-3500 hours), so they should not all be failing at the same time.

 

• "Do you have backups?"

 

For some data I have backups, but I am not sure exactly what is on the bad drives. However it is mostly non-critical media files. For most of it I have the original disc, but there might be some other data as well amongst it that I would rather not loose.

 

So, what is the best way forward?

1. Should I try to copy the data from the bad drives to someplace else before I do anything else?
2. Should I replace the failing drive at once and hope for a successful rebuild first? If successful, what do I do next? Replace the second drive that is doing bad and hope for a good rebuild or do I try to copy off the data first to someplace else?

 

Lastly, is it safe to boot up the server an look around to se what data is on the drives?

For future reference, your diagnostics zip contains the complete system log(s), SMART for all disks, and many other useful things. So, no need to post any of that separately, and a syslog snippet as you originally offered just takes up a lot of screen and isn't usually very helpful.

 

I would stop all writing to the server, including from any dockers/plugins/VMs you may have. If you have any important, irreplaceable files on the server, copy them somewhere else.

 

disk5:

Serial Number:    WD-WCAZA3073246
  5 Reallocated_Sector_Ct   0x0033   159   159   140    Pre-fail  Always       -       782
196 Reallocated_Event_Count 0x0032   001   001   000    Old_age   Always       -       725
197 Current_Pending_Sector  0x0032   199   199   000    Old_age   Always       -       476

disk4:

Serial Number:    WD-WCAZA3074765
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       1
196 Reallocated_Event_Count 0x0032   199   199   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0032   198   198   000    Old_age   Always       -       886

disk1:

Serial Number:    WD-WMAZA5009597
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       21
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       50

It is mostly the pending sectors that have me worried about rebuilds.

 

And it is actually disk6 with the read-only filesystem. That disk itself looks OK. Disk5 is giving read errors.

 

Since none of the disks are actually disabled yet, it probably makes sense to try to repair the filesystem on disk6 before trying to replace any of these disks. Repairing the filesystem should not even access the troublesome disks (assuming you aren't using Turbo Write).

 

Let's wait and see if there are any other opinions.

 

 

Repair filesystem on disk6 and try to copy any irreplaceable data to another PC.

 

Disk5 we know it's bad for sure, 1 and 4 look bad but sometimes pending sectors SMART info on WD disks is not correct, run and extended SMART on both.

Here are the extended SMART reports for disk 1 and 4. Both have status PASSED, however it looks like disk 1 has read errors. To me that looks like a failure. What do you make of the SMART reports for the drives?

 

I looked at the data as well. Almost all of it is uncritical media files.

 

Edit: SMART reports removed.

 

 

Edited September 4, 20178 yr by kno

Disk1 failed the extended test, disk4 passed, so it's good for now, the huge number of pending sectors are incorrect, it's rather common with WDs, still with 2 disks with problems you won't be able to do a successful rebuild, so if there are no critical files on both disks 1 and 5 you can do a new config with 2 new disks and resync parity, you can still try to copy anything you can from both.

Ok, I'll order some new HDD's and get started when they arrive.

 

Should I repair the filesystem on other HDDs (such as disc 6) before I change out the HDD's or after? Does one affect the other?

6 minutes ago, kno said:

Should I repair the filesystem on other HDDs (such as disc 6) before I change out the HDD's or after? Does one affect the other?

 

As you prefer, one doesn't affect the other.

Ok. Will do. It will be a couple of days to get it done.

 

I will keep the thread open. I might have some more questions underway.

Update:

I tried to copy all content from disks 1, 4 and 5. I managed to copy everything from disks 1 and 4, but there were some read errors on disk 1 but the copy did not stop. There were no errors on disk 4.

 

Copying data from disk 5 proved difficult, so I stopped trying and replaced the drive with a new one.

 

Today parity sync was completed. Disk 1 reported 2880 errors (of 2.543.421 reads). How do I know if the sync was successful? Would it have stopped if not?

 

I now plan to replace disk 1 and run a new sync.

 

Edit: Diagnostics removed.

 

Edited September 4, 20178 yr by kno

unRAID won't stop the rebuild when there are read errors on one disk while rebuilding another, so at least some or most data is recovered, but the rebuilt disk will have some (or a lot) of corrupt files, if you have checksums you can just check which ones are corrupt, if not, just by testing them, and in most cases it can be difficult, e.g., a movie can playback and just skip a couple of seconds in the middle due to corruption.

Should I start the next rebuild (of disk 1) first or should I check for corrupt files first? Does one affect the other?

3 minutes ago, kno said:

Does one affect the other?

 

No, but the rebuilt disk1 will also have some corrupt files, since the corrupt now disk5 will be used.

So, if I afterwards decide to change out disk 4 and rebuild from parity, will I get corrupt files there as well because disk5 got som errors because of the errors on disk 1?

 

If so, how do I isolate it to only affect disk1 and 5? Do I have to build new parity?

2 minutes ago, kno said:

So, if I afterwards decide to change out disk 4 and rebuild from parity, will I get corrupt files there as well because disk5 got som errors because of the errors on disk 1?

 

If so, how do I isolate it to only affect disk1 and 5? Do I have to build new parity?

All disks are involved in rebuild. There is no way to "isolate". When you repair corruption parity is updated just like any other write, so parity will also be correct.

 

See here for a better understanding of parity. It is not complicated, and if you understand it many of these things become obvious.

 

https://wiki.lime-technology.com/UnRAID_6/Overview#Parity-Protected_Array

1 hour ago, kno said:

So, if I afterwards decide to change out disk 4 and rebuild from parity, will I get corrupt files there as well because disk5 got som errors because of the errors on disk 1?

 

You need to run a correcting parity check after replacing all failed disks.

Ok, thank you for the advice.

 

I think I will do the following:
1. Replace disk 4 and rebuild from parity.
2. Check for corrupt files (checksums)
3. Check/repair filesystems on all disks
4. Parity check
5. Shink raid (remove disk 4 and move content to other disks)

6. Parity check
6. Add second parity disk and biuld new parity
7. Run new parity check once every 2 months?

 

What do you think of this stategy?

Edited August 26, 20178 yr by kno

Step 5 is not that simple as There is no facility built into unRAID to reduce the number of drives in the array while maintaining protection.

 

the only way ito reduce the number of array disks s to use the ‘New Config’ tool with the option to retain current assignments and remove the disk after doing that before starting the array.    Doing this will also invalidate parity so step 6 becomes ‘create parity’ rather than check it.   Your array is only protected again when the parity creation completes.   Note that if using the 6.4 rc7a Release there is a bug where the first time you start the array after doing a New Config all array disks show as unmountable.  Simply stopping the array and then starting it again resolves this issue.

I ran a check of the filesystem on disk 6. I need to rerun with --rebuild-tree according to the report below.

From what I understand, there is no other option if reiserfsck tell you to do so. From reading other posts I understand that this will likely throw away some files.

Should I make a checksum record of the disk before --rebuild-tree? If I compare before and after, will I be able to identify what data is lost or is this a waste of time?


reiserfsck 3.6.24

Will read-only check consistency of the filesystem on /dev/md6
Will put log info to 'stdout'
###########
reiserfsck --check started at Wed Aug 30 07:37:59 2017
###########
Replaying journal: 
Replaying journal: Done.
Reiserfs journal '/dev/md6' in blocks [18..8211]: 0 transactions replayed
Checking internal tree..  finished
Comparing bitmaps..Bad nodes were found, Semantic pass skipped
1 found corruptions can be fixed only when running with --rebuild-tree
###########
reiserfsck finished at Wed Aug 30 07:46:06 2017
###########
block 197131111: The level of the node (20417) is not correct, (1) expected
 the problem in the internal node occured (197131111), whole subtree is skipped
vpf-10640: The on-disk and the correct bitmaps differs.

You need to use --rebuild-tree, usually there's no data loss (but there can be some, check lost+found folder in the end), it may not be possible to create cheksums now with the corruption in the filesystem, you can create a file list to compare after the repair to see if any files are missing.

Edit: Diagnostics removed.

 

I ran --rebuild-tree on disk6.

 

When I start the server I get the following message. How do I proceed?

Unmountable disk present: Disk 6 • WDC_WD60EFRX-68L0BN1_WD-WX11DC6H1L2P (sdj)

Format will create a file system in all Unmountable disks, discarding all data currently on those disks. Yes I want to do this

 

Edited September 4, 20178 yr by kno

Did --rebuild-tree finish? Filesystem is still corrupt.