Replaced Failed Drive, Rebuilt Array, Rebuilt Drive Lost All Data


Recommended Posts

Short description of my problem – Had a hard drive failure, replaced and re-synced drives. UNRaid said re-sync completed successfully but new drive is empty.

 

My Structure

 

~~~ Backups Plex Podcasts ~~~~

 

Under Plex

Movies

Music

TV Shows

Other Folders

 

Plex Uses Disks 1,2,3,6,7,9. All of which are now 8 TB Drives

 

I can remotely access all files on UNRaid except for those under Plex

 

My desktop system runs Linux Debian Buster and I connect to the UNRaid system via NFS connections.

 

I am running UNRaid Vs 6.3.3.

 

 

Back Story

 

I replaced a 4 TB Drive with a 8 TB Drive.

 

I did a re-sync successfully but during the re-sync, a 2ed 8 TB Drive Failed. I continued with the re-sync and when that was completed, I replaced the failed 8 TB Drive. I did another re-sync which completed successfully.

 

About 8 hours later a 3ed drive (8 TB) failed. I replaced that with a new 8 TB Drive and re-synced it. Normally the re-syncs take a little over 3 days to complete. This re-sync took almost 5 days to complete. When it was completed, I had the message re-sync complete successfully . The problem was there was only one file on the drive of about 10 Gig.

 

If I try to access anything from my desk top linux machine to the plex folder via the command line, the plex folder has nothing in it. I can access and go down to the file level on all the other top level folders.

 

Opening the terminal on the UNRaid UNRaid machine, I can look all the folders and files in all the drives that are part of the Plex system except for the bassicly empty 8 TB drive. In addition I can play many of my videos via Plex except for a few which I suspect were on the replaced last 8 TB drive.

 

As it stands now, I can not add videos to the Plex file system on the UNRaid machine and I can not see what videos I have. Of course I can go the UNRaid UNRaid machine and look at each hard drive to see what files are there via mntdisk1/Plex/. I also have a listing of the videos I did have before all these problems started, so if necessary, I can hopefully reload the missing videos., That is under the assumption I can remotely l drill down the file systems to the video level to add new videos.

 

Any help recovering the missing videos and/or getting the file systems available from the remote system so I can manipulate the files would greatly be appreciated. I have attached a copy of the syslog.

 

x-special/nautilus-clipboard
copy
file:///home/gemckee/Documents/Troubleshooting%20UnRAID/syslog_data

 

 

syslog shemp-diagnostics-20191022-0801.zip shemp-diagnostics-20190223-1135.zip shemp-diagnostics-20190221-1207.zip shemp-diagnostics-20190220-2352.zip shemp-diagnostics-20190218-0753.zip shemp-diagnostics-20190217-1337.zip shemp-diagnostics-20190215-1604.zip shemp-diagnostics-20190214-2019.zip shemp-diagnostics-20181124-1141.zip shemp-diagnostics-20181120-1515.zip shemp-diagnostics-20181001-2217.zip shemp-diagnostics-20180802-0620.zip shemp-diagnostics-20170609-1345.zip

Link to comment

I have included 5 copies of the logs.  The 20201018 log was today, the 20201012 log was done before I powered down when I saw that the replaced drive had not been restored properly.  The rest of the logs are (I believe) were created while I was solving the eirler drive replacement and rebuilds.  I included them in the event they may have some bearing on the problem.

 

shemp-diagnostics-20201018-2100.zip shemp-diagnostics-20201012-0958.zip shemp-diagnostics-20200928-1936.zip shemp-diagnostics-20200928-1913.zip shemp-diagnostics-20200923-0839.zip

Link to comment

The version of Unraid you are using is more than 3 years old, and the diagnostics provided by that very old version doesn't contain as much information for us as newer version. It may take a while to piece those together. You should consider upgrading just for support purposes. And of course, newer versions have fixes, improvements, and enhancements.

 

 

 

 

Link to comment

All disks are mounting though, and all but disk3 have lots of data.

 

Can you tell us exactly what you mean when you say these drives failed? How did you determine the disks needed replacement? Bad connections are much more common than bad disks, especially when you have been mucking about in the case such as when replacing disks.

 

Sorry you waited until this point to make your first post on the forum. Probably should have asked for advice as soon as you began to have any problems at all.

Link to comment
13 minutes ago, trurl said:

diagnostics provided by that very old version doesn't contain as much information for us as newer version. It may take a while to piece those together

I'm trying to figure out where in all that information to look to see what your problem is, and the information in those old diagnostics don't help narrow things down as easily as newer versions.

 

Plus, you posted so many diagnostics, any of which might show something important. Another reason to ask for help earlier instead of dumping all this on us at once.

Link to comment

I'm going to wait now for answers to these:

11 minutes ago, trurl said:

At any time during these processes did you ever agree to format any disk?

4 minutes ago, trurl said:

Can you tell us exactly what you mean when you say these drives failed? How did you determine the disks needed replacement?

 

 

Link to comment

I now know I need to update UNRaid to the current version.  I was unaware there were newer version till I tried to see what the problem was.  I did not want to update until this problem was solved so as not to increase the complexity of the situation.

 

I also tried to resolve the issue myself because I have, on a previous occasion had a problem and the on line help (Not provider of the software) ended getting me into more problems and I had to start from scratch.  It appears that this group is much better at helping people that have problems and I appreciate that.

 

The sequence of events as far as I can reconstruct them are as follows.  (In the future I plan on keeping better notes of what I did)

 

I first replaced disk 7 which was a 4 TB drive with a 8 TB drive to give me more space for Plex.  That rebuild completed successfully, but about halfway through  the rebuild, UNRaid said disk 4 failed and needed to be replaced.  After disk 7 rebuild completed, All files seemed to be working except UNRaid still said disk 4 failed and needed to be replaced.  I installed a brand new 8 TB drive in the same slot that I took the failed disk 4 hard drive from.  When I tried to use it, UNRaid said I needed to format it.  Other times I replaced hard drives I did not have to format them.  I have in the past replaced over a dozen hard drives without having to format them or have any problems.  I formatted it.  Prior  to formatting the hard drive, as I remember, no available hard drive showed up as available for disk 4.  After reformatting the hard drive, the entry for disk 4 said missing, and the newly formatted drive was available.  I selected it and the rebuild completed.  Everything worked, all files were recovered and available.  Both of the rebuilds took a little over 3 days.  Several hours (Maybe 6 0r 8 ) latter, disk 3 failed according to UNRaid which said it was bad and needed to be replaced.  I replaced it as normal, except this time the rebuild took about 4 1/2 days.  When the rebuild was finished and available according to UNRaid, I noticed the hard drive was recognised with 8 TB of free space.  At this time I could not go down below the Plex directory from my desktop.  Using the command line on the UNRaid machine I could see the Disk 3 and the Plex folder, but nothing below it (such as movies or music folders).  All the other hard drives in the Plex share have the file structure  correct down to the actual files but I can not see anything in the Plex folder on my desktop machine or my laptop machine, both of which connect to the UNRaid machine via NFS.  I looked at the smart data and I believe all of the failed drives had excessive errors.

Link to comment

Format is never part of a rebuild. "Format" means "write an empty filesystem to this disk. That is what "format" has always meant in every operating system you have ever used.

 

Unraid treats this write operation exactly as it does every other write operation, by updating parity. So, after formatting a drive in the array, parity agrees the disk has an empty filesystem. Rebuilding a formatted disk results in a formatted disk. So that seems to explain why some of your data is missing.

 

I am sure Unraid warned you about formatting the disk and even made you check a box to allow it to format.

 

You say it said the disks needed replacing. I'm pretty sure it never says that. Unraid just disables a disk when a write to it fails. After it is disabled, it doesn't use the disk again. It doesn't continue to try to use it and possibly make things worse. Instead, a disabled disk is emulated by reading parity and all other disks, so reading the data for the slot can continue, and even writing of the data for the slot can continue by updating parity. Any writes to the slot can be recovered by rebuilding.

 

But, it leaves the decision of how to fix the problem that disabled the disk up to the user. Re-enabling of the disk is usually a rebuild, possibly to the same disk if it is good. As I said, bad connections are much more common than bad disks. In a situation where you have been replacing disks or otherwise mucking about in the case, bad connections are almost always the problem.

 

There are ways to decide whether a disk is bad or whether the problem is something else.

 

Really wish you had asked for advice before doing anything at all. Probably at each step where you replaced a disk there wasn't really anything wrong with the disk.

 

It sounds like you replaced a disk then formatted the replacement. What you probably should have done is repaired the filesystem then rebuild, or rebuild then repaired the filesystem. You definitely should not have formatted.

 

Do you still have the original disk you replaced? It is possible we can recover your files from that original.

Link to comment

As mentioned, I could see you were having problems with disk7 in the first diagnostics I looked at. I didn't investigate further to see if there really was a disk problem since there was just too much to sort through before I got some more answers.

 

You say you rebuilt disk7. According to those latest diagnostics, it looks like disk7 has plenty of data. Then you say you rebuilt disk4 and that is where you say format came into play. But disk4 also seems to have plenty of data. Then you say you rebuilt disk3. Disk3 is the one that seems to have been formatted, since it has little or no data, and you seem to agree it has little or no data.

Link to comment

I know I wrote a lot of information, and I encourage you to study it and ask questions.

 

But for now, this seems to be the most important question for you to answer:

12 minutes ago, trurl said:

Do you still have the original disk you replaced? It is possible we can recover your files from that original.

 

Link to comment

I haven't looked at all those diagnostics you posted. Those you initially posted were a year or more old as noted. It is possible they have some relevance in this case though. I looked at the newest of these old diagnostics and it tells us what disk you had assigned to slot 3 a year ago. If that was the "original" disk3 when you began all these rebuilds, then maybe that is where we will find your files.

 

The newest and oldest of this last batch of diagnostics you posted has a different disk assigned as disk3, but each of these diagnostics is showing the same for disk3, so it looks like we don't have a record of any other disk3 assignments between last year and now.

 

So, was the original disk3 the same disk that was in that slot last year? If so, do you still have that disk and haven't changed it in any way since you removed it for replacement?

Link to comment

I did not submit a post I was proposing and I just received your latest regarding having the old disk 3.  I found it.  If necessary I could do what I have proposed below, but if I can recover the data from the old disk 3 that would be great.

 

No, I formatted it in another machine to see if it would be usable again.  Dumb mistake made before I started this post.

 

I explained what happened from memory and the list I keep telling me what physical hard drive are used as what share, what location they are in  on the PC  and the size of free space.  I do identify the share drives by brand, model, serial number, size and the date purchased.  I also keep track of the drives I remove so I have some record of what was replaced.   I concur that what you said about what happened to what disk is probably correct as I can not be sure of my memory.  Old age sometimes makes you forget more than you will ever know :-).

 

The replaced hard drives were installed in a 5 in 3 device which did not result in the mucking around inside the computer which could have disrupted connections.

 

Fortunately I have a list of all the videos I should have and from that,  I can try to find copies of the missing videos and I should have them on on-line backup.

 

OK, I understand what you are saying and thanks for further understanding of how UNRaid works.  It looks like that data is lost then.  I will keep your explanation for further study if I have other problems in the future.

 

What I plan on doing unless you think my plan won't work is this:

 

I have 2 new spare 8 TB Drives and I presciently have (2) 8 TB parity drives.  I will remove all my drives, and reformat the parity drives on a different PC.  I will load the latest version on UNRain on a new flash drive.  I will then install one fresh 8 TB drive as a parity drive and the other 2 drives as data drives.  I will make them Plex drives and copy the files from the old 8 TB drives that have Plex data on them.  When those copies have been completed, I will format the old drives I copied from and install them.  I will then copy more data from the other hard drives and continue the process till I have moved all the data and reinstalled all the drives.  When I am finished with that, I will then use another 8 TB Drive for a 2ed parity drive. 

 

When I install the new UNRaid usb stick, I will also upgrade it to allow more drives.

 

Does this sound like a valid (although long process) path?

Link to comment

That is what I expected, but unfortunately your diagnostics aren't helping much. I've opened many of them. That disk appears in several of those older diagnostics in your first post, but for some reason, there is no SMART information for any disks in those diagnostics. And it doesn't appear in any of those newer diagnostics which does have SMART information for the disks.

 

So, I can't tell if there is anything wrong with the disk or not.

 

Do you have the Unassigned Devices plugin?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.