Data drive not rebuilding after corruption

[Scales]

I have a 4TB drive formatted in XFS and I was unable to repair corruption. I pulled the drive and it tested 100% using HD Sentinel so I thought that I could delete the partition on my computer then insert the drive back into my Unraid server and have the data rebuilt. Well I was wrong. The drive is showing as empty on the main gui page. Is my data gone or is there something that I am missing?

[JorgeB]

Parity can't help with filesystem corruption, but if it's showing empty it suggests you formatted the disk, and that's never part of a rebuild.

 

What was the original error with xfs_repair? You should post the command used and the output, also what Unraid release.

[itimpi]

12 hours ago, Scales said:

I have a 4TB drive formatted in XFS and I was unable to repair corruption. I pulled the drive and it tested 100% using HD Sentinel so I thought that I could delete the partition on my computer then insert the drive back into my Unraid server and have the data rebuilt. Well I was wrong. The drive is showing as empty on the main gui page. Is my data gone or is there something that I am missing?

It is not clear exactly what steps you took and in what order?  Did you ever take steps to get the drive treated as disabled or missing in Unraid?   If not then it will be looking at the current contents.   Have you tried starting Unraid without the disk actually plugged in to see what happens?  Depending on exactly what you did then if you are lucky and parity is valid it might show the disk is now being 'emulated' with contents available (and then a rebuild becomes a possibility).

[Scales]

I started unraid without the disk and it was emulated. I removed the partition on the corrupted disk using Windows based computer and inserted the drive back into unraid with no file system. I then checked the box to format and the drive was returned to the array with and XFS partition. After that the party-sync /  rebuild commenced. Upon completion it looks like there is only 5.2GB returned of the 4TB. I am thinking the corruption was picked up on the last parity check perhaps and that is why the data did not return. 

[itimpi]

1 hour ago, Scales said:

I started unraid without the disk and it was emulated. I removed the partition on the corrupted disk using Windows based computer and inserted the drive back into unraid with no file system. I then checked the box to format and the drive was returned to the array with and XFS partition. After that the party-sync /  rebuild commenced. Upon completion it looks like there is only 5.2GB returned of the 4TB. I am thinking the corruption was picked up on the last parity check perhaps and that is why the data did not return. 

The mistake was doing the format as that would end up formatting the emulated drive and updating parity to reflect the fact that the disk has an empty file system.   The rebuild would then have rebuilt the result (an empty file system) from the emulated drive.   The correct steps would have rebuilt the disk with the data intact.
 

What version of Unraid are you running.    In the 6.8 release a big warning pop-up was added to try and stop people formatting the drive when they should not (in previous releases the warning was still present but less intrusive so easier to overlook) as format is NEVER part of the rebuild process.  If you DID get this pop-up how could the wording have been improved to make you realise this was not the correct action to take?

[Scales]

I am running 6.7. It is strange that the disk appears have rebuilt with 5.4GB of data. On the bright side I now have approx 4TB free  Thanks for help, If anything happens again I will reach out before attempting to screw it up.

[trurl]

22 minutes ago, Scales said:

It is strange that the disk appears have rebuilt with 5.4GB of data.

Most likely just filesystem overhead and no actual files.

[Scales]

22 minutes ago, trurl said:

Most likely just filesystem overhead and no actual files.

The files are there and accessible. These would have been added a few years ago.

[trurl]

You formatted the disk. No doubt you have files somewhere on the array but what evidence do you have that the formatted disk has files?

[trurl]

If the disk does in fact have files, then they would have been written after the disk was formatted. This could have even been during the rebuild. Or if it is a cache-yes share, then maybe they were moved to the disk from cache.

[Blaqwolf]

Hi, I was looking for some answers myself and came across this.

 

1st Question: wouldn't formatting the "bad" drive outside of Unraid and then putting it back into the array be the same as putting in a brand new drive?

 

2nd question:  where does it say emulating, next to the disk # that is missing?

 

3rd question: I recently had an 8TB drive come up as no file system found ( I think it was caused by a faulty molex connector plugged into the backplane) I restarted the system and parity-sync/data-rebuild started. After 16 hours it finished, I restarted and nothing.... 17.3 MB were used. So I then decided to physically pull the hard drive, intending to do a format (like Scales did) but when I restarted the array without the 8TB drive I didn't see anything saying it was emulating. 

 

4th Question: can you see data on a drive being re-built while it is being rebuilt?

 

5th question: With as little information as I can provide now, what does everyone think I did wrong to lose the emulated data? I have 1 parity drive at 10TB and parity was done at the beginning of the month with no errors (until the drive showed no file system).

 

6th question: I have ordered some new Noctua fans to quiet down my Norco 20 bay case but to install them I am going to have to unplug all those SAS to Sata cables from my LSI HBA's and unplug all those power cables and also 10 sata cables from my motherboard. What steps should I do to not lose any data or drives? Is it important to plug the same cables back into the same spots on the backplane?

 

Please help me Obi-Wan!!

 

Thanks!

[trurl]

2 minutes ago, Blaqwolf said:

With as little information as I can provide now

Go to Tools - Diagnostics and attach the complete diagnostics zip file to your NEXT post.

 

Don't do anything without further advice.

 

[trurl]

3 minutes ago, Blaqwolf said:

1st Question: wouldn't formatting the "bad" drive outside of Unraid and then putting it back into the array be the same as putting in a brand new drive?

Totally pointless. If the idea is to rebuild the disk then formatting it outside Unraid accomplishes nothing at all. Many people have only a vague understanding of the word "format". It doesn't apply here.

 

5 minutes ago, Blaqwolf said:

2nd question:  where does it say emulating, next to the disk # that is missing?

If a disk is disabled then it is emulated. Sounds like instead you have (or had) filesystem corruption. Parity typically can't help with filesystem corruption.

 

7 minutes ago, Blaqwolf said:

3rd question: I recently had an 8TB drive come up as no file system found ( I think it was caused by a faulty molex connector plugged into the backplane) I restarted the system and parity-sync/data-rebuild started.

This is unclear and probably missing important details. If you didn't shutdown from the webUI then what you got from restarting it was a parity check for unclean shutdown. Simply restarting without doing anything else will not result in a parity-sync/data-rebuild.

 

12 minutes ago, Blaqwolf said:

4th Question: can you see data on a drive being re-built while it is being rebuilt?

You can see the data on a disabled/emulated disk even before you begin rebuilding it as well as while it is being rebuilt. As mentioned it doesn't sound as if you had a disabled/emulated disk.

 

14 minutes ago, Blaqwolf said:

5th question: With as little information as I can provide now, what does everyone think I did wrong to lose the emulated data? I have 1 parity drive at 10TB and parity was done at the beginning of the month with no errors (until the drive showed no file system).

As mentioned, doesn't sound as if emulation was ever involved. If you had filesystem corruption then possibly it can be repaired if you haven't done anything else. If it was showing no filesystem then that almost certainly indicates filesystem corruption.

 

16 minutes ago, Blaqwolf said:

6th question

I suggest not making any hardware changes until you get your other problems fixed (as well as they can be).

 

Do you have backups of anything important and irreplaceable? You should.

[Blaqwolf]

41 minutes ago, trurl said:

Totally pointless. If the idea is to rebuild the disk then formatting it outside Unraid accomplishes nothing at all. Many people have only a vague understanding of the word "format". It doesn't apply here.

Okay, if I had pulled the corrupted drive out (yes, I knew it was file corruption. After I pulled it I tested with WD tools and Spinrite, no physical problems detected. I think the bad power cable caused the corruption) and replaced it with a new drive would the array rebuilt the data to the new drive? If so then why wouldn't it do it with the same drive that had no partition?

41 minutes ago, trurl said:

 

If a disk is disabled then it is emulated. Sounds like instead you have (or had) filesystem corruption. Parity typically can't help with filesystem corruption.

Understand that, this is why I made the mistake of pulling it out and formatting it and then putting it back into the array as if it were a "new" drive. This is after I let the system rebuild it for 16 hours.... but you might be correct about dirty shutdown, as I was not able to shutdown from the gui after that...

41 minutes ago, trurl said:

You can see the data on a disabled/emulated disk even before you begin rebuilding it as well as while it is being rebuilt. As mentioned it doesn't sound as if you had a disabled/emulated disk.

I can't remember at this moment if it was emulating that drive or now, I do remember that it was disabled but not sure if that was before or after I had to do the hard shutdown.

41 minutes ago, trurl said:

 

As mentioned, doesn't sound as if emulation was ever involved. If you had filesystem corruption then possibly it can be repaired if you haven't done anything else. If it was showing no filesystem then that almost certainly indicates filesystem corruption.

I did end up wiping the partition, the 8TB went from almost full to less than 18MB after the hard shutdown. So there won't be any way to recover it, it hurts but I am more interested in how to avoid this when it happens again, or something like it. As you know, drives will fail.. not if but when.

41 minutes ago, trurl said:

 

I suggest not making any hardware changes until you get your other problems fixed (as well as they can be).

 

Do you have backups of anything important and irreplaceable? You should.

This all started after we moved houses, my temps were getting up there so I took off the top, saw that one fan was dead and another was moving too slowly. I powered down correctly and replaced those fans (with temps as I have ordered Noctua fan replacements for all fans but don't get them till Thursday), but to do that I had to unplug a lot of sata cables and a few power cables. That was when everything started going downhill.

 

Yes I have cloud backups of the important, can't replace stuff, home movies, pics, financial records but what I lost is a lot of DVD and bluray ripping!! LOL I am very well versed in "If you don't have backups then you don't have backups, period!" A hard earned lesson many decades ago.

 

What I really need to know is; does putting sata cables from HBA and/or motherboard back in different locations on a backplane cause problems? What are the correct steps (other than posting in these great forums for help) if you do get file corruption?

 

I have attached my diagnostics but I am pretty sure it won't show my mistakes from yesterday but I appreciate you taking the time.

tower-diagnostics-20200316-1825.zip

[trurl]

1 hour ago, Blaqwolf said:

After I pulled it I tested with WD tools and Spinrite, no physical problems detected. I think the bad power cable caused the corruption) and replaced it with a new drive would the array rebuilt the data to the new drive? If so then why wouldn't it do it with the same drive that had no partition?

You can rebuild to the same disk. But formatting before doing that would still be pointless. I suspect you don't know what "format" does. Format just writes an empty filesystem to a disk. Whether or not you format a disk will have no effect on the rebuild, since it will be completely overwritten by the rebuild, and it won't matter whether or not it has an empty filesystem on it before it is rebuilt.

 

1 hour ago, Blaqwolf said:

... I made the mistake of pulling it out and formatting it and then putting it back into the array as if it were a "new" drive. This is after I let the system rebuild it for 16 hours.... but you might be correct about dirty shutdown, as I was not able to shutdown from the gui after that...

This was all hypothetical in your original post, which is why I thought you were just getting a parity check. Now you say you did pulled it out and formatted it. You don't even say what filesystem you formatted it in, which makes me think you don't understand format as I said.

 

In any case, unless you had actually started the array without the drive assigned, and then reassigned the drive, then it wasn't rebuilding to that same disk. It probably was just an unclean shutdown parity check. It should have told you in the Array Operation section whether it was rebuilding/syncing or just doing a parity check.

 

1 hour ago, Blaqwolf said:

I can't remember at this moment if it was emulating that drive or now, I do remember that it was disabled but not sure if that was before or after I had to do the hard shutdown.

If it was disabled, there would have been a red X beside the drive instead of the green or white ball that normally appears there.

 

1 hour ago, Blaqwolf said:

I did end up wiping the partition, the 8TB went from almost full to less than 18MB after the hard shutdown. So there won't be any way to recover it, it hurts but I am more interested in how to avoid this when it happens again, or something like it. As you know, drives will fail.. not if but when.

The best thing you can do in the future if you have a problem is to ask for help before doing anything. The right thing to do will depend on the specifics of the situation.

 

If you have filesystem corruption and you rebuild, it will most likely result in that same filesystem corruption. If you have a disabled disk but the filesystem is OK, then rebuilding will result in a good filesystem. If you have filesystem corruption and the disk is disabled, we will usually suggest repairing the filesystem of the emulated disk before attempting the rebuild.

 

And if you can rebuild to another disk and keep the original as it was, you get another chance to try to get files from that original if there is a problem with the rebuild.

 

1 hour ago, Blaqwolf said:

I have attached my diagnostics but I am pretty sure it won't show my mistakes from yesterday

Syslog resets on reboot, so there isn't any way to tell what happened or what you did. You can setup Syslog Server so the syslog is saved somewhere you can get it after reboot.

 

Possibly the drive itself didn't fail. You said it had passed some tests. You have a lot of disks and you haven't said which disk it was. Do any of the disks give SMART warnings on the Dashboard? Give me the last 4 characters of the serial number of the disk and I will take a look at its SMART report in your diagnostics.

 

1 hour ago, Blaqwolf said:

What I really need to know is; does putting sata cables from HBA and/or motherboard back in different locations on a backplane cause problems?

Unraid keeps track of which disk is assigned to which slot by the disk serial number. It won't matter where they are plugged in. The only thing that sometimes causes an issue with this is if a controller doesn't pass the actual disk serial number to Unraid. You don't seem to have this problem.

[trurl]

2 hours ago, Blaqwolf said:

What are the correct steps (other than posting in these great forums for help) if you do get file corruption?

https://wiki.unraid.net/Check_Disk_Filesystems#Checking_and_fixing_drives_in_the_webGui

[Blaqwolf]

Trurl

 

Thanks for all your help and the link.  I feel more confident now about undoing all those cables to install my new fans. 

An update, the drive is healthy like I said, I didn't format it but I did wipe it using DBAN single pass (deleted all formatting and partitions) I installed it back into the array just a bit ago, not expecting anything since I don't see emulation but ... data is now appearing on the drive! So I am going to let it run again for about 16 hours and hope for the best. I can wait one more day to put in these quiet fans.

 

So thanks again, I will post a final update if the drive is fully reconstructed.

 

UPDATE: Unfortunately the rebuild just created 12 empty folders but I gained some experience and some knowledge so I guess not a total waste. My new Noctua fans arrived and what a difference! I can actually keep the server in the same room.

 

Thanks again trurl!

Edited March 18, 2020 by Blaqwolf
Update