First time hard drive fail, so this was the first time I've had to replace a drive.  Apparently I did something very wrong  as my disk as been "rebuilt" except  it's only at 114 GB usage rather than 2.45 TB usage.  I've likely already killed off any life lines by formatting the initial disk 3 that had an error

 

This was my order of operation.

 

1. I wake up with notifications that disk 3 isn't working.  I got the red x:

 

 

I posted on the forums and the recommended action was to replace the disk, but it was mentioned that I could attempt to rebuild with the same disk even though it was unlikely to work.  I decided to give it a go.  I stopped the array and removed disk 3 from the array on the GUI.  Then I started the array up, added disk 3 (my original and broken disk) and selected the option to rebuild.  After rebuild, I still had red x.  Bummer.  I ordered two new hard drives.  One to replace, and one to expand.

 

I received the hard drives, popped them in and pre cleared both.  Everything looks good.  I now stop the array and swap broken disk 3 with my new disk ending in RDHU, another WD 3TB disk.  It takes a good 8 hours, but I finally get a message that "Parity sync/Data Rebuild finished". 

 

Except one problem.  Disk 3 only has 114GB of data rather than the 2.45 it used to have.

 

 

While all this was going on, I continued to use the array as I've never read of any harm doing so.  I continued to read/write to the array.  All writing is done to shares, none of which have a specific allocation besides some only to cache drive.

 

Where did I go wrong in my order of operation, and am I SOL on recovering any of my disk 3 data?

 

Edited October 24, 20178 yr by Living Legend

Also now realizing that because I use shares which allocated the files, I really have no clue what I'm actually now missing or how to find out.

 

Bigger issue than I initially anticipated.  

 

One thing I'm noticing that I don't understand.  Why does it say that the last parity check was completed the same time the rebuild ended and found no errors?  Does this mean that somehow the parity disk was modified before the new disk was rebuilt?

Edited October 24, 20178 yr by Living Legend

Writing to the Array during a rebuild... i think that was a bad idea - NEVER do this

If you rebuild a Disk, "all" Disks are involved - also the parity disk. If you write Data during a rebuild, the parity changes - but it is needed for rebuild.

That may explain why you have now only 114GB instead of 2,45TB

Edited October 24, 20178 yr by Zonediver

9 minutes ago, Zonediver said:

Writing to the Array during a rebuild... i think that was a bad idea - NEVER do this

 

 

I can't remember for certain, but I thought that the array was up and live during the rebuild.  Had it spun down the disks and stopped the array, I certainly wouldn't have tried to over rule this.  But I believe the array was active, so I didn't see an issue, or remember reading anything that said not to.  Had it said no writing and left it running, I would have had to shut down every single docker, VM, plugin and user scripts manually to make sure that no modifications were made to the array.  Seems like a lot for the user to have to be cognizant of doing manually if it were truly a critical thing during the rebuild process.

 

I'm not going to say anything negative about unRAID as it's far more likely the error was on my end, but it'd sure be a shame if in my first disk failure something out of my control went wrong on the rebuild process.

Edited October 24, 20178 yr by Living Legend

13 minutes ago, Living Legend said:

 

 

I can't remember for certain, but I thought that the array was up and live during the rebuild.  Had it spun down the disks and stopped the array, I certainly wouldn't have tried to over rule this.  But I believe the array was active, so I didn't see an issue, or remembering reading anything that said not to.  Had it, I would have had to shut down every single docker, VM, plugin and user scripts manually to make sure that no modifications were made to the array.  Seems like a lot for the user to have to be cognizant of doing if it's truly that critical.

 

I'm not going to say anything negative about unRAID as it's far more likely the error was on my end, but it'd sure be a shame if in my first disk failure something out of my control went wrong on the rebuild process.

 

Dockers and VMs are normaly running on the Cache-SSD/HDD which is "not" part of the Array - except you dont use a cache-drive... also bad, sorry

Edited October 24, 20178 yr by Zonediver

Just now, Zonediver said:

 

Dockers and VMs are running on the Cache-SSD which is "not" part of the Array - except you dont use a cache-drive... also bad, sorry

 

Hmm, not sure I follow.  I am using a Cache drive.  It doesn't have to be, but it is an SSD.  VMs don't have to be on a cache drive.  Mine are actually located on an unassigned device outside of the array, but they do have access to the array through SMB.  Dockers are on my cache drive, but also interact quite a bit with the array per usual...

 

5 minutes ago, Living Legend said:

 

Hmm, not sure I follow.  I am using a Cache drive.  It doesn't have to be, but it is an SSD.  VMs don't have to be on a cache drive.  Mine are actually located on an unassigned device outside of the array, but they do have access to the array through SMB.  Dockers are on my cache drive, but also interact quite a bit with the array per usual...

 

 

Unassigned devices are ok but you have to be sure that nothing do a write to the array during a rebuild because it changes the parity which is needed for rebuild.

I dont use VMs but when i do a rebuild i deactivate all my dockers - just a precaution - but i have only three of them.

Edited October 24, 20178 yr by Zonediver

Apparently, that's not the case:

 

Intuitively, what you're saying makes sense, but I figured that if unRAID could not handle it, they would not allow users the ability to use it.  There are too many variables that could be accessing the array that it would be difficult to control.

 

 

Edited October 24, 20178 yr by Living Legend

As i said, just a precaution - but if unraid is able to handle this, the error must come from another side.

I was pretty shure it comes from the parity change...

If you have not rebooted yet, post a complete log for the experts to have a look on it.

Edited October 24, 20178 yr by Zonediver

If it makes ya feel better!

 

But if the devs say it's of no consequence, I don't think twice.

 

If you are replacing a parity drive however, then the array will be stopped.  My case was an array disk.  Still unsure where I went wrong...

Thats the question for the experts - maybe they can help.

Something went wrong for sure but we'd need the diagnostics from the rebuild to see what happened.

1 hour ago, Zonediver said:

Unassigned devices are ok but you have to be sure that nothing do a write to the array during a rebuild because it changes the parity which is needed for rebuild.

It is perfectly OK to write to the array during a rebuild.    It does update parity, but in a way that is consistent with the rebuild.  

 

The reason that this is normally recommended against is that it can badly affect performance, and also if something goes wrong with the rebuild then (depending on whether you were writing to the disk being rebuilt) it might make it harder to work out what data might have gone missing.

Is there something specific I need to do to get the diagnostics from the rebuild?  Does that differ from current diagnostics?

 

unraid-diagnostics-20171024-0253.zip

 

I'm currently running a parity check with writing corrections disabled.  I'm naively hoping that maybe the 3rd disk was somehow built incorrectly and that my parity disk is still capable of assisting in the rebuild of disk 3 the way it's supposed to be 

 

Forunately I'm mostly media, so it's of minimal consequence.  I do however have a few generic data shares, and really have no idea how to even tell what's missing.

Edited October 24, 20178 yr by Living Legend

Syslog starts over after every reboot, current diagnostics would be fine if you hand't rebooted since, but you have, when there's trouble you need to save them all before each reboot.

Just now, johnnie.black said:

Syslog starts over after every reboot, current diagnostics would be fine if you hand't rebooted since, but you have, when there's trouble you need to save them all before each reboot.

 

Soooooo, does that mean I'm SOL?

 

Any hail mary suggestions?

You can try running a file recovery util on both the formatted (if it was just a quick format) and rebuilt disks, like this one:

 

http://www.ufsexplorer.com/download_stdr.php

1 minute ago, johnnie.black said:

You can try running a file recovery util on both the formatted (if it was just a quick format) and rebuilt disks, like this one:

 

http://www.ufsexplorer.com/download_stdr.php

 

Two questions.

 

1. Why would I need to run this on the rebuilt disks?    

2. I assume I need to remove the disk from my server and place into a PC running this software?

The disk boxed in red was the original disk 3 of array.  It currently shows no file system.

 

Depending on what happened exactly the rebuilt disk may contain the original missing data but it was deleted or the filesystem corrupted.

You need to use a different computer to run it.

I looked in my log file out of curiosity.  I guess unRAID saves certain diagnostics without being prompted?  I have one from a few hours after the rebuild was completed on my flash drive.

 

Attatched below:

unraid-diagnostics-20171024-0200.zip

Edited October 25, 20178 yr by Living Legend

Rebuild looks normal, only thing that's not normal is preclear resuming the disk currently being rebuild, but IIRC that was just a bug and nothing was actually being written to the disk, though I still don't like seeing that on the log.
 

Oct 23 13:11:07 unraid kernel: md: import disk3: (sde) WDC_WD30EZRZ-00GXCB0_WD-WCC7K1XJRDHU size: 2930266532 erased
Oct 23 13:11:07 unraid kernel: md: import_slot: 3 replaced
...
Oct 23 13:13:54 unraid kernel: md: recovery thread: recon D3 ...
Oct 23 13:13:54 unraid kernel: md: using 1536k window, over a total of 2930266532 blocks.
...
Oct 23 13:13:56 unraid preclear.disk: Resuming preclear of disk 'sdg'
Oct 23 13:13:56 unraid preclear.disk: Resuming preclear of disk 'sde'

 

sde is the new disk3

 

When did you notice the missing data? Right after starting the rebuild, during or after it finished?

I just had exactly the same experience as Living Legend. My drive 12 (4TB) was disabled during a parity check which has more than 2000 errors. I suspect a too much bent cable. The SMART report on the disk is as clean as it can be. So I reinstalled the disk. Precleared it once. No problems. And then reinserted it as a 'new' disk back into the array. Rebuild took about 12 hours and was finished without any errors. Before, it had at least 3TB of data on it. Not it's only 100GB which were probably coming from a backup job that ran after it was rebuilt. The array was running during the rebuild.

 

I tried to look at the syslog but the syslog page was unresponsive. The parts that I could get to were from after the rebuild and didn't show any problems. I rebooted but the disk 12 is still empty. So I lost 3TB!

 

What is going on here? This severely shatters my trust in unRAID!

Edited October 30, 20178 yr by tazman

6 minutes ago, tazman said:

I just had exactly the same experience...

That is not obvious.

 

Best to start your own support thread and leave this one for helping the OP.

4 hours ago, tazman said:

What is going on here? This severely shatters my trust in unRAID!

 

Sounds more like you have a hardware issue, as trurl suggests better open a new topic.