Disk disabled - content not emulated (anymore?)

[Harlequin42]

Hey guys,

 

i think i made a big mistake and need help.

I searched the forum, but found nothing like that, maybe i am to nervous atm.

I had some read issues on disk 3, then i could not access my data anymore.
So i checked unraid and figured out, that disk 3 was disabled and it said "disk disabled / content emulated", but i could not access it.

 

So i stopped the array and then disk 3 was disabled and disk 4 was missing.

I checked the cables and found an issue with the power connector at the case of disk 3 & 4.
Then disk 4 appeared again, but disk 3 was still disabled.

So i restartet the system and was able to get disk 3 also back, but it was still disabled.

When i startet the array again, disk 3 was in status "unmountable - no file system found".

 

And then i think i made my big mistake.

I formated the disk hoping that i could rebuild ist afterwards.

Formatting was done, disk was accessable, but completly empty.

All my disks are encrypted so its more difficult.

 

I don't want do destroy more.

 

Attached the diagnostic file.
I hope you can help me.

Thank you
 

 

Edited April 29, 2020 by Harlequin42

[itimpi]

I am afraid that you have erased the contents of disk3.     The dialog that was displayed when you tried to do a format would have told you that this was going to erase the contents of the disk happen and that parity would be updated to reflect this.

 

i know it is a bit late now, but the correct recovery procedure would have been the one documented here in the online documentation.  
 

Did you actually start the rebuild, or have you still got the physical disk3 in the state it was when it got disabled.    If so, then it may be possible to get the data off the physical drive.     Do you have backups of the data if necessary?

 

[Harlequin42]

Thanks for your reply.

See attached the actual state.
It's still disabled, but accessable, but empty.

4TB are gone, but i think i got a backup of the most of the files, only the newest files should be missing.

 

Then i have to compare the incomplete shares with my backup shares.
Is there an easy way to compare them? rsync?

[Harlequin42]

But what to do with the disabled device?
It still shows up as disabled and some shares show that they are partly unprotected.

 

Edit:
I stopped the array, changed disk 3 to no device.

Started the array again, stopped it an changed the disk 3 back to it's original one.
It starts now to rebuild and says "content emulated", but if i browse the device now, its empty, but its still rebuilding and only writing to disk 3.

It will take 10h, we will see. 

Edited April 27, 2020 by Harlequin42

[trurl]

1 hour ago, itimpi said:

Did you actually start the rebuild, or have you still got the physical disk3 in the state it was when it got disabled.    If so, then it may be possible to get the data off the physical drive.

1 hour ago, Harlequin42 said:

It's still disabled, but accessable, but empty.

So you formatted the emulated disk, not the physical disk. And that emulated disk is empty. The physical disk may still have your files. Stop the array, unassign the disk, then restart the array with the disk unassigned. Do NOT reassign the disk to the array or it will be rebuilt as the empty drive that is currently being emulated.

 

See if the disk will mount as an Unassigned Device using the Unassigned Devices plugin.

 

[Harlequin42]

i started the rebuild like 15 minutes ago.
I stopped it at 2% and tried what you said, but i think its too late.

When i try to mout it as unassigned device it says as filesystem luks not xfs like when its in the array and i am unable to mount it.

 

"Apr 28 02:12:15 SN-HOME01 unassigned.devices: Mount of '/dev/mapper/ST6000VN0033-2EE110_ZAD35XZT' failed. Error message: mount: /mnt/disks/ST6000VN0033-2EE110_ZAD35XZT: wrong fs type, bad option, bad superblock on /dev/mapper/ST6000VN0033-2EE110_ZAD35XZT, missing codepage or helper program, or other error."

 

i dont know where i can change it.

Edited April 28, 2020 by Harlequin42

[trurl]

If you started rebuilding the disk then you might as well let it complete and accept the formatted empty disk result. Then it will be in sync with parity and your array will be protected again. Then you can go to your backups for the files.

 

First mistake was not asking for advice before doing anything. Even then we probably could have helped if you had waited for further advice.

[Harlequin42]

4 hours ago, trurl said:

If you started rebuilding the disk then you might as well let it complete and accept the formatted empty disk result. Then it will be in sync with parity and your array will be protected again. Then you can go to your backups for the files.

Yeah that's what i was thinking was my only option now.

Quote

First mistake was not asking for advice before doing anything. Even then we probably could have helped if you had waited for further advice.

Thanks for your help, but i want to learn how this all works and want to be able to fix the most things on my own.
I had to swap devices already and i had to rebuild the array after a disk failed and i figured it out.
I thought i could handle this too.
As i mentioned, i was able to access disk 4 again and when there is only one disk missing it is normally no problem.

If you tell me this, it sounds for me like that i shouldn't do anything but asking for help. This makes me more insecure and probably asking dumb questions instead of getting more familiar with the system and be more confident in solving issues and maybe some day helping others like you are able today.

 

Can you tell me with the logs why disk 3 wasn't recognized as part of the array after the reboot? Is there an issue or just bad luck?

Anyway thanks guys for your help.
I learned my lesson

[itimpi]

1 hour ago, Harlequin42 said:

Can you tell me with the logs why disk 3 wasn't recognized as part of the array after the reboot? Is there an issue or just bad luck?

Once a disk is disabled (because a write failed) it stays in that state until action is taken to bring it back into sync with parity.   The normal action is to rebuild the disk to get it back into the array in match with parity.

 

The problem was that you almost certainly took the wrong action to regain access as the commonest cause of an 'unmountable' disk is some file system level correction that can easily be fixed by running the appropriate tool (file system check/repair) as documented here.  This can be done on the "emulated" drive before attempting a rebuild.  If you had not used the Format option then I would think all your data would have been recovered intact.

 

If you have any suggestions on how the wording of the Warning dialog that pops up when you attempt to format a drive can be improved in a way that might have stopped you proceeding with it then please feel free to make them.

[Harlequin42]

1 hour ago, itimpi said:

Once a disk is disabled (because a write failed) it stays in that state until action is taken to bring it back into sync with parity.   The normal action is to rebuild the disk to get it back into the array in match with parity.

 

The problem was that you almost certainly took the wrong action to regain access as the commonest cause of an 'unmountable' disk is some file system level correction that can easily be fixed by running the appropriate tool (file system check/repair) as documented here.  This can be done on the "emulated" drive before attempting a rebuild.  If you had not used the Format option then I would think all your data would have been recovered intact.

 

If you have any suggestions on how the wording of the Warning dialog that pops up when you attempt to format a drive can be improved in a way that might have stopped you proceeding with it then please feel free to make them.

Thanks for the explanation.

So what i should had done would be:
1. Check hardware if everything is fine so far.

2. Stop the array and check if all devices are there and shown up correct.

3. If one disc is not in config anymore, attach disc als unassigned device and follow your suggested guide and never ever formatting it and thinking rebuild would kick in and restore all data.

4. Try to fix file system errors.

5. If this doesn't work, format disk (or better insert a new disc) and put it back in the array and restore it.

 

But i am not sure if step 5 would work, because disc config said not "disc xxxxx missing", there was noting just the disabled icon not mentioning there was a disc before.

 

Edit:
i don't think the wording of the warning message is bad.
It's more about the whole process.
I was sure i need an empty disk to rebuild the data, but in this case the rebuild did not start automatically.

But it was a bad idea to format it when disc config says it is not part of the array.

Edited April 28, 2020 by Harlequin42

[itimpi]

16 minutes ago, Harlequin42 said:

Thanks for the explanation.

So what i should had done would be:
1. Check hardware if everything is fine so far.

2. Stop the array and check if all devices are there and shown up correct.

3. If one disc is not in config anymore, attach disc als unassigned device and follow your suggested guide and never ever formatting it and thinking rebuild would kick in and restore all data.

4. Try to fix file system errors.

5. If this doesn't work, format disk (or better insert a new disc) and put it back in the array and restore it.

 

But i am not sure if step 5 would work, because disc config said not "disc xxxxx missing", there was noting just the disabled icon not mentioning there was a disc before.

 Not quite (I think you are probably over-thinking this ) as the recovery is simpler:

 

2.  See if you can run file system/check repair on the emulated disk.  Ideally after doing this if repair worked you stop/restart the. array and the emulated disk will now mount and you will be able to see its contents.

3 . Decide if the disk marked as disable is actually OK and the failed write was caused (commonest) by an external factor or whether you need to use a replacement disk.

4.  Follow the documented procedure for rebuilding a disk (on to itself if not using a replacement).   This will rebuild the contents of the emulated drive onto the physical drive.

 

If step 2 failed then you might want to ask for advice if not sure how to proceed.

[trurl]

6 hours ago, Harlequin42 said:

i want to learn how this all works and want to be able to fix the most things on my own.

I encourage that attitude, and I enjoy teaching how this all works. Some people probably get tired of all my explanations.

 

3 hours ago, Harlequin42 said:

I was sure i need an empty disk to rebuild the data

Doesn't matter what is on the disk since it will be completely overwritten during the rebuild. But formatting a disk in the array is where the mistake is.

 

Many people only have a vague idea about what format does. They often think it is just "prepare a disk for use", whatever that might mean. Format doesn't do anything except write an empty filesystem to the disk. That is what format has always meant in every operating system you have ever used. That empty filesystem is just the metadata that represents an empty top level folder ready to receive new folders and files.

 

But note that format is a write operation. Unraid treats it just like any other write operation, by updating parity. How could it be otherwise? Parity needs to be in sync with that empty filesystem you just created. So when you rebuild a formatted disk, the result is a formatted disk.

 

While you don't need an empty disk, there is one (and only one) situation where you need a clear disk. But a clear disk is not an empty disk, a clear disk has all zeros written to it. A clear disk is required when you add a disk to a new data slot in an array that already has valid parity. This is so parity will remain valid since all those zeros have no effect on parity.

[trurl]

3 hours ago, Harlequin42 said:

attach disc als unassigned device

3 hours ago, itimpi said:

Not quite (I think you are probably over-thinking this ) as the recovery is simpler

The only reason I brought Unassigned into the picture is because you had already formatted the emulated disk. I thought the physical disk might still be something we could work with Unassigned, since putting the disk back into the array would have rebuilt it with the empty filesystem.

[itimpi]

2 hours ago, trurl said:

The only reason I brought Unassigned into the picture is because you had already formatted the emulated disk. I thought the physical disk might still be something we could work with Unassigned, since putting the disk back into the array would have rebuilt it with the empty filesystem.

I agree that might be the best thing to try at the moment :).    The question asked was what SHOULD have been done (at least that is how I took it)

[trurl]

1 hour ago, itimpi said:

I agree that might be the best thing to try at the moment :).    The question asked was what SHOULD have been done (at least that is how I took it)

That's the way I took it also. I was just explaining why UD was even brought into the discussion since it would not normally be needed.

[Harlequin42]

Thanks guys for your help and explanation. 

Parity sync is nearly done and then i will copy back the data from my backup.

That should recreated 90% of the data, 5% are not important like iso files etc and the rest is maybe half a day of scanning and renaming documents again.

 

 

Edited April 28, 2020 by Harlequin42