Array rebuild stuck

[xtrips]

Hello,

I had 2 out of 9 HDDs with errors, so I decided to replace them, one at a time of course.

First I ran a parity check that ended with no errors.

Then I replaced one of the faulty HDDs. 

I went through the usual interface that popped up about a missing disk and I assigned the replacement disk for that position and them started the array.

It started to rebuilt the array but after 2 hours I went to check if everything was fine only to find out that it was stuck at 25%. 

The error counter kept rising on the parity disk and I couldn’t use any of my files on the NAS. 

I could browse the files though, but not access them. 

I gave it a chance for one more hour but since only the error counter kept rising I decided to Cancl the parity check and reboot. None of the commands responded through the interface. So I decided to power off manually and reboot.

It started again rebuilding the Parity from 0, and no access to the files.

For the moment it is progressing and I hope it will until the end but I would appreciate your insight. 

Thanks

[trurl]

What sort of errors were the disks showing? You probably should have asked for advice. Possibly none needed replacing. Connection issues are much more common than failed disks.

 

Go to Tools - Diagnostics and attach the complete diagnostics zip to your next post.

[xtrips]

43 minutes ago, trurl said:

What sort of errors were the disks showing? You probably should have asked for advice. Possibly none needed replacing. Connection issues are much more common than failed disks.

 

Go to Tools - Diagnostics and attach the complete diagnostics zip to your next post.

Thanks for asking about the old disks but this is beside the point. The disks were very old, too small and the errors were pointing to a future failure.

Now the urgency is to restore the access to my NAS and the Errors count on the Parity disk is still growing all the time, while the Parity rebuild is still slowly progressing though. But as I said it was exactly like that before until it got stuck at 25%.

[trurl]

56 minutes ago, trurl said:

Go to Tools - Diagnostics and attach the complete diagnostics zip to your next post

 

[xtrips]

I would like to say that it is not the 1st time that I swap a faulty disk. So I know something is not right this time. I stopped the Parity/Array rebuild and tried to stop the array, but again the NAS dos not stop the array and just waits, Stopping services, with the CPU's at 100%.

novanas2-diagnostics-20181230-1646.zip

Edited December 30, 2018 by xtrips

[trurl]

Your parity disk has dropped offline. Check connections, try again, and post a new diagnostic.

[xtrips]

I manually powered down the whole NAS again. This time at startup it started with Array started !?!
So I decided to proceed as follows:
- I stopped the array

- I assigned the new available HDD to the missing slot (the bad HDD I removed)

- I started in maintenance mode
This time it is checking Parity and rebuilding array without errors on the Parity drive, and the only counter that progresses is the Write counter on the new HDD, which is a good sign. 
The Array rebuild progress seems to be faster too.

I am crossing my fingers.

[xtrips]

The problem happened again.

The Array rebuild progress went up to 19% and then the Write counter on the new HDD stopped and the Errors counter on the Parity disc is climbing through the roof.

What should I do?

 

novanas2-diagnostics-20181230-2022.zip

Edited December 30, 2018 by xtrips

[trurl]

You didn't mention it, so I have to ask. Did you?

3 hours ago, trurl said:

Check connections

 

[xtrips]

You didn't mention it, so I have to ask. Did you?
 

Forgot to mention it.
I checked all the connections, electrical and data. All is hooked fine.
In the meantime the Data rebuild started its progress back and stands at 31% now, but the errors on the parity disk are still growing fast.


Sent from my iPhone using Tapatalk Pro

[trurl]

That new diagnostic you posted still has no parity disk. How is that disk connected?

 

Cables should not be bundled and have enough slack to allow the connector to sit square. Possibly cable should be changed.

[xtrips]

What am I seeing there then?


Sent from my iPhone using Tapatalk Pro

[trurl]

You're seeing errors from trying to read the parity disk that isn't connected.

[xtrips]

You're seeing errors from trying to read the parity disk that isn't connected.

I do not understand, so how does the Parity disk show up on the list with the greenlight?


Sent from my iPad using Tapatalk Pro

[JorgeB]

Swap parity disk's both cables with another disk and try to rebuild again, see if the problem follows the disk or the cables

[xtrips]

17 hours ago, trurl said:

You're seeing errors from trying to read the parity disk that isn't connected.

You were right of course. At the end of the Data Rebuild it finally refreshed the status of the Parity disk and it was missing.
So I powered off, hooked and tightened everything inside (although I didn't find any obvious disconnection).

After a restart it looks like everything is running smoothly and it is rebuilding the Parity disk.

But in the meantime it looks like the content of the replaced disk is missing as the new disk has nothing written on it.

I prefer to let it finish the Parity rebuild as is and end up with a functional and protected NAS.

I still have the replaced disk and although it a few reallocated sectors it was still readable.
Is there a way to read the content on a Windows PC and transfer the content to the NAS?

[trurl]

I didn't know you were going to try to complete the rebuild without fixing the connection problem. The data rebuild was pointless with all those parity read errors. Not sure why you thought you should continue.

 

If you are rebuilding parity now then there will be no way to rebuild the data disk.

[trurl]

Windows can't read Unraid disks, and it would be faster to use the Unassigned Devices plugin to access the original disk anyway.

[xtrips]

4 minutes ago, trurl said:

Windows can't read Unraid disks, and it would be faster to use the Unassigned Devices plugin to access the original disk anyway.

How can I do that? All my HDD ports are used.
Are you sure there is no XFS plugin I could use to read the content?

 

Strike that. I understand the only way is to load Linux on a VM.
To much wasted time.
Would it be possible to ensure my data is safe somehow, maybe after parity rebuild, then disconnect the parity disk (how?) and therefore use its connection for the replaced disk. Copy its data to another disk, disconnect it, reconnect the parity and rebuild.

Possible?

Edited December 31, 2018 by xtrips

[trurl]

Post a screenshot of Main - Array Devices

[trurl]

And a new diagnostic

[xtrips]

1 minute ago, trurl said:

Post a screenshot of Main - Array Devices

 

Disk 4 is the new HDD

[xtrips]

Would it be possible to ensure my data is safe somehow, maybe after parity rebuild, then disconnect the parity disk (how?) and therefore use its connection for the replaced disk. Copy its data to another disk, disconnect it, reconnect the parity and rebuild.

Possible?

novanas2-diagnostics-20181231-1529.zip

[trurl]

 

1 minute ago, xtrips said:

 

Disk 4 is the new HDD

So it looks like you must have told it to Format disk4, which would explain why it is empty instead of just unmountable. Not that it matters in this particular case since the rebuild was bad anyway, but for future reference, Format is never part of the rebuild process.

[xtrips]

BTW, disk 1 is also bound to be discarded and replaced. Could I copy its content to disk 4, then after that connect the replaced disk (with the missing data) instead of disk 1 and also copy its content, or rebuild parity?